Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add command to download dataset #390

Open
drobnikj opened this issue Oct 10, 2023 · 3 comments
Open

Add command to download dataset #390

drobnikj opened this issue Oct 10, 2023 · 3 comments
Assignees
@vladfrangu
Labels
backend Issues related to the platform backend. debt Code quality improvement or decrease of technical debt. medium priority Medium priority issues to be done in a couple of sprints. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@drobnikj
Copy link
Member

Users are struggling with downloading large datasets.

We need to introduce new command, that handle downloading dataset for them.
see slack https://apifier.slack.com/archives/C010Q0FBYG3/p1696943625090939
@drobnikj drobnikj added the t-platform Issues with this label are in the ownership of the platform team. label Oct 10, 2023
@jancurn
Copy link
Member

jancurn commented Oct 10, 2023 •
edited

Pls let's discuss the syntax before we implement it, to make sure it's future proof and consistent with other commands. For example:

apify download-dataset [ID_OR_NAME] --format csv --limit 100 --offset 50 --output /tmp/file.csv ...

We can start with fewer options and add more later

@fnesveda
Copy link
Member

Let's check if we have some analytics which formats people use the most, but I think for starters it would be enough to support CSV, JSON and JSONL.

Also we could take the format transformation code that we have on the platform, put it in apify-shared and use it in CLI from there.

@fnesveda fnesveda added debt Code quality improvement or decrease of technical debt. medium priority Medium priority issues to be done in a couple of sprints. backend Issues related to the platform backend. labels Oct 11, 2023
@drobnikj drobnikj assigned drobnikj and unassigned drobnikj Oct 31, 2023
@jancurn
Copy link
Member

jancurn commented Mar 21, 2024

Once again we stumbled upon lack of this command, see https://apify.slack.com/archives/C06819MJN9Z/p1710923614723909?thread_ts=1710831604.067589&cid=C06819MJN9Z

@jirimoravcik jirimoravcik added t-tooling Issues with this label are in the ownership of the tooling team. and removed t-platform Issues with this label are in the ownership of the platform team. labels Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vladfrangu vladfrangu

Labels
backend Issues related to the platform backend. debt Code quality improvement or decrease of technical debt. medium priority Medium priority issues to be done in a couple of sprints. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jirimoravcik @fnesveda @jancurn @drobnikj @vladfrangu