Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Requesting max_items in scrapyrt #107

Open
xaander1 opened this issue May 5, 2020 · 2 comments · May be fixed by #129
Open

Requesting max_items in scrapyrt #107

xaander1 opened this issue May 5, 2020 · 2 comments · May be fixed by #129
Labels
enhancement help wanted this is easy to fix issue that is usually beginner friendly

Comments

@xaander1
Copy link

xaander1 commented May 5, 2020

max_items like the max_requests argument would be really helpful.
@pawelmhm pawelmhm added enhancement help wanted this is easy to fix issue that is usually beginner friendly labels Jan 29, 2021
@pawelmhm
Copy link
Member

To do this look up how max_requests are done now: https://github.com/scrapinghub/scrapyrt/blob/master/scrapyrt/core.py#L99 and add it in a similar way, need to count items collected somewhere here: https://github.com/scrapinghub/scrapyrt/blob/master/scrapyrt/core.py#L207 and close spider if it exceeds limits.

@pawelmhm
Copy link
Member

assigned to @serhii73

@serhii73 serhii73 linked a pull request Jun 1, 2021 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement help wanted this is easy to fix issue that is usually beginner friendly
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[WIP] Add max_items serhii73/scrapyrt
2 participants
@pawelmhm @xaander1