Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redis data persistence #248

Open
MRPTY1 opened this issue Apr 11, 2022 · 4 comments
Open

Redis data persistence #248

MRPTY1 opened this issue Apr 11, 2022 · 4 comments
Labels
documentation

Comments

@MRPTY1
Copy link

MRPTY1 commented Apr 11, 2022

I hope to add a function of persistence of fingerprint in scrapy-redis after the end of the crawl.
@songhao8080
Copy link
Contributor

a function of persistence of fingerprint in scrapy-redis

set SCHEDULER_PERSIST = True in scrapy settings.py , It will works.

@MRPTY1
Copy link
Author

MRPTY1 commented Apr 13, 2022

I prefer to store in a database or file,there are many data in Redis, but I don't want to delete them. If you read the fingerprint file before running, then store it again, this is very convenient.

@LuckyPigeon
Copy link
Collaborator

@MRPTY1
Sorry for late reply. Pull Requests are welcome! Or we will schedule it to our TODO list.

@LuckyPigeon
Copy link
Collaborator

LuckyPigeon commented Sep 17, 2022 •
edited

IMO, this more like a documentation feature. We could add exmaple relate to this. PRs are welcome!
For now you can add correspond data persistence code in close_spider of your item pipeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@songhao8080 @LuckyPigeon @MRPTY1