How do I correctly configure crawlers to run simultaneously? #1240
Unanswered
windbridges
asked this question in
Q&A
Replies: 1 comment
-
yes, all of them will write to the same file without doing any lock and/or using a semaphore. you can solve this by creating different key names for this by changing the |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I'm using the Apify SDK inside a console application that has multiple crawlers running simultaneously inside the same process. At first glance, everything works fine. But sometimes crashes like:
Unexpected token { in JSON at position 430641
orUnexpected end of JSON input
.I guess the problem is that all crawlers store their session pool state in the same storage ([storage]/key_value_stores/default/SDK_SESSION_POOL_STATE.json) because I don't change their default settings. Because of the simultaneous access the data in the file sometimes overlaps, the format integrity is broken and the next time it is read it fails. This also happens again if I try to rerun the application. If I manually delete this json file, the error disappears, so it is about this file.
Can you please suggest ways to solve this problem.
Beta Was this translation helpful? Give feedback.
All reactions