Add special "session" option to RequestOptions & EnqueueLinksOptions or session/cookie sharing for new requests

[WebStrider]

Hi, great framework but i've found some kind of not obvious ideas behind of its architecture (or this is just my personal misunderstanding)

so, is it possible (and how) to use only one session (with its cookies, selected user-agent, userData etc) which has been initialized and settled updated after previous response for all the following requests i'll enqueue in requests handlers in crawling process?

as i could understand the BasicCrawler creates new session (or picks any random) for each new request from RequestQueue, and this is not accepted behaviuor in case if we have to use some preconditions for next requests (auth and some other monitoring & tracking cookies, selected User-Agent, all the other headers, previously selected and binded "fingerprints" with the session etc)

so, i'd like to ask to add this important feature - ability to reuse current session in new requests.

IMO the most handy way to do it - provide session option for RequestOptions & EnqueueLinksOptions arguments, for example:

// some code...

const router = Router.create<CheerioCrawlingContext>()

/****************************
 *  Adding Route Handlers 
 *************************** */
router.addHandler("category1.index", async (ctx) => {
	const cookies_for_this_session = ctx.session.getCookieString(ctx.request.loadedUrl)
	ctx.log.info(`Cookies after http://example.com/category1/ response: ${cookies_for_this_session}`)
	
	ctx.session.userData.cat1_data = "some_data_from_category1_index"
	
	const parsed_link_from_cat1 = ctx.$("...")

	ctx.enqueueLinks({
		label: 'category1.subcat1',
		// i need to use the same session with already received cookies (and all headers, "fingerprints" for sure), proxy(or just static IP) for this new request
		session: ctx.session,

		urls: [parsed_link_from_cat1],

		// some other...
	})
})

router.addHandler("category2.index", async (ctx) => {
	const cookies_for_this_session = ctx.session.getCookieString(ctx.request.loadedUrl)
	ctx.log.info(`Cookies after http://example.com/category2/ response: ${cookies_for_this_session}`)

	ctx.session.userData.cat2_data = "some_data_from_category2_index"
	
	const parsed_link_from_cat2 = ctx.$("...")

	ctx.enqueueLinks({
		label: "category2.subcat1",
		urls: [parsed_link_from_cat2]
		selector: parsed_link_from_cat2,
		// no explicitly provided session for this new request so BasicCrawler._runTaskFunction will ask SessionPool for new clean session with no cookies, userData, choosen user-agent etc
	})
})

router.addHandler("category1.subcat1", async (ctx) => {
	const cookies_for_this_session = ctx.session.getCookieString(ctx.request.loadedUrl)
	ctx.log.info(`Cookies after http://example.com/category1/subcategory1/ response: ${cookies_for_this_session}`)

	// should print "some_data_from_category1_index"
	ctx.log.info("userData for first session:", ctx.session.userData.cat1_data)
})

router.addHandler("category1.subcat2", async (ctx) => {
	// should print undefined (no explicit "session" for this request been provided)
	ctx.log.info("userData for first session:", ctx.session.userData.cat2_data)
})

//////////////////////
// And in the similar way should work with crawler.addRequests (RequestOptions):
//////////////////////
router.addHandler("category1.index", async (ctx) => {
	const cookies_for_this_session = ctx.session.getCookieString(ctx.request.loadedUrl)
	ctx.log.info(`Cookies after http://example.com/category1/ response: ${cookies_for_this_session}`)

	ctx.session.userData.cat1_data = "some_data_from_category1_index"

	const parsed_link_from_cat1 = ctx.$("...")

	ctx.crawler.addRequests([{
		label: "category1.subcat1",
		session: ctx.session,
		url: parsed_link_from_cat1,
	}])
)

/****************************
 *  Create Crawlee and enqueue initial URL (as usual)
 *************************** */
const crawler = new CheerioCrawler({
	requestHandler: router,
	ignoreSslErrors: true,
	
	useSessionPool: true,
	persistCookiesPerSession: true,
	sessionPoolOptions: {
		maxPoolSize: 25,
	}
}, {})

await crawler.addRequests([
	{
		label: 'category1.index',
		url: 'http://example.com/category1/',
	}
	{
		label: 'category2.index',
		url: 'http://example.com/category2/',
	}
])

const finalStats = await crawler.run()

here we explicitly use the received session from previous response to enqueue new links (with ctx.enqueueLinks or crawler.addRequest) to fetch.

Thanks!

[metalwarrior665]

Thanks a lot. I definitely agree that SessionPool needs to get more flexible and customizable, especially in assigning sessions to specific requests. But we should do a bigger think through since there was no big update since the original implementation. Here are some other ideas - #1573

[metalwarrior665]

Your idea has a potential problem: What would happen when after you enqueue the session, it is used is some other context and it is retired before the new request gets processed?

[WebStrider]

well this is personal user's responsibility to explicitly manage provided sessions for new requests. this also means that the session being once explicitly specified for new request shouldn't be provided by SessionPool.getSession() with no sessionid param.

From the other hand concurrency with this kind of explicitly specified sessions could be managed with FSM (state machine) logic using labels as states and session.userData as conditions defining new transitions.... and this behaviour should be implemented totally by the user who develops scraper but he has to have some referential and sticky data binded together in this way (FSM is just for example, the main goal is that the user/crawler dev should be responsible for all states changes etc)

[B4nan]

I will add to this, that the plan for next year is to have a user pool - you would specify how many user sessions you want to generate, and add some constraints around them (e.g. fingerprint options), that would then first generate all the virtual user sessions together with their fingerprints, and this pool would be then used instead.

I guess with this setup, the enqueueLinks (or even crawler.addRequests()) could respect the current user/session and reuse it. Indeed we should have a lower level control so you can pass the user/session in the request options, that makes total sense.

[WebStrider]

Thanks a lot! this one will be the greatest improvement !