requestHandler and postNavigationHooks seem redundant

[matjaeck]

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/browser (BrowserCrawler)

Feature

Providing requestHandler is required but IMO it is redundant with what postNavigationHooks are there for. The semantics are IMO also rather confusing, as the function does not handle network requests but rather navigation.

Replace requestHandler(ctx) with new navigationHandler(ctx, gotoOptions) and fix confusing request-handling semantic when navigation-handling is meant.

Thoughts? What can you do in a user defined requestHandler that you cannot do in user defined post navigation hooks?

Motivation

Fix #1631.
Reduce confusion.

Ideal solution or implementation, and any additional constraints

not sure, needs feedback

Alternative solutions or implementations

not sure, needs feedback

Other context

No response

[B4nan]

Thanks for the feedback, but requestHandler is an integral part of the crawler design and its life cycle, we are not interested in changing this, nor in renaming it (maybe we can reiterate on that in a year or so, but a new major version is out of scope for discussions at this point). If you don't need it, just pass in an empty callback like () => {}.

Providing requestHandler is required but IMO it is redundant with what postNavigationHooks are there for.

All hooks are optional, so requestHandler can be redundant only if you actually use them. Changing this (making some hooks required) would be a huge breaking change and only cause more confusion. Removing the concept of requestHandler would be equally huge breaking change, for no clear reason.

[matjaeck]

Hi, alright, thanks for the response.

All hooks are optional, so requestHandler can be redundant only if you actually use them.

This is not really a counter argument to that nothing stops you from moving all crawling logic from post hooks to requestHandler, i.e. they are used in the same way. This equals redundancy in my book. But I get your point about breaking changes, hence the request for feedback.

Sidenote: I liked the original "handlePageFuntion" thingy a lot more, as it does what the name says. Handling requests is not what requestHandler does, as it handles a crawling context, in most use cases after requests have been loaded and handled. I understand that in your view "request" equals "navigation request" (same with request queue). It just a little bit a preoccupied term :)

Removing the concept of requestHandler would be equally huge breaking change, for no clear reason.

Totally understandable that you don't want to introduce breaking changes in a hurry.
If you like redundant features and don't care about misleading naming, then I agree there's no reason for a change :)