user-agent header Issue resolved in headless mode

[michael2to3]

Issue with Setting User-Agent Header in Headless Mode

I've encountered an issue where setting the User-Agent header in headless mode was not effective, despite specifying it through both -H "User-Agent: 123" and the headless driver option --user-agent=123. The header would not change as expected. Below are the before and after screenshots for clarity:

Before:
This image demonstrates the simultaneous use of both methods, showing that the User-Agent header remained unchanged.

After:
This image shows that after my modifications, setting the header using -H "User-Agent: 123" alone is now sufficient to change the User-Agent.

Please review these changes. If there are any inaccuracies or if further adjustments are necessary, I'm open to feedback and willing to help enhance this functionality

[GeorginaReeder]

Thanks for your contribution @michael2to3 , we really appreciate it.

We also have a Discord server, which you’re more than welcome to join. It's a great place to connect with fellow contributors and stay updated with the latest developments!

[dogancanbakir]

I believe the issue can be resolved by removing this function completely, which overrides the user-agent in a few places

katana/pkg/utils/utils.go

Line 67 in 9ba3bb8

func WebUserAgent() string {

. Removing the function should solve the problem.

@Mzack9999, have we encountered any practical issues that were solved by overriding the user agent with this one?

[michael2to3]

Yeah that much better

[michael2to3]

the first part is with headless mode, the second without after deleting the Web User Agent. Will need to remove either only for headless mode, or leave the overwrite in crawl.go

[Mzack9999]

@dogancanbakir I believe the helper function was implemented in a few places to avoid using the go default user agent, which is filtered out in many cases