crawlee's uniqueKey generator does not treat hash path as complete URL

[shuiRong]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/core

Issue description

I dynamically generated a batch of URLs for the puppeteer crawler, but the crawler ended the program immediately, showing that the number of completed task was 1.

This was very unusual, and it took me a while to figure out that the default uniqueKey generator might have generated the same key for this batch of URLs.

"https://heuristic-stonebraker-33e1b8.netlify.app/#/topic/57d1c1b63d3520a5387c2b49"
"https://heuristic-stonebraker-33e1b8.netlify.app/#/user/lijunray"

this.uniqueKey = uniqueKey || this._computeUniqueKey({ url, method, payload, keepUrlFragment, useExtendedUniqueKey });

I found the location of the generated program, but I haven't had a chance to debug where the problem is

Code sample

const links: RequestOptions[] = [];
  pageHrefList.forEach((href) => {
    const link = new URL(href, loadedUrl);
    if (link.hostname === hostname) {
      if (!findCrawledURL(encodeEntryURL, link.href)) {
        links.push({
          url: link.href,
          userData: {
            encodeEntryURL,
          },
          uniqueKey: link.href,
        });
      }
    }
  });

  await crawler.addRequests(links);

I solved this problem in my project by passing in a uniqueKey, and felt the need to raise an issue at the time

Package version

3.3.3

Node.js version

v19.8.1

Operating system

macos

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

This is on purpose, you can use Request.keepUrlFragment to respect them.

https://crawlee.dev/api/core/interface/RequestOptions#keepUrlFragment