Add support for all tags defined by the sitemap protocol

[simonbrunel]

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/utils

Feature

Loading sitemaps using Sitemap.load() should give access to the other tags defined by the Sitemaps format: loc, lastmod, changefreq and priority.

Motivation

Sitemaps give information about when each page has been last modified, priority, etc... and while I'm sure there are other libraries to load sitemaps, it's easier to rely on the Crawlee utils instead (consistency and less dependencies). Since you already provide a Sitemap util, it should be relatively easy to expose other tags other than the url.

Ideal solution or implementation, and any additional constraints

export type SitemapEntry = {
  changefreq?: 'always' | 'hourly' | 'daily' ...,
  lastmod?: ISODate;
  priority?: number;
  url: string;
}

class Sitemap {
  readonly urls: string[]; // deprecated (maybe)

  constructor(readonly entries: SitemapEntry[]) {
    this.urls = entries.map((d) => d.url);
  }
}

// Usage
const sitemap = await Sitemap.load(...);
for (const entry of sitemap.entries) {
   console.log(entry.url, entry.lastmod, ...);
}

Alternative solutions or implementations

No response

Other context

No response