Skip to content

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

License

Notifications You must be signed in to change notification settings

chrisakroyd/robots-txt-parser

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

80 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

robots-txt-parser

Coverage Status npm version npm

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Installing

Via NPM: npm install robots-txt-parser --save.

Getting Started

After installing robots-txt-parser it needs to be required and initialised:

const robotsParser = require('robots-txt-parser');
const robots = robotsParser(
  {
    userAgent: 'Googlebot', // The default user agent to use when looking for allow/disallow rules, if this agent isn't listed in the active robots.txt, we use *.
    allowOnNeutral: false // The value to use when the robots.txt rule's for allow and disallow are balanced on whether a link can be crawled.
  });

Example Usage:

const robotsParser = require('robots-txt-parser');

const robots = robotsParser(
  {
    userAgent: 'Googlebot', // The default user agent to use when looking for allow/disallow rules, if this agent isn't listed in the active robots.txt, we use *.
    allowOnNeutral: false, // The value to use when the robots.txt rule's for allow and disallow are balanced on whether a link can be crawled.
  },
);

robots.useRobotsFor('http://example.com')
  .then(() => {
    robots.canCrawlSync('http://example.com/news'); // Returns true if the link can be crawled, false if not.
    robots.canCrawl('http://example.com/news', (value) => {
      console.log('Crawlable: ', value);
    }); // Calls the callback with true if the link is crawlable, false if not.
    robots.canCrawl('http://example.com/news') // If no callback is provided, returns a promise which resolves with true if the link is crawlable, false if not.
      .then((value) => {
        console.log('Crawlable: ', value);
      });
  });

Condensed Documentation

Below is a condensed form of the documentation, each is a function that can be found on the robotsParser object.

Method Parameters Return
parseRobots(key, string) key:String, string:String None
isCached(domain) domain:String Boolean for whether robots.txt for url is cached.
fetch(url) url:String Promise, resolved when robots.txt retrieved.
useRobotsFor(url) url:String Promise, resolved when robots.txt is fetched.
canCrawl(url) url:String, callback:Func (Opt) Promise, resolves with Boolean.
getSitemaps() callback:Func (Opt) Promise if no callback provided, resolves with [String].
getCrawlDelay() callback:Func (Opt) Promise if no callback provided, resolves with Number.
getCrawlableLinks(links) links:[String], callback:Func (Opt) Promise if no callback provided, resolves with [String].
getPreferredHost() callback:Func (Opt) Promise if no callback provided, resolves with String.
setUserAgent(userAgent) userAgent:String None.
setAllowOnNeutral(allow) allow:Boolean None.
clearCache() None None.

Full Documentation


parseRobots

robots.parseRobots(key, string) Parses a string representation of a robots.txt file and cache's it with the given key.

Parameters
  • key -> Can be any URL.
  • string -> String representation of a robots.txt file.
Returns

None.

Example
robots.parseRobots('https://example.com',
  `
   User-agent: *
   Allow: /*.php$
   Disallow: /
  `);

isCached

robots.isCached(domain) A method used to check if a robots.txt has already been fetched and parsed.

Parameters
  • domain -> Can be any URL.
Returns

Returns true if a robots.txt has already been fetched and cached by the robots-txt-parser.

Example
robots.isCached('https://example.com'); // true or false
robots.isCached('example.com'); // Attempts to check the cache for only http:// and returns true or false.

fetch

robots.fetch(url) Attempts to fetch and parse a robots.txt file located at the url, this method avoids checking the built-in cache and will always attempt to retrieve a fresh copy of the robots.txt.

Parameters
  • url -> Any URL.
Returns

Returns a Promise which will resolve once the robots.txt has been fetched with the parsed robots.txt.

Example
robots.fetch('https://example.com/robots.txt')
    .then((tree) => {
        console.log(Object.keys(tree)); // Will log sitemap and any user agents.
    });

useRobotsFor

robots.useRobotsFor(url) Attempts to download and use the robots.txt at the given url, if the robots.txt has already been downloaded, reads from the cached copy instead.

Parameters
  • url -> Any URL.
Returns

Returns a promsise that resolves once the URL is fetched and parsed.

Example
robots.useRobotsFor('https://example.com/news')
    .then(() => {
        // Logic to check if links are crawlable.
    });

canCrawl

robots.canCrawl(url, callback) Tests whether a url can be crawled for the current active robots.txt and user agent. If a robots.txt isn't cached for the domain of the url, it is fetched and parsed before returning a boolean value.

Parameters
  • url -> Any URL.
  • callback -> An optional callback, if undefined returns a promise.
Returns

Returns a Promise which will resolve with a boolean value.

Example
robots.canCrawl('https://example.com/news')
    .then((crawlable) => {
        console.log(crawlable); // Will log a boolean value.
    });

getSitemaps

robots.getSitemaps(callback) Returns a list of sitemaps present on the active robots.txt.

Parameters
  • callback -> An optional callback, if undefined returns a promise.
Returns

Returns a Promise which will resolve with an array of strings.

Example
robots.getSitemaps()
    .then((sitemaps) => {
        console.log(sitemaps); // Will log an list of strings.
    });

getCrawlDelay

robots.getCrawlDelay(callback) Returns the crawl delay on requests to the current active robots.txt.

Parameters
  • callback -> An optional callback, if undefined returns a promise.
Returns

Returns a Promise which will resolve with an Integer.

Example
robots.getCrawlDelay()
    .then((crawlDelay) => {
        console.log(crawlDelay); // Will be an Integer greater than or equal to 0.
    });

getCrawlableLinks

robots.getCrawlableLinks(links, callback) Takes an array of links and returns an array of links which are crawlable for the current active robots.txt.

Parameters
  • links -> An array of links to check for crawlability.
  • callback -> An optional callback, if undefined returns a promise.
Returns

A Promise that will resolve to contain an Array of all the crawlable links.

Example
robots.getCrawlableLinks([])
    .then((links) => {
        console.log(links);
    });

getPreferredHost

robots.getPreferredHost(callback) Returns the preferred host name specified in the active robots.txt's host: directive or null if there isn't one.

Parameters
  • callback -> An optional callback, if undefined returns a promise.
Returns

An String if the host is defined, undefined otherwise.

Example
robots.getPreferredHost()
    .then((host) => {
        console.log(host);
    });

setUserAgent

robots.setUserAgent(userAgent) Sets the current user agent to use when checking if a link can be crawled.

Parameters
  • userAgent -> A string.
Returns

undefined

Example
robots.setUserAgent('exampleBot'); // When interacting with the robots.txt we now look for records for 'exampleBot'.
robots.setUserAgent('testBot'); // When interacting with the robots.txt we now look for records for 'testBot'.

setAllowOnNeutral

robots.setAllowOnNeutral(allow) Sets the canCrawl behaviour to return true or false when the robots.txt rules are balanced on whether a link should be crawled or not.

Parameters
  • allow -> A boolean value.
Returns

undefined

Example
robots.setAllowOnNeutral(true); // If the allow/disallow rules are balanced, canCrawl returns true.
robots.setAllowOnNeutral(false); // If the allow/disallow rules are balanced, canCrawl returns false.

clearCache

robots.clearCache() The cache can get extremely long over extended crawling, this simple method resets the cache.

Parameters

None

Returns

None

Example
robots.clearCache();

Synchronous API

Synchronous variants of the API, will be deprecated in a future version.

canCrawlSync

robots.canCrawlSync(url) Tests whether a url can be crawled for the current active robots.txt and user agent. This won't attempt to fetch the robots.txt if it is not cached.

Parameters
  • url -> Any url.
Returns

Returns a boolean value depending on whether the url is crawlable. If there is no cached robots.txt for this url, it will always return true.

Example
robots.canCrawlSync('https://example.com/news') // true or false.

getSitemapsSync()

robots.getSitemapsSync() Returns a list of sitemaps present on the active robots.txt.

Parameters

None

Returns

An Array of Strings.

Example
robots.getSitemapsSync(); // Will be an array e.g. ['http://example.com/sitemap1.xml', 'http://example.com/sitemap2.xml'].

getCrawlDelaySync()

robots.getCrawlDelaySync() Returns the crawl delay on specified in the active robots.txt's for the active user agent

Parameters

None

Returns

An Integer greater than or equal to 0.

Example
robots.getCrawlDelaySync(); // Will be an Integer.

getCrawlableLinksSync

robots.getCrawlableLinksSync(links) Takes an array of links and returns an array of links which are crawlable for the current active robots.txt.

Parameters
  • links -> An array of links to check for crawlability.
Returns

An Array of all the links are crawlable.

Example
robots.getCrawlableLinks(['example.com/test/news', 'example.com/test/news/article']);  // Will return an array of the links that can be crawled.

getPreferredHostSync

robots.getPreferredHostSync() Returns the preferred host name specified in the active robots.txt's host: directive or undefined if there isn't one.

Parameters

None

Returns

An String if the host is defined, undefined otherwise.

Example
robots.getPreferredHostSync(); // Will be a string if the host directive is defined .

License

See LICENSE file.

Resources

About

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published