New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WebSite dataource loader: Blacklist urls #330

Open

gmpetrov opened this issue Feb 15, 2024 · 0 comments

Assignees

Owner

gmpetrov commented Feb 15, 2024

Customers want to be able to specify a list of URLs that should not be processed by the crawler

in the WebSite datasource form, add a textarea input called Blacklist (each line is one URL)
check if easy to implement globbing or regex e.g. https://domain.com/products/*

gmpetrov assigned OdapX

OdapX mentioned this issue

feat: 🎸 blacklist urls from website loader #333

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment