Web Crawler Stopping Issue

[dwk601]

I'm trying to crawl the website by using the feature in the app, but it kept stopping even the max links is set to over 100. I've even deleted and reset the project, but kept stopping in a random tasks, first time it was stopped at somewhere 60 and this top somewhere 20.
Is there any other way to automate the web crawling, and feed to the ai model.

Then after that bug, you cannot use the web crawler again. Even you create a new chatbot it just says complete load data but in the data there's nothing there.

[n4ze3m]

Hey, The web crawler is still in beta and has bugs; it will be fixed in upcoming updates

[dwk601]

Okay thanks for letting me know

…

________________________________ From: Muhammed Nazeem ***@***.***> Sent: Wednesday, January 3, 2024 5:47:55 AM To: n4ze3m/dialoqbase ***@***.***> Cc: Dongwook Kim ***@***.***>; Author ***@***.***> Subject: Re: [n4ze3m/dialoqbase] Web Crawler Stopping Issue (Issue #185) Hey, The web crawler is still in beta and has bugs; it will be fixed in upcoming updates — Reply to this email directly, view it on GitHub<#185 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZYNBTANJ45JFOVCILMBLFTYMVHPXAVCNFSM6AAAAABBKZUE26VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVGMZDCMZQHE>. You are receiving this because you authored the thread.Message ID: ***@***.***>