Library usage MaxDepth issues

[leonjza]

katana version:

cli: version 1.02
go.mod line: github.com/projectdiscovery/katana v1.0.2

Current Behaviour:

Using katana as a library appears to always behave as if the scope depth was set to 1. In the below example, I have configured MaxDepth to be 2.

❯ go run main.go
[INF] Started standard crawling for => https://tesla.com
https://tesla.com
[INF] https://tesla.com

The source code used is available in this Gist, though it doesn't differ from the example in the README: https://gist.github.com/leonjza/82f7b89df1abe7b4a768349313a58920

Using the command line program however, the output appears as expected.

❯ katana -u https://tesla.com -d 2
[ ... snip logo ...]

[INF] Current katana version v1.0.2 (latest)
[INF] Started standard crawling for => https://tesla.com
https://www.tesla.com/
https://tesla.com
https://www.tesla.com/modules/custom/tesla_banners/js/index.js?v=1.x
https://www.tesla.com/themes/custom/tesla_frontend/assets/manifest.json
https://www.tesla.com/_flysystem/s3/css/css_nUTC6NHVOqULHggJlfgzEQ2EtlOWuU94ec0FiBsGsOg.css
https://www.tesla.com/_flysystem/s3/css/css_xW-cg_AzHoBA7lMpqERJ5452cKtnXx8j_vFdYDwrYSk.css
https://www.tesla.com/_flysystem/s3/css/css_WgCBnkNyCBuMX6gyOoWzsDjeC_de6DH55PdxvAKPnls.css

[... snip ...]

Expected Behaviour:

When using katana as a library, crawling occurs with the same (or similar) results as the command line client.

Steps To Reproduce:

1. Integrate katana as the example in the README.
2. Run your custom program and observe what appears to be the MaxDepth field being ignored.

[Mzack9999]

@leonjza I've tried to run the code snippet with the dev branch, and the issue seems fixed:

$ go run .
[INF] Started standard crawling for => https://tesla.com
https://www.tesla.com/
[INF] https://www.tesla.com/
https://tesla.com
[INF] https://tesla.com
[INF] https://www.tesla.com/themes/custom/tesla_frontend/assets/favicons/favicon-32x32.png
https://www.tesla.com/modules/custom/tesla_banners/js/index.js?v=1.x

If you checkout the dev branch in your project until a new release is created, you should be able to perform crawling at the wanted nesting level:

go install github.com/projectdiscovery/katana/cmd/katana@dev