Add preresolving feature

[rom1504]

It was recently noticed that laion 400m only contains urls from 5M domains.
The same is probably true for other datasets.

Pre-resolving the domains would decrease the charge on the dns process by a lot and increase downloading speed.

[rom1504]

would need to handle dns load balancing properly

[rom1504]

tried https://stackoverflow.com/a/15065711/1658314 with no much success

[rom1504]

This approach seems to work very well:

1. get unique domains
2. configure knot resolver to have a very long TTL
3. use dnsperf to fetch all the domains
4. run img2dataset

dnsperf -f inet -s 10.80.97.250 -d /tmp/list.txt -l 3600 -S 10 -Q 1000 -q 100 2>&1 | grep -v Timeout | grep -v "maybe timed out"

https://github.com/DNS-OARC/dnsperf/blob/master/README.md

I think that is pretty promising and it may be interesting to try and put that directly in img2dataset (at least the dnsperf part)

[rom1504]

https://knot-resolver.readthedocs.io/en/stable/daemon-scripting.html

[rom1504]

https://knot-resolver.readthedocs.io/en/stable/modules-stats.html#built-in-statistics