Rate Limit Minute (-rlm) behavior does not distribute requests evenly

[swdbo]

httpx version:

v1.6.0

Current Behavior:

When using the -rlm (rate limit per minute) argument, the tool sends the specified amount of requests all at once instead of distributing them evenly across the minute. This behavior is counterintuitive, as it leads to a burst of traffic followed by a period of inactivity, rather than spreading the requests out to avoid overwhelming target servers.

Expected Behavior:

The expected behavior for the -rlm argument is to distribute the specified number of requests evenly across the minute. For instance, if -rlm 10 is specified, one would expect a request to be sent every 6 seconds, thereby evenly pacing the load on the target server(s) and adhering more closely to a "rate limit."

Steps To Reproduce:

1. Prepare a urls.txt file with multiple target URLs.
2. Run httpx with verbose logging and the -rlm argument set to 10, like so: cat urls.txt | httpx -v -rlm 10
3. Observe the output and timing of requests; all 10 requests are made at the same time, not spaced out at 1 request every 6 seconds as expected.

Anything else:

This unexpected behavior could lead to potential flooding of target websites, which is especially concerning in scenarios where careful rate limiting is necessary to comply with target server policies or to avoid unintentional Denial-of-Service conditions. An adjustment to ensure requests are distributed evenly throughout the specified time frame would greatly enhance the utility and reliability of the -rlm feature.

[Mzack9999]

@swdbo we opted for a rate limit with burst at maximum speed capability as httpx is mostly meant to interact with many different servers. Hence the general usage is interacting with targets ideally only once per port at maximum speed. The rate limit is mostly meant to impose a generic very large upper bound, and use maximum concurrency of connections towards different servers. Did you face any scenario where you experienced idle period of inactivity or overwhelmed the servers? I'm just trying to understand the context better, so that we could change the internals to address common use cases. Thanks!

[Mzack9999]

Closing due to inactivity - Feel free to reopen