-
-
Notifications
You must be signed in to change notification settings - Fork 360
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
seems like quinn 0.11 not working well under heavy load #1867
Comments
quinn-proto 0.11.1 was released on crates.io 9 days ago. Are you using it? This isn't an actionable report. What is the specific behavior? Do you have a reproducible test case? |
Oh, I went to crates.io, I saw quinn is 0.11, I thought quinn-proto is the same version. I am still trying to find a way to prove it's the stream, but not yet, maybe it's my own problem. |
What exactly does "got stuck" mean? Is the sender unable to write data to a stream? Is the receiver unable to read data that was successfully written? Are other functions of the connection degraded in any way? There have been some reports of stream flow control issues in #1818; I wonder if that might be related. If this is a flow control issue, then you should see all previously written data successfully received, but an inability to write new data. You can track this by logging the total number of bytes written to/read from the stream in question.
Can you run your workload many times concurrently to deliberately trigger the behavior more frequently? |
Some interesting internal Quinn state you could try to capture when your application stops making progress:
Receive side:
|
yes, it seems a flow-control problem. It's working fine, and suddenly, it cant' write new data. The write_all got stuck. |
It seems these infos are not publicly available?
|
I did a test, I am creating a VPN. sending packet through Quic. It still hangs from time to time, I still can't find out why yet. But I can be pretty sure it's because the data can't be sent somehow. Not only can't be sent, but some block the flow. i.e, write_all().await got stuck. Very tough to re-produce. will continue to watch. |
Yes, they are internal Quinn state. You can use a modified version of Quinn to insert whatever logging or getters you like. |
If using short-lived streams fails much more often, can you build a test case using that pattern? If you're observing the same behavior when using short-lived streams, it is much less likely to be a flow control issue.
That parameter governs concurrency. It will not cause your application to hang unless your application is incorrect. In most cases, you should be able to set it to 1 and have no adverse effects beyond degraded throughput. |
Is there a way to require the stream to send data and receive ACK in a certain time frame? If it does not get the ack back in time, the stream invalidate the connection? I think what I am looking for is a "ACK timeout" setting on transport config. Is it possible?
|
The health of a connection is independent of the state of an individual stream. If a connection is healthy, then so are its streams. If a peer stops responding, the connection will time out according to the idle timeout. |
Did you root-cause your issue? Is there something we could document better to avoid similar issues in the future? |
I can not be sure. But my debug tells me it only can be quinn problem. :-)
I am not sure if I hit the bug you guys fixed in 0.11.1. Since you didn't release to crates.io, not sure how to use 0.11.1.
Anyway, seems the latest release not as stable as 0.10. But I could be wrong! It seems the data got stuck under heavy load, can't be sent on the stream.
The text was updated successfully, but these errors were encountered: