WSASetUdpRecvMaxCoalescedSize is not working !

[markyang59]

Describe the bug

Below is a sample case

1. packet size is set at 1420 bytes. send 6 packets 6 * 1420 = 8520 bytes sent
2. Coalescing size is set to 1420 * 10
3. Used IOCP for completion
4. IOCP completion returns an actual byteio of 14200 bytes in the ideal case.
   But the problem is that the actual returned byteio size fluctuates between 1420 ~ and 14200.
5. and at other PCs (Windows Server 2022) this value dropped to 1420 ~ 4260
6. The real problem is not receiving packets that are dropped! not in the buffer.

Affected OS

• Windows
• Linux
• macOS
• Other (specify below)

Additional OS information

For Windows 11 above symptom does not always
But for Windows Server 2022, it always happens

MsQuic version

main

Steps taken to reproduce bug

1. packet size is set at 1420 bytes. send 6 packets 6 * 1420 = 8520 bytes sent (with WSASetUdpSendMessageSize)
2. Coalescing size is set to 1420 * 10 (with WSASetUdpRecvMaxCoalescedSize)
3. Used IOCP for completion
4. IOCP completion returns an actual byteio of 14200 bytes in the ideal case.
   But the problem is that the actual returned byteio size fluctuates between 1420 ~ and 14200.
5. and at other PCs (Windows Server 2022) this value dropped to 1420 ~ 4260
6. The real problem is not receiving packets that are dropped! not in the buffer.

Expected behavior

IOCP completion returns coalesced buffer 14200 for all supported OSs

Actual outcome

Only works with windows 11

Additional details

No response

[nigriMSFT]

"The real problem is not receiving packets that are dropped! not in the buffer."
Could you explain what you mean with this statement?

Have you tried collecting logs? (I think you can use logs.ps1 with Full.Verbose profile)

In general, RX packets are coalesced opportunistically. It is not a guarantee that when WSASetUdpRecvMaxCoalescedSize is called all RX packets are coalesced to the maximum size.

[markyang59]

Dear. Thank you for your kind reply. I need this tech now really. I am sending big files with UDP. For macOS and Linux. there are *sendmmsg* and *sendmsg_x* for this purpose. but for windows..I can't find a way to do it. and finally found this USO. my situation is like the one below 1. packet size is 1420 with WSASetUdpSendMessageSize(sock, 1420) 2. 6 packets sent at once (8520 bytes) 3. try to coalesce 10 packets with WSASetUdpRecvMaxCoalescedSize (1420 * 10) 4. Using IOCP completion 5. IOCP completion return coalesced recv bytes at (5) return value varying 1420 ~ 14200 but when the return value is smaller than 14200. rest packets are lost ! when the return value is 1420...9 packets are *disappeared!* *and* coalesced size varies system by system. at one PC size is 14200. but on another PC, the size is just 1420. Can I maintain coalesce size to fixed size ? and How could I collect log ? Could you give me a tip ? Thank you !

…

On Wed, Mar 13, 2024 at 2:13 AM Nick Grifka ***@***.***> wrote: "The real problem is not receiving packets that are dropped! not in the buffer." Could you explain what you mean with this statement? Have you tried collecting logs? (I think you can use logs.ps1 with Full.Verbose profile) In general, RX packets are coalesced opportunistically. It is not a guarantee that when WSASetUdpRecvMaxCoalescedSize is called all RX packets are coalesced to the maximum size. — Reply to this email directly, view it on GitHub <#4186 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAL5IPEBN5FF3FUMC4CD6DLYX4ZUVAVCNFSM6AAAAABEQCDBFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJSGE3DCNJYGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- I SWEAR BY MY LIFE AND MY LOVE OF IT THAT I WILL NEVER LIVE FOR THE SAKE OF ANOTHER MAN, NOR ASK ANOTHER MAN TO LIVE FOR MINE.

[nibanks]

Have you tried WSASendMsg? Take a look at how we use this in datapath_winuser.c.

[markyang59]

USO is only work with WSASendMsg / WSARecvMsg ???

…

On Thu, Mar 14, 2024 at 9:24 PM Nick Banks ***@***.***> wrote: Have you tried WSASendMsg? Take a look at how we use this in datapath_winuser.c <https://github.com/microsoft/msquic/blob/main/src/platform/datapath_winuser.c#L4170> . — Reply to this email directly, view it on GitHub <#4186 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAL5IPDFG3QYBVKQOA5OVELYYGJGDAVCNFSM6AAAAABEQCDBFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJXGMZDQNJZGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- I SWEAR BY MY LIFE AND MY LOVE OF IT THAT I WILL NEVER LIVE FOR THE SAKE OF ANOTHER MAN, NOR ASK ANOTHER MAN TO LIVE FOR MINE.

[nibanks]

USO (and URO) require ancillary data to be set, so the stack understands data is to be split among different packets. That is per-send information you must pass. I'm not sure if any other API may be used. @nigriMSFT to comment on that.

[markyang59]

I just used sendto function. send one big buffer which contains several packets. Maybe I used USO wrong way..and don't know how to set ancillary data to send. Could you show me example for USO send, recv ? I just tried WSASendMsg without control data. but it seems to set control data..but don"t know how. I SWEAR BY MY LIFE AND MY LOVE OF IT THAT I WILL NEVER LIVE FOR THE SAKE OF ANOTHER MAN, NOR ASK ANOTHER MAN TO LIVE FOR MINE. 2024년 3월 18일 (월) 20:23, Nick Banks ***@***.***>님이 작성:

…

USO (and URO) require ancillary data to be set, so the stack understands data is to be split among different packets. That is per-send information you must pass. I'm not sure if any other API may be used. @nigriMSFT <https://github.com/nigriMSFT> to comment on that. — Reply to this email directly, view it on GitHub <#4186 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAL5IPHQXEXLOYYMX2ILT73YY3FBZAVCNFSM6AAAAABEQCDBFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBTGY2TCOBUGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[nibanks]

You need to set the ancillary data. Please take a look at how our datapath_winuser.c does it.

[markyang59]

Thank you for your advice !! I found socket RCV_BUF setting seriously affects RECV. if the buffer size is small. packet dropped. Setting buffers size 128 MB, solved packet drops. but only 10 packets have coalesced even if I set a bigger max coales value. What other conditions affect coalescing? Could you give me a hint?

…

On Mon, Mar 18, 2024 at 10:29 PM Nick Banks ***@***.***> wrote: You need to set the ancillary data. Please take a look at how our datapath_winuser.c does it. — Reply to this email directly, view it on GitHub <#4186 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAL5IPGFD7HX73VV6QHOZH3YY3T2FAVCNFSM6AAAAABEQCDBFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBTHEYDOMJZGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- I SWEAR BY MY LIFE AND MY LOVE OF IT THAT I WILL NEVER LIVE FOR THE SAKE OF ANOTHER MAN, NOR ASK ANOTHER MAN TO LIVE FOR MINE.