Connection between Leafnode and Core NATS over satellite link fails to get established

[piotrgalecki]

Observed behavior

Siden is running into an odd issue with NATS Leafnode not being able to connect to our Core NATS server.
After TCP 3-way handshake Core NATS sends INFO message and then NATS Leafnode responds with INFO and 400+ms later sends closes connection (sends TCP FIN).
The latency of connection is 500ms.

Here are packets exchanged. Packets with P flag are INFO messages, followed by FIN packet

22:00:32.440826 IP 10.99.0.99.34786 > 3.136.10.169.7422: Flags [S], seq 1607668652, win 65535, options [mss 1460,sackOK,TS val 2932985413 ecr 0,nop,wscale 8], length 0
22:00:32.980351 IP 3.136.10.169.7422 > 10.99.0.99.34786: Flags [S.], seq 2383013068, ack 1607668653, win 65535, options [mss 1460,sackOK,TS val 1779976004 ecr 2932985413,nop,wscale 8], length 0
22:00:32.980410 IP 10.99.0.99.34786 > 3.136.10.169.7422: Flags [.], ack 1, win 256, options [nop,nop,TS val 2932985953 ecr 1779976004], length 0
22:00:33.520223 IP 3.136.10.169.7422 > 10.99.0.99.34786: Flags [P.], seq 1:468, ack 1, win 256, options [nop,nop,TS val 1779976544 ecr 2932985953], length 467
22:00:33.520276 IP 10.99.0.99.34786 > 3.136.10.169.7422: Flags [.], ack 468, win 261, options [nop,nop,TS val 2932986493 ecr 1779976544], length 0
22:00:33.520719 IP 10.99.0.99.34786 > 3.136.10.169.7422: Flags [P.], seq 1:164, ack 468, win 261, options [nop,nop,TS val 2932986493 ecr 1779976544], length 163
22:00:33.981008 IP 10.99.0.99.34786 > 3.136.10.169.7422: Flags [F.], seq 164, ack 468, win 261, options [nop,nop,TS val 2932986954 ecr 1779976544], length 0

in the console the NATS Leafnode outputs the following:
[7] [INF] 3.13.54.170:7422 - lid:53 - Leafnode connection closed: Read Error - Account: $G
[7] [INF] 3.13.54.170:7422 - lid:54 - Leafnode connection created for account: $G
[7] [INF] 3.13.54.170:7422 - lid:54 - Leafnode connection closed: Read Error - Account: $G

After investigation the issue is caused by too short timeout DEFAULT_LEAFNODE_INFO_WAIT, which is hardcoded to 1s.
Unfortunately it's a constant and not configurable,
We changed the timeout value in the vendor directory to 3s and that resolved the issue.

Please make this parameter configurable.

Expected behavior

NATS connection can be established successfully

Server and client version

v2.10.10

Host environment

No response

Steps to reproduce

Test NATS Leafnode - Server connection over high latency link.