TaskCanceledException exceptions occurring randomly

[jasidhu]

Describe the bug

After deploying YARP, in our production environment we are seeing random exceptions as seen below. Most requests are successful, but a small portion (~1%) of them are having this issue.

To Reproduce

I am unable to reproduce the issue in the test environments. I suspected that the request was timing out because most of the exception were thrown at approx. 1.8mins, so I have set the Activity Timeout in proxy config and HttpClient timeout on client application to 10 minutes, but it didn't resolve the issue and proxy still randomly throws these exceptions.

Error message: The request was canceled before receiving a response.

System.Threading.Tasks.TaskCanceledException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Threading.Tasks.Sources.ManualResetValueTaskSourceCore`1.ThrowForFailedGetResult (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.Http2Connection+Http2Stream.System.Threading.Tasks.Sources.IValueTaskSource.GetResult (System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Net.Http.Http2Connection+Http2Stream+<ReadResponseHeadersAsync>d__73.MoveNext (System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.Http2Connection+<SendAsync>d__119.MoveNext (System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__86.MoveNext (System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.DiagnosticsHandler+<SendAsyncCore>d__10.MoveNext (System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Yarp.ReverseProxy.Forwarder.HttpForwarder+<SendAsync>d__8.MoveNext (Yarp.ReverseProxy, Version=2.0.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)

or

Error message: The operation was canceled.

System.Threading.Tasks.TaskCanceledException:
   at System.Net.Http.HttpConnection+<SendAsyncCore>d__61.MoveNext (System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__86.MoveNext (System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.DiagnosticsHandler+<SendAsyncCore>d__10.MoveNext (System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Yarp.ReverseProxy.Forwarder.HttpForwarder+<SendAsync>d__8.MoveNext (Yarp.ReverseProxy, Version=2.0.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.IO.IOException handled at System.Net.Http.HttpConnection+<SendAsyncCore>d__61.MoveNext:
   at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException (System.Net.Sockets, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult (System.Net.Sockets, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__157`1.MoveNext (System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1+StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Security.SslStream+<ReadAsyncInternal>d__159`1.MoveNext (System.Net.Security, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1+StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnection+<InitialFillAsync>d__98.MoveNext (System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Net.Http.HttpConnection+<SendAsyncCore>d__61.MoveNext (System.Net.Http, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.Net.Sockets.SocketException handled at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException:

Further technical details

The service is running on a Windows Azure App Service using Yarp.ReverseProxy 2.0.1

The config is below:

  "ReverseProxy": {
    "Routes": {
      "route1": {
        "ClusterId": "cluster1",
        "Match": {
          "Methods": [ "GET" ],
          "Path": "/_api/v1/test/{**remainder}"
        }
      }
    },
    "Clusters": {
      "cluster1": {
        "Destinations": {
          "production": {
            "Address": "https://example.com"
          }
        },
        "HttpRequest": {
          "ActivityTimeout": "00:10:00"
        }
      }
    }
  }

[MihaZupan]

The most likely reason for cancellations is that the client disconnected/canceled its request to the proxy.

You can check whether that happened by looking at the httpContext.RequestAborted.IsCancellationRequested after seeing the error.

(If the client talking to the proxy is an HttpClient, note that 1.8 min is roughly ~100 seconds, which happens to be the default HttpClient timeout)

[jasidhu]

The most likely reason for cancellations is that the client disconnected/canceled its request to the proxy.

You can check whether that happened by looking at the httpContext.RequestAborted.IsCancellationRequested after seeing the error.

(If the client talking to the proxy is an HttpClient, note that 1.8 min is roughly ~100 seconds, which happens to be the default HttpClient timeout)

@MihaZupan Thanks for replying. I can add the httpContext.RequestAborted.IsCancellationRequested check in the proxy but just want to mention again that we have set the HttpClient timeout to 10minutes and still seeing this exception and not all the exceptions occurred at 1.8mins marks. We have noticed same exceptions occurring anywhere between 2min to 4min mark as well.

Couple of questions:

1. Is there anything in the proxy which can trigger Request Cancellation?
2. Apart from the httpContext.RequestAborted.IsCancellationRequested check is there anything else which can confirm that indeed client has disconnected/canceled its request to the proxy.

[MihaZupan]

Is there anything in the proxy which can trigger Request Cancellation?

The things that can currently trigger a cancellation are

• The httpContext.RequestAborted CancellationToken
• The CancellationToken passed to IHttpForwarder.SendAsync manually (none by default)
• The activity timeout (which you've set to 10 minutes already)
• Features like the request timeouts middleware in ASP.NET Core, but that would show up as the httpContext.RequestAborted being signaled

Apart from the httpContext.RequestAborted.IsCancellationRequested check is there anything else which can confirm that indeed client has disconnected/canceled its request to the proxy.

You could double-check debug-level Kestrel logs, but I believe they should correspond to RequestAborted firing.

[MihaZupan]

Closing as this is not actionable from our side without more info.
Feel free to reopen if you're able to gather more info.

Possibly related: #2195