IProxyConfigProvider and 404 behavior

[chandramouleswaran]

Our service uses Reverse proxy for some really concurrent workloads and it's great to see the performance benefits we are getting with .NET Core and HTTP2.

We are a Service Fabric based service and have implemented our own IProxyConfigProvider which probes SF periodically and also listens to notification changes to influence the Cluster/Route config.

Recently, we ran into a live site issue where reverse proxy returned a 404 because it was not able to match the route. On digging deeper, we noticed that the request happened exactly at the same time when the old change token was cancelled.

Some questions/discussion points

1. Does the 404 happen at route matching layer and not the proxy layer?
2. If so, how can we tell if it's 404 without reaching configuration "vs" after reaching the configuration? (Would this be throw checking if ReverseProxy feature exists on the HttpContext at a layer above routing?)
3. Lastly, our API clients are tuned to retry on 5XX, 429 status codes. On 404 - client automatically retries if the content type is "text/html" (comes from Web API and HTTP sys behavior?).

What would your recommendation be deterministically say if
a. It's a 404 because of missing config (entire config or via change token) --> IMO, this is mostly a 500 or 503
b. It's a 404 because of missing route in an active configuration

[Tratcher]

1. Yes, 404s would be because of failing to match a route
2. I don't understand. Routing only happens once in the pipeline. YARP routes are integrated into the Asp.Net route table along with other components like controllers.

Config is updated atomically. If a request arrives for a new route before that route is fully loaded there's no way to distinguish that from requesting a non-existing route.

[chandramouleswaran]

If a request arrives for a new route before that route is fully loaded there's no way to distinguish that from requesting a non-existing route.

I understand the above - what happens for an existing route where the cluster destination alone changes to a different address?

Call flow as follows:

1. We identified a change in endpoint; updated configuration and triggered the change token.
2. Request comes in for an existing route (this served a 404 - the observation is that ReverseProxy feature was missing)
3. Configuration was loaded and new calls were served correctly.

Keep in mind that it might be just one destination which changed here but everything on the configuration was dropped.

At step 2 - I think it's almost always better to serve traffic with older configuration and the backend to respond with 404/503 or HttpRequestException/SocketException in case it no longer can accept traffic or is unavailable respectively.

We are doing something like this to simulate the above behavior - it might be better to bake this directly into ReverseProxy and providing the capability to onboard to this behavior (using existing config in-between change token based new config trigger OR enable a retry when config was missing but used to be there) via options?

Here is the piece of the middleware code which we are injecting before ReverseProxy to tackle this edge case

public async Task InvokeAsync(HttpContext context)
{
    await this.next(context).ConfigureAwait(false);
    if (context?.Response?.StatusCode == 404)
    {
        try
        {
            var feature = context.GetReverseProxyFeature();
            if (feature != null)
            {
                if (feature.Route == null)
                {
                    this.logger.LogInformation($"The route model for this request was null");
                }
                else if (feature.ProxiedDestination != null)
                {
                    this.logger.LogInformation($"We did pick a final destination of {feature.ProxiedDestination} - do not change response");
                }
            }
            else
            {
                this.logger.LogError($"Reverse proxy feature is missing - returning 503 GatewayNoAvailableDestinationsException");
                await this.responseProvider.WriteErrorResponse(context.Response, Exception).ConfigureAwait(false);
            }
        }
        catch (Exception ex)
        {
            this.logger.LogError(ex, "Ran into error while gathering data from reverse proxy featuer - returning 503 GatewayNoAvailableDestinationsException");
            await this.responseProvider.WriteErrorResponse(context.Response, Exception).ConfigureAwait(false);
        }
    }
}

[Tratcher]

I understand the above - what happens for an existing route where the cluster destination alone changes to a different address?

If only a destination address is modified that shouldn't affect routes at all and shouldn't cause 404s. The config is diff'd to see what's changed and routes are only re-loaded if route specific data has changed. Changing a destination address does not affect a route.

It's possible this is an issue with your custom config provider. E.g. do the route/cluster names change when you reload config? That would affect the diff. If you enable debug level logs and then trigger a config change, what does it say changed?

[chandramouleswaran]

Interestingly, it looks like almost all the cases on our side has been around GW restarting because of an upgrade or restarting because of a new host placement. And the 404s are when host comes up for the first time and we did not have any routes populated.

Will ping back after we have around a week worth of data to discuss further or call this closed.

[samsp-msft]

This sounds similar to the problem in #306. If you are routing traffic to YARP before its been able to fetch the routes, then it will likely 404. What is the load balancer in front of YARP and how does it know to add the YARP instance to its collection? Delaying that until the proxy is ready is probably required.

[Tratcher]

Note YARP calls IProxyConfigProvider.GetConfig() at startup in such a way that it would block the proxy from starting and accepting requests until the first config were loaded. Your implementation can choose to make that first call blocking until SF clusters are available.