Request Affinity key decryption failures

[pinkfloydx33]

We are running a YARP-based API gateway inside of a Kubernetes cluster. Recently we started getting a lot of exceptions in our logs across multiple environments related to DataProtection by way of Yarp.ReverseProxy.SessionAffinity.CookieSessionAffinityPolicy:

{
    "Message": "The request affinity key decryption failed.",
    "SourceContext": "Yarp.ReverseProxy.SessionAffinity.CookieSessionAffinityPolicy",
    "EventId": {
        "Id": 38,
        "Name": "RequestAffinityKeyDecryptionFailed"
    },
    "Exception": {
        "Type": "System.Security.Cryptography.CryptographicException",
        "Message": "The key {3c41e5c9-ba87-4b12-8a17-312e7326b765} was not found in the key ring. For more information go to http://aka.ms/dataprotectionwarning",
        "Source": "Microsoft.AspNetCore.DataProtection",
        "TargetSite": "Byte[] UnprotectCore(Byte[], Boolean, UnprotectStatus ByRef)"
    }
}

Our configurations include Cookie-based session affinity, however we are not managing data protection features ourselves (i.e. no call to AddDataProtection). It's odd that this seems to have only started within the last week. We haven't updated nor re-deployed our application and the pods have been running without interruption--nothing has changed.

Any ideas on why this might be happening? Is it something to be concerned with, and if so, how might we fix it? It doesn't seem to impact actual request proxying so I think it may be harmless... but I wanted to verify anyways.

[Tratcher]

There are a few possible causes for this:

• Are you running multiple instances of YARP, maybe behind an L4 TCP load balancer? In that case you'd have to make sure to use shared DataProtection keys across instances
• Are the keys being saved somewhere outside the container? If not, the keys will be reset on app restart, causing this error for any existing session cookie.

The fix for both cases is to configure AddDataProtection to store keys outside of the container in a central location.

[pinkfloydx33]

1. Multiple instances of YARP, exposed via a Kubernetes Service of type LoadBalancer (Azure LB).
2. No, not saving them

OK. Sounds like I need to configure data protection then. Hadn't even considered it was needed but I guess that makes sense since yarp is dealing with sessions there.

[eyalg-msft]

Hi,
Jumping on the thread for the same problem but not on containers.
The problem is that cookie affinity is not working. we get RequestAffinityKeyDecryptionFailed almost all the time,
then a redistribute fallback strategy adds a new cookie to the response, in turn it gets the RequestAffinityKeyDecryptionFailed on the next request, so affinity is completely not working.

We are running multiple YARP instances on multiple VMs.
I have debugged YARP with a minimal sample that shows the problem. YARP is using data protection to protect the destination as a key. seems like the Data protection key is only stored in the process. So other VMs that runs YARP gets different keys and get exception when trying to decrypt keys from another machines.

I am wondering, what are we trying to protect here? the destination machine name is not a secret. Also, an attacker can use the protected affinity key to DDos a specific destination even without the destination explicit name. It just seems more appropriate to protect secrets and authentication. Is this security by inconvenient? Or am I missing an attack vector for sharing the actual destination name (which is anyway private and inaccessible outside of the datacenter)

So, my question is how do we use multiple instances of YARP without some redesign of a new feature to store something outside of the process?

Can we choose not to protect the cookie value? when configuring the cookie affinity?

If both answers are no, can you please refer me to how we solve it?
Do I need to manually save the keys to some REDIS cache for example? or is there a better way? Sounds like over engineering to me...

Will something as simple as this will solve the issue?
services.AddDataProtection().SetApplicationName("SharedName");

Thanks,
Eyal.

[eyalg-msft]

Currently Affinity in YARP is broken out of the box form more than 1 YARP instance. The proposed work around is a design decision to take dependency on a distributed cache to share hashing keys to allow affinity. This is something which is not acceptable for our product.
I suggest following industry standards and use a one-way hashing algorithm like SHA256. We can provide it by default in YARP and allow anyone who wants to do something else to use DI to define a different data protection. This way it will work out of the box and will not be dependent on other components like distributed cache. We started implementing it in our solution. The problem is that it makes the use of YARP affinity redundant, since we must save the hashed values of the destination list, which is a duplication of the YARP logic. What do you think can we add it as YARP default affinity hashing behavior?

Anyway IMO, in the short term we must update the affinity page documentation to include a big bold section on data protection and distributed caching and mention it will not work otherwise with more than 1 YARP instance since the keys are different.
What do you think?

[davidfowl]

I suggest following industry standards and use a one-way hashing algorithm like SHA256. We can provide it by default in YARP and allow anyone who wants to do something else to use DI to define a different data protection. This way it will work out of the box and will not be dependent on other components like distributed cache. We started implementing it in our solution. The problem is that it makes the use of YARP affinity redundant, since we must save the hashed values of the destination list, which is a duplication of the YARP logic. What do you think can we add it as YARP default affinity hashing behavior?

Let's upgrade this discussion to an issue so we can discuss it there.

Anyway IMO, in the short term we must update the affinity page documentation to include a big bold section on data protection and distributed caching and mention it will not work otherwise with more than 1 YARP instance since the keys are different.

I agree this is a painful design choice to use by default and should be documented as such.

[Tratcher]

Doc issue: #1981

[Tratcher]

Feature request: #1982

[Tratcher]

Woops, #1980 was already created 😁