Infinispan Deadlock issue when file-store persistent enabled

[DmytroKozlovskyi]

Hi,
Looking for help here to understand what is wrong with our Keycloak/Infinispan setup.
We upgraded to Keycloak v23.0.7 a few months ago and decoupled Infinispan with v15.0. Also, we have enabled file-based persistence for caches. All worked well until May 7, when we started to have deadlocks (2 incidents).
In the first case there was an error on Pod infinispan-0:

2024-05-07 14:19:48,945 ERROR (blocking-thread--p3-t7) [org.infinispan.persistence.sifs.Index] Error encountered with index, SIFS may not operate properly. java.io.IOException: Cannot read record [441057:4085] (already read 0), file size is 441057

and more than an hour later deadlocks appeared on other pods:

[org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand on Cache 'actionTokens', writing keys [WrappedByteArray[* (39 bytes)]] org.infinispan.commons.TimeoutException: ISPN000476: Timed out waiting for responses for request 2946380 from infinispan-0-52640 after 17.5 seconds

and on pod infinispan-0 at this time

2024-05-07 15:34:19,088 ERROR (non-blocking-thread--p2-t3) [org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand on Cache 'actionTokens', writing keys [WrappedByteArray[* (39 bytes)]] org.infinispan.commons.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key WrappedByteArray[* (39 bytes)] and requestor CommandInvocation:infinispan-1-8999:3347917. Lock is held by CommandInvocation:infinispan-0-52640:3485477

The second case looks a little bit different but again, started from SIFS error:

2024-05-08 01:18:47,354 ERROR (blocking-thread--p3-t20) [org.infinispan.persistence.sifs.Compactor] ISPN029021: File id -1 encountered an exception while compacting, file may be orphaned java.lang.IllegalStateException: Error reading from 9:6897714


2024-05-08 02:01:39,830 ERROR (non-blocking-thread--p2-t7) [org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command GetCacheEntryCommand on Cache 'clientSessions', writing keys [] org.infinispan.persistence.spi.PersistenceException: ISPN029019: Cannot load key WrappedByteArray[*... (85 bytes)] from index.

And then 18 hours later again deadlocks

2024-05-08 19:07:55,027 ERROR (non-blocking-thread--p2-t2) [org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command ReplaceCommand on Cache 'clientSessions', writing keys [WrappedByteArray[... (85 bytes)]] org.infinispan.commons.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key WrappedByteArray[... (85 bytes)] and requestor CommandInvocation:infinispan-0-8012:384510. Lock is held by CommandInvocation:infinispan-0-8012:384500

Are these both cases related to persistence enabled for caches? We have disabled persistence and so far no issue. But we are worried, that we might be wrong, and the issue could come back at any time again. If it's because of persistence enabled, than why do we have the problem: is it a bug or did we mess up with the configuration?

Our full infinispan config:

  replicas: 3
  podAntiAffinity:
    crossZone: {{ if eq .Values.environment "live" }}true{{ else }}false{{ end }}
  container:
    extraJvmOpts: "-XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:MaxRAMPercentage=50 -XX:MinRAMPercentage=50 -XX:InitialRAMPercentage=50 -XX:+UseG1GC -Xlog:gc*"
    resources:
      limits:
        cpu: {{ if eq .Values.environment "live" }}500m{{ else }}25m{{ end }}
        memory: {{ if eq .Values.environment "live" }}8Gi{{ else }}2Gi{{ end }}
      requests:
        cpu: {{ if eq .Values.environment "live" }}500m{{ else }}25m{{ end }}
        memory: {{ if eq .Values.environment "live" }}8Gi{{ else }}2Gi{{ end }}
    storage:
      size: 8Gi
      storageClassName: ""
      # [USER] Set `ephemeral: true` to delete all persisted data when clusters shut down or restart.
      ephemeral: false
  makeDataDirWritable: true
  expose:
    type: ""
  monitoring:
    enabled: true
  security:
    authentication: true
    secretName: infinispan-users
  ssl:
    endpointSecretName: infinispan-endpoint-cert-tls
    transportSecretName: infinispan-transport-cert-tls
{{- if eq .Values.environment "live" }}
  nodeSelector:
    karpenter.sh/capacity-type: on-demand
    kubernetes.io/arch: arm64
{{- end }}
  lifecycle:
    preStop:
      exec:
        command:
          - /bin/sh
          - -c
          - |
            sleep 15
  logging:
    categories:
      # [USER] Specify the FQN of a package from which you want to collect logs.
      - category: com.arjuna
        # [USER] Specify the level of log messages.
        level: warn
      # No need to warn about not being able to TLS/SSL handshake
      - category: io.netty.handler.ssl.ApplicationProtocolNegotiationHandler
        level: error
  infinispan:
    cacheContainer:
      # [USER] Add cache, template, and counter configuration.
      name: default
      globalState:
        uncleanShutdownAction: IGNORE
      statistics: "true"
      # [USER] Specify `security: null` to disable security authorization.
      security:
        authorization: {}
      transport:
        cluster: ${infinispan.cluster.name:cluster}
        node-name: ${infinispan.node.name:}
        stack: kubernetes
        urn:infinispan:server:15.0:securityRealm: "cluster-transport"
      serialization:
        allowList:
          class: org.keycloak.cluster.infinispan.WrapperClusterEvent
          regex: .*
        marshaller: org.infinispan.jboss.marshalling.commons.GenericJBossMarshaller
      distributedCacheConfiguration:
        name: "distributed-cache-cfg"
        mode: "SYNC"
        owners: "3"
        statistics: "true"
        encoding:
          mediaType: "application/x-jboss-marshalling"
        persistence:
          passivation: "false"
          fileStore:
            shared: "false"
            preload: "true"
            writeBehind:
              modificationQueueSize: "2048"
            data:
              path: "data"
            index:
              path: "index"
        state-transfer:
          timeout: "600000"
      replicatedCacheConfiguration:
        name: "replicated-cache-cfg"
        mode: SYNC
        encoding:
          mediaType: "application/x-jboss-marshalling"
        statistics: "true"
      caches:
        work:
          replicatedCache:
            configuration: replicated-cache-cfg
        sessions:
          distributedCache:
            configuration: distributed-cache-cfg
        authenticationSessions:
          distributedCache:
            configuration: distributed-cache-cfg
        offlineSessions:
          distributedCache:
            configuration: distributed-cache-cfg
        clientSessions:
          distributedCache:
            configuration: distributed-cache-cfg
        offlineClientSessions:
          distributedCache:
            configuration: distributed-cache-cfg
        loginFailures:
          distributedCache:
            configuration: distributed-cache-cfg
        actionTokens:
          distributedCache:
            configuration: distributed-cache-cfg
    server:
      endpoints:
      # [USER] Hot Rod and REST endpoints.
      - securityRealm: default
        socketBinding: default
        connectors:
          rest:
            restConnector:
          hotrod:
            hotrodConnector:
      # [METRICS] Metrics endpoint for cluster monitoring capabilities.
      - connectors:
          rest:
            restConnector:
              authentication:
                mechanisms: BASIC
        securityRealm: metrics
        socketBinding: metrics
      interfaces:
      - inetAddress:
          value: ${infinispan.bind.address:127.0.0.1}
        name: public
      security:
        credentialStores:
        - clearTextCredential:
            clearText: secret
          name: credentials
          path: credentials.pfx
        securityRealms:
        # [USER] Security realm for the Hot Rod and REST endpoints.
        - name: default
          # [USER] Comment or remove this properties realm to disable authentication.
          propertiesRealm:
            groupProperties:
              path: groups.properties
            groupsAttribute: Roles
            userProperties:
              path: users.properties
          serverIdentities:
            ssl:
              keystore:
                path: "/etc/encrypt/endpoint/keystore.p12"
                alias: "1"
                credentialReference:
                  store: credentials
                  alias: keystore
        # [METRICS] Security realm for the metrics endpoint.
        - name: metrics
          propertiesRealm:
            groupProperties:
              path: metrics-groups.properties
              relativeTo: infinispan.server.config.path
            groupsAttribute: Roles
            userProperties:
              path: metrics-users.properties
              relativeTo: infinispan.server.config.path
        - name: cluster-transport
          serverIdentities:
            ssl:
              keystore:
                alias: "1"
                path: "/etc/encrypt/transport/keystore.p12"
                credentialReference:
                  store: credentials
                  alias: keystore
              truststore:
                path: "/etc/encrypt/transport/truststore.p12"
                credentialReference:
                  store: credentials
                  alias: keystore
      socketBindings:
        defaultInterface: public
        portOffset: ${infinispan.socket.binding.port-offset:0}
        socketBinding:
          # [USER] Socket binding for the Hot Rod and REST endpoints.
        - name: default
          port: 11222
          # [METRICS] Socket binding for the metrics endpoint.
        - name: metrics
          port: 11223

[jabolina]

I can't say for sure that the SIFS errors are causing the timeouts. The big reason is I can't tell by the logs if both operations are trying to access the same keys. If we make this link, then it could be some bug hidden in SIFS. So, I also can't confirm that without persistence, the issue won't happen again ☹️.

By 15.0, I assume you meant 15.0.0.Final? Since then, we added a few fixes to SIFS (ISPN-15943, ISPN-15930, ISPN-15894). Are you able to test with a more recent version? The latest is 15.0.3.Final.

Configuration-wise, it looks good to me. However, @ryanemerson knows more here.

[DmytroKozlovskyi]

Hi, @jabolina
Thank you for getting back to me.
We are on Infinispan Server 15.0.1.Final currently. I can upgrade to the latest is 15.0.3.Final, but problem, is that we can't recreate the issue on our test environment, and on live we don't want to produce any new outages. I'll try to review keycloak logs to find out if the keys the same and return back.
So far, after disabling persistence, we don't have any errors in logs.

[pruivo]

A couple of questions.

• Can you share your Infinispan configuration in Keycloak?

• I've noticed you configured 0.5vCPU to Infinispan. Are you monitoring the CPU usage? Any spike around those events?

• Can you explain the reason for persistence and why it is write-behind enabled?

As @jabolina mentioned, it is difficult to say. One possible case is when the write-behind queue gets full, which blocks further operation until it is flushed to disk. The lock is kept acquired during all this time. In addition, the external Infinispan sends events to the Keycloak nodes and Keycloak needs to process them. Once again, the lock is kept acquired during this period. Can you correlate a peak in your load to the time of those 2 incidents?

Offtopic

I'm helping the Kecloak team with their Infinispan deployment in multi-site HA scenarios. Can you share the reasons why you are deploying an external Infinispan if cross-site is not enabled? Basically, what problem does the external Infinispan solve that Keycloak itself can't? An external Infinispan is not required for single cluster Keycloak deployments. Thanks.

[DmytroKozlovskyi]

Hi, @pruivo
Thank you for your response.
Our Infinispan config for Keycloak 23.0.7:

<?xml version="1.0" encoding="UTF-8"?>
<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:14.0 http://www.infinispan.org/schemas/infinispan-config-14.0.xsd"
        xmlns="urn:infinispan:config:14.0">

    <cache-container name="keycloak">
        <transport lock-timeout="60000"/>
        <local-cache name="realms" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="users" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="authorization" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="keys" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="3600000"/>
            <memory max-count="1000"/>
        </local-cache>
        <distributed-cache name="sessions" owners="2">
            <expiration lifespan="-1"/>
            <persistence passivation="false">
                <remote-store xmlns="urn:infinispan:config:store:remote:14.0"
                            cache="sessions"
                            raw-values="true"
                            shared="true"
                            segmented="false">
                    <!-- This is a workaround for the following issue https://github.com/keycloak/keycloak/issues/27117 and should be removed when the issue is fixed -->
                    <write-behind modification-queue-size="1024"/>
                    <!-- End of the workaround -->
                    <remote-server host="${env.KC_REMOTE_STORE_HOST}"
                                port="${env.KC_REMOTE_STORE_PORT}"/>
                    <connection-pool max-active="16"
                                    exhausted-action="CREATE_NEW"/>
                    <security>
                        <authentication server-name="infinispan">
                            <digest username="${env.KC_REMOTE_STORE_USERNAME}"
                                    password="${env.KC_REMOTE_STORE_PASSWORD}"
                                    realm="default"/>
                        </authentication>
                        <encryption protocol="TLSv1.3"
                                    sni-hostname="${env.KC_REMOTE_STORE_HOST}">
                            <truststore filename="/etc/encrypt/certs/root-certs.pem"
                                        type="pem"/>
                        </encryption>
                    </security>
                </remote-store>
            </persistence>
        </distributed-cache>
        <distributed-cache name="authenticationSessions" owners="2">
            <expiration lifespan="-1"/>
            <persistence passivation="false">
                <remote-store xmlns="urn:infinispan:config:store:remote:14.0"
                            cache="authenticationSessions"
                            raw-values="true"
                            shared="true"
                            segmented="false">
                    <!-- This is a workaround for the following issue https://github.com/keycloak/keycloak/issues/27117 and should be removed when the issue is fixed -->
                    <write-behind modification-queue-size="1024"/>
                    <!-- End of the workaround -->
                    <remote-server host="${env.KC_REMOTE_STORE_HOST}"
                                port="${env.KC_REMOTE_STORE_PORT}"/>
                    <connection-pool max-active="16"
                                    exhausted-action="CREATE_NEW"/>
                    <security>
                        <authentication server-name="infinispan">
                            <digest username="${env.KC_REMOTE_STORE_USERNAME}"
                                    password="${env.KC_REMOTE_STORE_PASSWORD}"
                                    realm="default"/>
                        </authentication>
                        <encryption protocol="TLSv1.3"
                                    sni-hostname="${env.KC_REMOTE_STORE_HOST}">
                            <truststore filename="/etc/encrypt/certs/root-certs.pem"
                                        type="pem"/>
                        </encryption>
                    </security>
                </remote-store>
            </persistence>
        </distributed-cache>
        <distributed-cache name="offlineSessions" owners="2">
            <expiration lifespan="-1"/>
            <persistence passivation="false">
                <remote-store xmlns="urn:infinispan:config:store:remote:14.0"
                            cache="offlineSessions"
                            raw-values="true"
                            shared="true"
                            segmented="false">
                    <!-- This is a workaround for the following issue https://github.com/keycloak/keycloak/issues/27117 and should be removed when the issue is fixed -->
                    <write-behind modification-queue-size="1024"/>
                    <!-- End of the workaround -->
                    <remote-server host="${env.KC_REMOTE_STORE_HOST}"
                                port="${env.KC_REMOTE_STORE_PORT}"/>
                    <connection-pool max-active="16"
                                    exhausted-action="CREATE_NEW"/>
                    <security>
                        <authentication server-name="infinispan">
                            <digest username="${env.KC_REMOTE_STORE_USERNAME}"
                                    password="${env.KC_REMOTE_STORE_PASSWORD}"
                                    realm="default"/>
                        </authentication>
                        <encryption protocol="TLSv1.3"
                                    sni-hostname="${env.KC_REMOTE_STORE_HOST}">
                            <truststore filename="/etc/encrypt/certs/root-certs.pem"
                                        type="pem"/>
                        </encryption>
                    </security>
                </remote-store>
            </persistence>
        </distributed-cache>
        <distributed-cache name="clientSessions" owners="2">
            <expiration lifespan="-1"/>
            <persistence passivation="false">
                <remote-store xmlns="urn:infinispan:config:store:remote:14.0"
                            cache="clientSessions"
                            raw-values="true"
                            shared="true"
                            segmented="false">
                    <!-- This is a workaround for the following issue https://github.com/keycloak/keycloak/issues/27117 and should be removed when the issue is fixed -->
                    <write-behind modification-queue-size="1024"/>
                    <!-- End of the workaround -->
                    <remote-server host="${env.KC_REMOTE_STORE_HOST}"
                                port="${env.KC_REMOTE_STORE_PORT}"/>
                    <connection-pool max-active="16"
                                    exhausted-action="CREATE_NEW"/>
                    <security>
                        <authentication server-name="infinispan">
                            <digest username="${env.KC_REMOTE_STORE_USERNAME}"
                                    password="${env.KC_REMOTE_STORE_PASSWORD}"
                                    realm="default"/>
                        </authentication>
                        <encryption protocol="TLSv1.3"
                                    sni-hostname="${env.KC_REMOTE_STORE_HOST}">
                            <truststore filename="/etc/encrypt/certs/root-certs.pem"
                                        type="pem"/>
                        </encryption>
                    </security>
                </remote-store>
            </persistence>
        </distributed-cache>
        <distributed-cache name="offlineClientSessions" owners="2">
            <expiration lifespan="-1"/>
            <persistence passivation="false">
                <remote-store xmlns="urn:infinispan:config:store:remote:14.0"
                            cache="offlineClientSessions"
                            raw-values="true"
                            shared="true"
                            segmented="false">
                    <!-- This is a workaround for the following issue https://github.com/keycloak/keycloak/issues/27117 and should be removed when the issue is fixed -->
                    <write-behind modification-queue-size="1024"/>
                    <!-- End of the workaround -->
                    <remote-server host="${env.KC_REMOTE_STORE_HOST}"
                                port="${env.KC_REMOTE_STORE_PORT}"/>
                    <connection-pool max-active="16"
                                    exhausted-action="CREATE_NEW"/>
                    <security>
                        <authentication server-name="infinispan">
                            <digest username="${env.KC_REMOTE_STORE_USERNAME}"
                                    password="${env.KC_REMOTE_STORE_PASSWORD}"
                                    realm="default"/>
                        </authentication>
                        <encryption protocol="TLSv1.3"
                                    sni-hostname="${env.KC_REMOTE_STORE_HOST}">
                            <truststore filename="/etc/encrypt/certs/root-certs.pem"
                                        type="pem"/>
                        </encryption>
                    </security>
                </remote-store>
            </persistence>
        </distributed-cache>
        <distributed-cache name="loginFailures" owners="2">
            <expiration lifespan="-1"/>
            <persistence passivation="false">
                <remote-store xmlns="urn:infinispan:config:store:remote:14.0"
                            cache="loginFailures"
                            raw-values="true"
                            shared="true"
                            segmented="false">
                    <!-- This is a workaround for the following issue https://github.com/keycloak/keycloak/issues/27117 and should be removed when the issue is fixed -->
                    <write-behind modification-queue-size="1024"/>
                    <!-- End of the workaround -->
                    <remote-server host="${env.KC_REMOTE_STORE_HOST}"
                                port="${env.KC_REMOTE_STORE_PORT}"/>
                    <connection-pool max-active="16"
                                    exhausted-action="CREATE_NEW"/>
                    <security>
                        <authentication server-name="infinispan">
                            <digest username="${env.KC_REMOTE_STORE_USERNAME}"
                                    password="${env.KC_REMOTE_STORE_PASSWORD}"
                                    realm="default"/>
                        </authentication>
                        <encryption protocol="TLSv1.3"
                                    sni-hostname="${env.KC_REMOTE_STORE_HOST}">
                            <truststore filename="/etc/encrypt/certs/root-certs.pem"
                                        type="pem"/>
                        </encryption>
                    </security>
                </remote-store>
            </persistence>
        </distributed-cache>
        <distributed-cache name="actionTokens" owners="2">
            <expiration max-idle="-1" lifespan="-1" interval="300000"/>
            <persistence passivation="false">
                <remote-store xmlns="urn:infinispan:config:store:remote:14.0"
                            cache="actionTokens"
                            raw-values="true"
                            shared="true"
                            segmented="false">
                    <!-- This is a workaround for the following issue https://github.com/keycloak/keycloak/issues/27117 and should be removed when the issue is fixed -->
                    <write-behind modification-queue-size="1024"/>
                    <!-- End of the workaround -->
                    <remote-server host="${env.KC_REMOTE_STORE_HOST}"
                                port="${env.KC_REMOTE_STORE_PORT}"/>
                    <connection-pool max-active="16"
                                    exhausted-action="CREATE_NEW"/>
                    <security>
                        <authentication server-name="infinispan">
                            <digest username="${env.KC_REMOTE_STORE_USERNAME}"
                                    password="${env.KC_REMOTE_STORE_PASSWORD}"
                                    realm="default"/>
                        </authentication>
                        <encryption protocol="TLSv1.3"
                                    sni-hostname="${env.KC_REMOTE_STORE_HOST}">
                            <truststore filename="/etc/encrypt/certs/root-certs.pem"
                                        type="pem"/>
                        </encryption>
                    </security>
                </remote-store>
            </persistence>
        </distributed-cache>
        <replicated-cache name="work">
            <expiration lifespan="-1"/>
            <persistence passivation="false">
                <remote-store xmlns="urn:infinispan:config:store:remote:14.0"
                            cache="work"
                            raw-values="true"
                            shared="true"
                            segmented="false">
                    <remote-server host="${env.KC_REMOTE_STORE_HOST}"
                                port="${env.KC_REMOTE_STORE_PORT}"/>
                    <connection-pool max-active="16"
                                    exhausted-action="CREATE_NEW"/>
                    <security>
                        <authentication server-name="infinispan">
                            <digest username="${env.KC_REMOTE_STORE_USERNAME}"
                                    password="${env.KC_REMOTE_STORE_PASSWORD}"
                                    realm="default"/>
                        </authentication>
                        <encryption protocol="TLSv1.3" sni-hostname="${env.KC_REMOTE_STORE_HOST}">
                            <truststore filename="/etc/encrypt/certs/root-certs.pem"
                                        type="pem"/>
                        </encryption>
                    </security>
                </remote-store>
            </persistence>
        </replicated-cache>
    </cache-container>
</infinispan>

Yes, we are monitoring CPU and there is no spikes during incidents. Also, we have cpuCFSQuota disabled, so it's not hard limit.
But, there is a more than 2x CPU usage drop after disabling persistence.

Why we choose such architecture?
We are using custom themes for log in pages. For that we have pipelines in CI which build new images and then deploy them into K8s cluster. Because new themes can be released by dev team at any time, we want to be sure, that this pipeline will not produce any outages. So, we trying to make Keycloak pods stateless. Also, we want prevent loosing sessions during Keycloak upgrade. This are the reasons to have external Infinispan.
For persistence the main reason was also, preventing to loose active customer sessions during upgrade or cluster issues/outages. At the same time we don't want to affect performance. That is why we have enabled write-behind.

We don't have so huge experience with Keycloak and maybe we choose wrong architecture. And we found documentation for Keycloak quite tricky. For example, it valid only for latest version, and when you use not latest version, hard to find right configuration for your version. Also, for me it's not clear, why when we have external Infinispan, Keycloak continue to create internal Infinispan cluster. Maybe it can cause also some issue. Don't know if it's by design or also bug.

[wburns]

To second what @pruivo mentioned.. the SIFS error shouldn't cause any issues with timeouts, at worst case it would give a different exception, which you didn't seem to receive.

The async modification queue could possibly cause timeouts, but I don't see the disk being backed up by 15+s worth of writes unless something is going wrong with the underlying disk. I don't suspect this as I would normally expect other issues to crop us as well.

However, I can't stress how much I advise to upgrade to 15.0.3 as @jabolina mentioned. There are quite a few fixes to SIFS that should prevent the exceptions from occurring in the first place.

After upgrading if you still see timeouts than I would suggest taking a thread and heap dump if possible when you are seeing timeouts. If you cannot do that you can enable TRACE level for org.infinispan.persistence.async.AsyncNonBlockingStore category which will generate many messages but will be able to tell you when the queue has filled and you can see how long a batch takes to update. Each one has a unique identifier associated with it that is printed in the TRACE messages.

[pruivo]

@DmytroKozlovskyi thanks for the description. Well, for Keycloak 23, you have the chance to connect a database to the embedded caches (see keycloak/keycloak#24774). It will keep your sessions safe and avoid deploying an external Infinispan.

The next release, Keycloak 25, brings support to persistent sessions (details here keycloak/keycloak#28265). I'm unsure at this point if it is a fully supported feature or a preview.

Also, for me it's not clear, why when we have external Infinispan, Keycloak continue to create internal Infinispan cluster. Maybe it can cause also some issue. Don't know if it's by design or also bug.

That is a limitation of Keycloak. The remote store with external Infinispan is a requirement for multi-site deployments (geographically distributed Keycloak clusters). I'm working on improving Keycloak by removing the embedded caches and relying only on the external Infinispan. See keycloak/keycloak#28745.

[DmytroKozlovskyi]

The next release, Keycloak 25, brings support to persistent sessions (details here keycloak/keycloak#28265). I'm unsure at this point if it is a fully supported feature or a preview.

Oh, this is what exactly we are looking for. Looking forward to Keycloak 25 release. For now, we decided to upgrade Infinispan to 15.0.3 but keep persistence off and wait for the Keycloak 25 release. Thank you all for your help.