Cluster restore from file, huge performance difference between versions 14.0.7 and 14.0.19

[mtrejo]

Hello,

Does anyone know why would a cluster restore take considerably longer in version 14.0.19, compared against version 14.0.7? We are restoring a ~50MB file with 2 caches, each containing around 700K entries. We are using the same configuration in both scenarios.

• xsite enabled
• persistence enabled
• 5 members, 3 owners
• distributed cache

    distributedCache:
      mode: "SYNC"
      owners: "3"
      statistics: "true"
      partition-handling:
        merge-policy: "PREFERRED_NON_NULL"
      state-transfer:
        timeout: "360000"
        await-initial-transfer: "true"
      persistence:
        passivation: "false"
        file-store:
          shared: "false"
          data:
            path: "/opt/infinispan/server/data/store"
          index:
            path: "/opt/infinispan/server/data/index"
      backups:
        SiteGreen:
          backup:
            strategy: "ASYNC"
            stateTransfer:
              mode: "AUTO"

The restore takes < 20 minutes in version 14.0.7, whereas in version 14.0.19 is taking > 2 hours. We already tried disabling xsite replication and persistence, to no avail.

Thank you in advanced.

[jabolina]

@mtrejo Thanks for reporting. I was able to reproduce it. Created ISPN-15523. If I find a workaround, I'll add it here.

[mtrejo]

Thank you @jabolina, we found a workaround, but I would still like to know why the restore performance changed that much. The workaround we found was to scale down the cluster to 1 member, do the restore, then scale back up to 5 members, it must be something related to the cache distribution between the members. We did some profiling, and couldn't find anything obvious, something that seemed strange was the big difference in the number of jgroups threads between the 2 versions, version 14.0.7 had way more jgroups threads than version 14.0.19.

[jabolina]

Hey @mtrejo, are you using the default tcp stack?

I identified the regression in Infinispan on commit c96c275. And JGroups in commit belaban/JGroups@1c7218c. Using the default tcp stack, you can extend it and override the tcp_nodelay=true to have the same behavior as before. For example:

<infinispan
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="urn:infinispan:config:14.0 https://infinispan.org/schemas/infinispan-config-14.0.xsd
                            urn:infinispan:server:14.0 https://infinispan.org/schemas/infinispan-server-14.0.xsd"
      xmlns="urn:infinispan:config:14.0"
      xmlns:server="urn:infinispan:server:14.0">

  <jgroups>
    <stack name="custom-tcp" extends="tcp">
      <TCP tcp_nodelay="true" stack.combine="COMBINE"/>
    </stack>
  </jgroups>

   <cache-container name="default" statistics="true">
      <transport cluster="${infinispan.cluster.name:cluster}" stack="custom-tcp" node-name="${infinispan.node.name:}"/>
   </cache-container>

  ...
</infinispan>

[jabolina]

This should be fixed on 14.0.28.Final.