NPE after updating to 14.0.21 in JGroupsMetricsManagerImpl

[mira-silhavy]

Hi, after we updated from 14.0.19 to 14.0.21 and we started getting NPE during startup on our second node. Frirst node starts fine, but second node can't.
We are using kubernetes cache config with jgroups enabled. This used to work fine in 14.0.19 and with new version the second node doesn't start anymore because of NPE below.

Did we forget to configure something, I have checked the release notes and migration guide and there's no mention on JGroupsMetricsManagerImpl, I saw this was added in https://issues.redhat.com/browse/ISPN-15029 in 14.0.21. @pruivo can you help on what is wrong with our config?

This is the jgroups config

<!--
   This configuration file uses DNS_PING. You should set the DNS_PING dns_query parameter (or the jgroups.dns.query
   property) to point to the DNS record which should return all members of the cluster.
-->
<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-5.2.xsd">

    <!-- jgroups.tcp.address is deprecated and will be removed, see ISPN-11867 -->
    <TCP bind_addr="${jgroups.bind.address,jgroups.tcp.address:SITE_LOCAL}"
         bind_port="${jgroups.bind.port,jgroups.tcp.port:7800}"
         diag.enabled="${jgroups.diag.enabled:false}"
         thread_naming_pattern="pl"
         send_buf_size="640k"
         sock_conn_timeout="300"
         logical_addr_cache_expiration="360000"
         port_range="0"
         bundler_type="transfer-queue"
         bundler.max_size="${jgroups.bundler.max_size:64000}"

         thread_pool.min_threads="${jgroups.thread_pool.min_threads:0}"
         thread_pool.max_threads="${jgroups.thread_pool.max_threads:200}"
         thread_pool.keep_alive_time="60000"

         thread_pool.thread_dumps_threshold="${jgroups.thread_dumps_threshold:10000}"
    />
    <org.jgroups.protocols.kubernetes.KUBE_PING
            port_range="1"
            namespace="${unimarket.jgroups.kubeping.k8s.namespace}"
            labels="${unimarket.jgroups.kubeping.k8s.labels}"
    />

    <RED/>
    <!-- Disabled because in k8s we want to use KUBE_PING
    <dns.DNS_PING dns_query="${jgroups.dns.query}"
                  num_discovery_runs="3"
    />
    -->
    <MERGE3 min_interval="10000"
            max_interval="30000"
    />
    <FD_SOCK2 offset="${jgroups.fd.port-offset:50000}"/>
    <FD_ALL3/>
    <VERIFY_SUSPECT2 timeout="1000"/>
    <pbcast.NAKACK2 use_mcast_xmit="false"
                    xmit_interval="200"
                    xmit_table_num_rows="50"
                    xmit_table_msgs_per_row="1024"
                    xmit_table_max_compaction_time="30000"
                    resend_last_seqno="true"
    />
    <UNICAST3 conn_close_timeout="5000"
              xmit_interval="200"
              xmit_table_num_rows="50"
              xmit_table_msgs_per_row="1024"
              xmit_table_max_compaction_time="30000"
    />
    <pbcast.STABLE desired_avg_gossip="5000"
                   max_bytes="1M"
    />
    <pbcast.GMS print_local_addr="false"
                join_timeout="${jgroups.join_timeout:2000}"
    />
    <UFC max_credits="${jgroups.max_credits:4m}"
         min_threshold="0.40"
    />
    <MFC max_credits="${jgroups.max_credits:4m}"
         min_threshold="0.40"
    />
    <FRAG4 frag_size="${jgroups.frag_size:60000}"/>
</config>

Cache config

<?xml version="1.0" encoding="UTF-8"?>
<!--
 !!! WARNING !!!
 This file is only used for all gcp environments.
 Ensure that any required changes are also made in infinispan-configuration-local.xml
 !!! WARNING !!!
 -->
<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:13.0 https://infinispan.org/schemas/infinispan-config-13.0.xsd"
        xmlns="urn:infinispan:config:13.0">

    <jgroups>
        <stack-file name="hibernate-jgroups" path="${hibernate.cache.infinispan.jgroups_cfg:com/unimarket/core/config/infinispan-jgroups-kubernetes.xml}"/>
    </jgroups>

    <cache-container name="HibernateL2CacheManager" statistics="false" default-cache="entity-cache" shutdown-hook="DEFAULT">
        <transport stack="hibernate-jgroups" cluster="infinispan-hibernate-cluster"/>

        <jmx enabled="true"/>

        <!-- Default configuration is appropriate for entity/collection caching. Entities stored on JVM HEAP memory -->
        <invalidation-cache-configuration name="entity-cache" remote-timeout="20000" mode="SYNC" statistics="false" statistics-available="false">
            <encoding media-type="application/x-java-object"/>

            <transaction mode="NONE"/>

            <!-- idle timeout = 1 hour, check interval = 1 minute -->
            <expiration max-idle="3600000" interval="60000"/>

            <!-- limit on entity count, store in JVM heap memory
            Easier to set VM memory limits in cloud configuration
            Load testing also showed that performance is acceptable -->
            <memory max-count="1000"/>
        </invalidation-cache-configuration>


        <invalidation-cache-configuration name="entries-20000-template" configuration="entity-cache">
            <memory max-count="20000"/>
        </invalidation-cache-configuration>

        <invalidation-cache-configuration name="entries-10000-template" configuration="entity-cache">
            <memory max-count="10000"/>
        </invalidation-cache-configuration>

        <invalidation-cache-configuration name="entries-2000-template" configuration="entity-cache">
            <memory max-count="2000"/>
        </invalidation-cache-configuration>

        <invalidation-cache-configuration name="entries-200-template" configuration="entity-cache">
            <memory max-count="200"/>
        </invalidation-cache-configuration>

        <invalidation-cache-configuration name="entries-20-template" configuration="entity-cache">
            <memory max-count="20"/>
        </invalidation-cache-configuration>

        <invalidation-cache-configuration name="natural-id-template" configuration="entity-cache">
            <expiration max-idle="-1" interval="-1"/>
            <memory max-count="10000"/>
        </invalidation-cache-configuration>

        <invalidation-cache-configuration name="eternal">
            <encoding media-type="application/x-java-object"/>
            <transaction mode="NONE"/>
            <expiration max-idle="-1" interval="-1"/>
            <memory max-count="1000"/>
        </invalidation-cache-configuration>

        <invalidation-cache-configuration name="custom-query-cache" configuration="entity-cache">
            <!-- idle timeout = 30 min, check interval = 1 minute -->
            <expiration max-idle="1800000" interval="60000"/>
            <memory max-count="50000"/>
        </invalidation-cache-configuration>

    </cache-container>

</infinispan>

Exception

ISPN000660: DefaultCacheManager start failed, stopping any running components
org.infinispan.commons.CacheConfigurationException: ISPN000659: Component ComponentWrapper{name=org.infinispan.topology.ClusterTopologyManager, instance=org.infinispan.topology.ClusterTopologyManagerImpl@794bb32a, status=FAILED} failed to start
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:584)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:807)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:634)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:598)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:576)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:807)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:634)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:598)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:576)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:807)
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:379)
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:252)
	at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:779)
	at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:747)
	at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:411)
	at org.infinispan.hibernate.cache.commons.DefaultCacheManagerProvider.getEmbeddedCacheManager(DefaultCacheManagerProvider.java:45)
	at org.infinispan.hibernate.cache.v62.InfinispanRegionFactory.createCacheManager(InfinispanRegionFactory.java:274)
	at org.infinispan.hibernate.cache.v62.InfinispanRegionFactory.start(InfinispanRegionFactory.java:241)
	at org.hibernate.cache.internal.EnabledCaching.<init>(EnabledCaching.java:83)
	at org.hibernate.engine.spi.CacheInitiator.initiateService(CacheInitiator.java:30)
	at org.hibernate.engine.spi.CacheInitiator.initiateService(CacheInitiator.java:23)
	at org.hibernate.service.internal.SessionFactoryServiceRegistryImpl.initiateService(SessionFactoryServiceRegistryImpl.java:77)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:263)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:238)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:215)
	at org.hibernate.service.internal.SessionFactoryServiceRegistryImpl.getService(SessionFactoryServiceRegistryImpl.java:114)
	at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:241)
	at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:450)
	at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:1507)
	at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:75)
	at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:376)
	at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:409)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.NullPointerException: null
	at java.base/java.util.Objects.requireNonNull(Objects.java:209)
	at org.infinispan.remoting.transport.jgroups.JGroupsMetricsManagerImpl$DestinationMetricsBuilder.build(JGroupsMetricsManagerImpl.java:253)
	at org.infinispan.remoting.transport.jgroups.JGroupsMetricsManagerImpl.createDestinationMetrics(JGroupsMetricsManagerImpl.java:185)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at org.infinispan.remoting.transport.jgroups.JGroupsMetricsManagerImpl.get(JGroupsMetricsManagerImpl.java:168)
	at org.infinispan.remoting.transport.jgroups.JGroupsMetricsManagerImpl.trackRequest(JGroupsMetricsManagerImpl.java:108)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeCommand(JGroupsTransport.java:1022)
	at org.infinispan.topology.TopologyManagementHelper.executeOnCoordinator(TopologyManagementHelper.java:94)
	at org.infinispan.topology.ClusterTopologyManagerImpl.fetchRebalancingStatusFromCoordinator(ClusterTopologyManagerImpl.java:156)
	at org.infinispan.topology.ClusterTopologyManagerImpl.start(ClusterTopologyManagerImpl.java:147)
	at org.infinispan.topology.CorePackageImpl$3.start(CorePackageImpl.java:74)
	at org.infinispan.topology.CorePackageImpl$3.start(CorePackageImpl.java:58)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:616)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:607)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:576)
	... 33 common frames omitted

[pruivo]

@mira-silhavy You can update your Infinispan version or use the workaround described here: https://issues.redhat.com/browse/ISPN-15351