Rocksdb cache store - The crash happened outside the Java Virtual Machine in native code

[Satishshelake]

Hello,
I am using infinispan 14.0.27 version with rocksdb cache store.
Java.exe process working set memory is growing and java process get crash. It's due to native code.
Does any one facing same issue with huge data in cache store?

Infinispan cache store configuration is -

10 10485760 10485760 300 10485760 10 5

Below is summery -

Current thread (0x000002180908d060): JavaThread "blocking-thread--p3-t28" daemon [_thread_in_native, id=9772, stack(0x00000066d7600000,0x00000066d7700000)]

Stack: [0x00000066d7600000,0x00000066d7700000], sp=0x00000066d76fdd40, free space=1015k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [KERNELBASE.dll+0x349d9]
C [VCRUNTIME140.dll+0x6220]
C [librocksdbjni11785071052815030558.dll+0x502653]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 33050 org.rocksdb.RocksDB.put(J[BII[BIIJ)V (0 bytes) @ 0x000002183ee24438 [0x000002183ee243a0+0x0000000000000098]
J 44555 c2 java.util.concurrent.CompletableFuture$AsyncRun.run()V java.base@17.0.8 (61 bytes) @ 0x000002183fe9f23c [0x000002183fe9f140+0x00000000000000fc]
J 54353% c2 org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run()V (504 bytes) @ 0x00000218411c7f78 [0x00000218411c76e0+0x0000000000000898]
J 30756 c1 java.lang.Thread.run()V java.base@17.0.8 (17 bytes) @ 0x00000218389208fc [0x00000218389207c0+0x000000000000013c]
v ~StubRoutines::call_stub

[tristantarrant]

Since this is happening inside the native code, maybe you could try using a more recent version of org.rocksdb:rocksdbjni

[Satishshelake]

I am using infinispan-wildlfy-module-14.0.21.zip, which is downloaded from infinispan.
It contains rocksdbjni-7.1.2.jar.

[Satishshelake]

[tristantarrant]

Nice.

[Satishshelake]

It might be bug in infinispan rocksdb cache store. I have observed native meory consumption high more than 2GB (java native memory)
I have upgraded infinispan 14.0.24 ( Widlfly module distribution) with latest rocksdbjni-8.10.0. jar)

[Satishshelake]

Java.exe memory is getting increase. Also analysed process, native memory is increasing gradually.
Below is cache config -



10
10485760
256000
300
10485760
10
5

[tristantarrant]

If there is a bug on the Infinispan side, then it would be about not closing some rocksdb handle. @wburns knows best here

[Satishshelake]

I have upgraded rocksdbjni-8.10.0. jar in widfly module 14.0.24. But same issue persist. Also analysed heap and non heap memory. Native memory allocation increased gradually in java.exe process on widnows.

[wburns]

Looking at the code every reference to a Closeable object we use from RocksDB is either closed with a try finally block or via Flowable.using which would all guarantee these are closed. Maybe it is a setting within rocks DB itself that is growing with memory? In that past some customers have had issues that required reducing the RocksDB internal buffers. We specifically allow any user to override the RocksDB settings via properties on the cache configuration that start with database.<.name> for the database and data.<name> for the column families we create.

The guide talks about configuring these values at https://infinispan.org/docs/stable/titles/configuring/configuring.html#rocksdb-cache-store_persistence

[Satishshelake]

In previous version of inifnispan, My usecse works fine. but if I upgraded to infinispan 14, Memory relate dissue occured..

[wburns]

Yeah, so ISPN 9 did not have segmented rocks DB, so I am really guessing that is the cause here as the block size is being taken up per keyspace instead of once. So trying the 3 things I suggested in my previous comment should hopefully remedy the issue.

[Satishshelake]

Thank You for your reply. I will check it.

[Satishshelake]

I am using very extensive read & write on cache, I have tried below configuration.
Is 10KB block size enough or shall I reduce it as key size is UUID string for cache ?.

<local-cache name="TaskCache" statistics="false"> <locking acquire-timeout="60000"/> <persistence passivation="true"> <rocksdb-store path="C:\CacheStore\Data" preload="true" shared="false" purge="false" read-only="false" **block-size="10000"** cache-size="67108864" **segmented="false"**> <expiration path="C:\CacheStore\Expired"/> <property name="database.WAL_size_limit_MB">10</property> <property name="database.max_log_file_size">10485760</property> **<property name="data.write_buffer_size">10000</property>** <property name="database.WAL_ttl_seconds">300</property> <property name="database.max_total_wal_size">10485760</property> <property name="database.keep_log_file_num">10</property> <property name="data.max_write_buffer_number">5</property> </rocksdb-store> </persistence> <memory max-count="1000"/><indexing enabled="true" storage="filesystem" path="${java.io.tmpdir}/CacheIndexes"><indexed-entities><indexed-entity>com.test.cls.TaskElement</indexed-entity></indexed-entities><index-reader refresh-interval="0"/></indexing></local-cache>

[Satishshelake]

Still memory is getting increased by java.exe

Below are some observations -

"heap-memory-usage" => {
"init" => 268435456L,
"used" => 686875648L,
"committed" => 1342177280L,
"max" => 1342177280L
},
"non-heap-memory-usage" => {
"init" => 7667712L,
"used" => 402652112L,
"committed" => 414253056L,
"max" => 1224736768L
},

Java process memory -
java.exe - 2,130,408 K

[Satishshelake]

As I understood, Memory alocation is based on block size and cache size. Also native memory is based on buffer size parameter of rocksdb cache. @wburns block size is being taken up per keyspace instead of once, means does it allocatated block size as per cache key entry? I am bit confused about your comments.
Do we have any tools to analysis rocksdb native memory allocation in java.exe?