Potential explanation for empty cache issue

[vmgbrown]

Per earlier discussion, our application exhibited reduced performance after upgrading from WildFly 20 (Infinispan 10.1) to WildFly 26 (Infinispan 12.1). It seems that the issue may have been due to our use of native SQL queries in conjunction with L2 caches. By default, Hibernate will invalidate all caches when a native update or delete query is executed. We are now working around this issue by calling addSynchronizedQuerySpace() where necessary. However, we are still trying to understand why this was not a problem in earlier versions. I am wondering if it might be related to this commit:

ISPN-12787 Non-transactional cache needs to be invalidated after commit on JPQL update/delete operation

This change was merged to 12.1 and could potentially explain the behavior we are seeing.

Would it be possible to see the issue description for ISPN-12787? I was not able to find it anywhere.

[vmgbrown]

Found the issue description: https://issues.redhat.com/browse/ISPN-12787

It doesn't directly explain the behavior, but still seems related. Would be interested in hearing from @tristantarrant or @wburns on this.

[wburns]

The description is on the JIRA although it might be a bit hard to read. EAP/Wildfly changed how they called to invalidate caches and the QE team found that due to that there was stale data in the 2LC as the cache was cleared eagerly before the commit. This JIRA didn't cause more clears to be done it just made it so the clear was done at the commit time. This JIRA is likely not the culprit, but since EAP/Wildlfy changed how they invoked the invalidation step, there may be a change there that is causing this.

@pferraro may know more as well.

[vmgbrown]

EAP/Wildlfy changed how they invoked the invalidation step

Well, that certainly sounds related. :-) Thanks for the heads-up - any thoughts @pferraro?

[pferraro]

I'm not sure I fully understand the problem. Native update/delete queries are not generally compatible with read/write 2LC regions hence the need to conservatively clear these cache regions as a consequence.
AIUI, this scenario was prone to dirty reads in earlier versions, hence the change in default behavior.

[vmgbrown]

We are basically trying to understand why the call to addSynchronizedQuerySpace() was not required in WildFly 20/Infinispan 10.1 but is required in WildFly 26/Infinispan 12.1. Obviously the call should have been there all along, but we would like to know what changed between 20 and 26 to make the issue more apparent.

[pferraro]

Are you having any issues using this?

[vmgbrown]

No - see my follow-up comment above. We are now seeing good performance with addSynchronizedQuerySpace() in WildFly 26. But we also saw good performance in WildFly 20 without addSynchronizedQuerySpace(). Just trying to reconcile what might have changed.

[pferraro]

I don't recall the specific fix that precipitated https://hibernate.atlassian.net/browse/HHH-14325

[vmgbrown]

Thanks - I'll take a look at that ticket.

Interestingly, we are using the same version of Hibernate (5.4) in both versions. Are you aware of anything that might have changed outside of Hibernate between WildFly 20 and 26 that might affect L2 cache invalidation?

[vmgbrown]

We found that there were actually two issues at play here. One was the use of native update queries without calls to addSynchronizedQuerySpace(). This caused Hibernate to invalidate all L2 caches every time a native query was executed. As a result, the application had to reload entities from the DB more often than necessary, decreasing performance. We are now calling addSynchronizedQuerySpace() as needed and have seen a significant improvement in performance.

The other problem was that, in some cases, the native queries were being executed outside of a transaction. By default, Hibernate does not allow this and will generate an exception. However, we had apparently set the hibernate.allow_update_outside_transaction property to true, which prevents the exception from being thrown.

Unfortunately, this can cause an L2 cache to become permanently locked and refuse to accept new entries. The calls to lock and clear the cache occurs when a native query's executeUpdate() method is called. However, the call to unlock the cache does not occur until the session's transaction commits. If there is no transaction, there is no commit, and the cache remains locked indefinitely.

This is not an Infinispan issue; however, I wanted to share this information in case it is of interest to the developers (@wburns and @tristantarrant) or other users.