Distributed cache data consistency ?

[qaoru]

Hello,

I have to deploy an infinispan cluster and I'm doing some tests to compare a replicated vs a distributed setup but I have some troubles understanding how a distributed cache is managed.
I've created a distributed cache with owners = 3 (synchronous mode).
With an api call, I can see

    "required_minimum_number_of_nodes": 2

which seems legit to me, but if I shutdown all nodes but one, I get

    "required_minimum_number_of_nodes": 2

and the remaining node is able to write on the cache.
I'm used to applications that requires a quorum to write, I guess that's not the case for infinispan ?

What happens if the link between the nodes is down (but each instance still lives and there are writes on it for the same key), and then goes back up ? Something like an inner timestamp so when the nodes see each other again, the most recent value for a given key is propagated ?

Thanks in advance :)

[wburns]

The required_minimum_number_of_nodes is just telling you how many nodes you need to keep up to not lose any data. Did you by chance have a cluster of 4 nodes before? If so that is the only way I can see that you had to maintain a minimum of 2 nodes.

In regards to why it didn't change, the stats are only updated every 3 seconds by default, maybe it had not yet been updated? The required minimum should be 1 when the number of owners is >= number of nodes. Although this also is assuming you aren't using eviction which can also cause you to need additional nodes to ensure data is not lost.

In regards to quorum, the stat required_minimum_number_of_nodes has nothing to do with that as it just informational to tell how many nodes you have to keep up currently to not lose data. The feature you are looking for is partition handling which allows you to prevent writes and even reads when you have a sub cluster https://infinispan.org/docs/stable/titles/configuring/configuring.html#partition-handling

[qaoru]

In regards to why it didn't change, the stats are only updated every 3 seconds by default, maybe it had not yet been updated? The required minimum should be 1 when the number of owners is >= number of nodes. Although this also is assuming you aren't using eviction which can also cause you to need additional nodes to ensure data is not lost.

That was juste a wrong copy/paste, it was indeed set to 1 when the nodes were shutdown.

The required_minimum_number_of_nodes is just telling you how many nodes you need to keep up to not lose any data. Did you by chance have a cluster of 4 nodes before? If so that is the only way I can see that you had to maintain a minimum of 2 nodes.

Yup, that it was a 4 nodes cluster.

The feature you are looking for is partition handling which allows you to prevent writes and even reads when you have a sub cluster https://infinispan.org/docs/stable/titles/configuring/configuring.html#partition-handling

Exactly what I was looking for, thanks !