A while after upgrade to v3.12.0, unable to create documents: "Corruption: Compaction sees out-of-order keys"

[matcho]

My Environment

• ArangoDB Version: 3.12.0 Enterprise
• Deployment Mode: Single Server
• Deployment Strategy: sytemd
• Infrastructure: own
• Operating System: Ubuntu 22.04
• Total RAM in your machine: 64GB
• Disks in use: SSD
• Used Package: Ubuntu .deb

Component, Query & Data

Affected feature:
RocksDB integrity.

For some reason, after upgrading from 3.11.8 Enterprise to 3.12.0 Enterprise, the database went suddenly corrupt so that any attempt to add a new document to any collection triggers an error: Corruption: Compaction sees out-of-order keys.

Upgrade took place on April 8th. For a few days the database worked as usual, until April 11 (yesterday) when the error appeared.

Restarting the server had no effect. Triggering compaction on a collection then trying to write to it had no effect either.

Example with collection pro_queries:

127.0.0.1:8529@mydb> db.pro_queries.save({ "_key": "test_key_01", "test": true });
JavaScript exception in file '/usr/share/arangodb3/js/client/modules/@arangodb/arangosh.js' at 90,7: ArangoError 1100: Corruption: Compaction sees out-of-order keys.
!      throw error;
!      ^
stacktrace: ArangoError: Corruption: Compaction sees out-of-order keys.
    at exports.checkRequestResult (/usr/share/arangodb3/js/client/modules/@arangodb/arangosh.js:88:21)
    at ArangoCollection.save.ArangoCollection.insert (/usr/share/arangodb3/js/client/modules/@arangodb/arango-collection.js:869:14)
    at <shell command>:1:16

Dataset:
30 collections, having from 10 documents to 200M documents.

Size of your Dataset on disk:
700G

Steps to reproduce

1. Try to add a document to any collection, for ex. pro_queries, either using arangosh, aardvark or arangojs
2. Get the error Corruption: Compaction sees out-of-order keys
3. Try to run a compaction on the collection via arangosh, wait 1 hour, still get the error
4. Try to restart ArangoDB, still get the error

Problem:
Database is read-only: impossible to add a document to any collection.

Expected result:
A way to repair the corrupted DB?

Thank you,
Best regards

[dothebart]

Hi,
we are already working on this. An ICU upgrade had unexpected side effects a hot fix will soon be available.
At the moment the quickest known way to get out of this is to dump the database using arangodump and restore it to a 3.11.8.

[matcho]

Hi, thank you for your quick answer. We'll do that and wait for the fix.

[dothebart]

3.12.0.2 containing a fix for this has been released; closing as solved.