Setting database for performance

[cezar-radan]

Hi
I performed some load tests over an immudb database and I didn't get good scores, therefore I decided to ask for guidance related to the database settings / immudb service settings, in case of heavy load.

Testing environment:
test machine
CPU - Intel® Core™ i7-10750H CPU @ 2.60GHz × 12
MEM - 32,0 GiB
SSD disk
immudb (v 1.3.2) - docker container running locally, on test machine
immuclient (v 1.3.2) - used in a golang application running locally, on test machine

Test scenario is:
Step1 - create some entries (
- create client + open session
- prepare 10 key-values
- insert key-values using ExecAll (precondition key must not exist)
)
Step2 - update entries created in Step1 (
- create client + open session
- prepare new values for the 10 keys
- update keys using ExecAll (precondition key must exist))
)
Step3 - get entries created in Step2(
- create client + open session
- retrieve key-values using GetAll
)

Test Results:

• Serial running, 1 process that cycle (Step1 -> Step2 -> Step3) 1000 times.
  In this case the runtime is acceptable, example:
  - create client + open session = 50 - 62 milliseconds
  - ExecAll = 10 - 13 milliseconds
  - GetAll = 5 - 10 milliseconds

• Concurrent running, 100 process that cycle (Step1 -> Step2 -> Step3) 1000 times.
  In this case the runtime is not so good, example:
  - create client + open session = 600 - 800 milliseconds
  - ExecAll = 6500 - 9000 milliseconds
  - GetAll = 100 - 150 milliseconds

So, it is very clear that in a concurrent environment MY immudb doesn't respond as expected.

I had a look over your mentions and over your code :
https://github.com/codenotary/immudb#performance-figures
https://github.com/codenotary/immudb/tree/master/tools/testing/stress_tool_test_kv
https://github.com/codenotary/immudb-tools/tree/main/stresstest
and, it seems there are various database parameters/configs that could be changed in order to improve the performance:
maxConcurrency (30 <-> 100)
maxIOConcurrency (1 <-> 100)
indexSettings.cleanupPercentage (0.1 <-> 10)
.... so on
I have run tests with various database parameters values (many combinations) but, unfortunately, I didn't obtain any improvement.

Please, could you provide some guidance for database settings/configs in order to obtain a better performance (heavy inserts) ?
I am pretty sure, in my case, the indexing process should be configured with care too.

Thank you very much.

[jeroiraz]

Hi @cezar-radan, thanks for reaching out. Your description is very well described :)

Currently, to get benefits from concurrent writers, transactions must be read-only e.g. Set, SetAll.

ExecAll is acquiring a lock, thus why increasing the number of writers is not producing a significative benefit.

Multi-version Concurrency Control (MVCC) is partially implemented but will be fully implemented in the near future. With that, it will be possible to obtain benefits from concurrency when the transaction does not only involves writes but read and writes.

[cezar-radan]

Hi @byo. I removed the Step2 and Step3 preconditions but, unfortunately, I didn't get any significant runtime improvement.

[cezar-radan]

Update - What I have noticed.
My test uses concurrent goroutines (100), and each goroutine manages itself the connection to immudb (create client -> open session -> close session).
I have changed a little bit and I have created a single/unique client+session.
If all goroutines use this unique connection (client + session) the runtime decreases by 3 times, and this is a tremendous improvement.

[byo]

@cezar-radan that's a very interesting finding, what was the flow of OpenSession / CloseSession before switching to a single client? Was the session opened / closed for a single round of changes? Even with multiple sessions it should work faster assuming that the session is opened once, reused between rounds and only closed before the test is finished. If it is not working like that then it is definitely something to investigate on immudb side.

[cezar-radan]

Before the unique client+session, each goroutine opens (once) a client+session, do all the work and, at the end, close the session.
So, 100 concurrent workers (goroutines) means 100 connections (client+session) open at the same time.

[cezar-radan]

I have tried, also, the version where a goroutine deals with multiple connections. This means for each round/step a goroutine performs create client + open session + close session, but the runtime was not really encouraging.
After that, I have used a single connection for each goroutine and I have noticed a good improvement.
This determine me to create a unique connection (client+session) and to use it for all goroutines.