Enable raft message batching (#1519)

[ffuugoo]

This PR enables built-in raft message batching. (See #1519.)

All Submissions:

• Contributions should target the dev branch. Did you create your branch from dev?
• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

1. Does your submission pass tests?
2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
3. Have you checked your code using cargo clippy --all --all-features command?

Changes to Core Features:

• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your core changes, as applicable?
• Have you successfully ran tests with your changes locally?

[agourlay]

Could you elaborate a bit on the observable effects of this change? 📈

[ffuugoo]

Could you elaborate a bit on the observable effects of this change? 📈

Slight (and more stable) speedup on top of #3931. E.g., instead of sending 100 separate messages (doing 100 sequential gRPC calls) we send 1 batched message (doing 1 gRPC call).

IIRC, result were something like ~2500-2800 ops / 5 seconds with #3931 and ~2900-3000 ops / 5 seconds with batching enabled.

The only possible concern would be to check if we have any limit on maximal size of a single gRPC message, maybe we should increase it.

[timvisee]

The only possible concern would be to check if we have any limit on maximal size of a single gRPC message, maybe we should increase it.

I thought we didn't have a limit on the internal API, but we should confirm it.

[ffuugoo]

The only possible concern would be to check if we have any limit on maximal size of a single gRPC message, maybe we should increase it.

I thought we didn't have a limit on the internal API, but we should confirm it.

Seems like we don't.

[generall]

I don't think would I have any insights without giving it a run on CI & chaos