Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug(duplication):some nodes can not send duplication data in master when big difference in data size #1840

Open
ninsmiracle opened this issue Jan 9, 2024 · 0 comments
Labels
type/bug This issue reports a bug.

Comments

@ninsmiracle
Copy link
Contributor

ninsmiracle commented Jan 9, 2024

Bug Report

  1. What did you do?
    In online production situation. Some nodes in matser cluster can not send dupcalition data to backup cluster.
    image

  2. What version of Pegasus are you using?
    pegasus2.4

3.Why?
The root cause is that the master cluster sent a write RPC to the backup cluster, and the request body size exceeded the max_allowed_write_size set by the backup cluster. The reason why the master cluster sends this illegal data is due to the mechanism in the process of packaging and sending mutations:

  • the master cluster traverses the writes received in a time period
  • checks whether all writes have been traversed or whether the current batch bytes are already greater than the duplicate_log_batch_bytes set in the cluster hot standby configuration (config.ini).
  • If it is not greater than the batch size, then two mutations are combined into one batch.

This can happen if the data length distribution of the table is too large.

For example:
The length of the first mutation A is 200byte. This naturally is to be combined with the next piece.
But the length of the next mutation B is 1048376(1048576-200) .
At this point, A and B are already in a batch, RPC will be sent out smoothly, but the standby cluster can not accept such a large write, and the standby cluster throws an ERR error. The master cluster is delayed recovery, throwing an ERR error.

@ninsmiracle ninsmiracle added the type/bug This issue reports a bug. label Jan 9, 2024
empiredan pushed a commit that referenced this issue Mar 27, 2024
…ax_allowed_write_size` (#1841)

#1840

Add config `dup_max_allowed_write_size` to restrict the size of dup request in case that the request is reject by remote cluster.

New configuration is added:
```diff
[replication]
dup_max_allowed_write_size = 1048576
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug This issue reports a bug.
Projects
None yet
Development

No branches or pull requests

1 participant