Handle overflow in rd_buf_write_remains #4689
Conversation
anchitj
changed the title
anchitj
force-pushed
the
dev_rd_buf_write_remains
branch
from
a1622dd
to
66fa00b
Compare
|
Hi @emasab, Could you please review and approve this PR? It's critical for us as it addresses a blocking issue with the confluent-kafka-dotnet library update. Your approval is eagerly awaited. |
tests/0011-produce_batch.c
Outdated
| @@ -106,7 +115,14 @@ static void test_single_partition(void) { | |||
| TEST_SAY("test_single_partition: Created kafka instance %s\n", | |||
| rd_kafka_name(rk)); | |||
|
|
|||
| rkt = rd_kafka_topic_new(rk, test_mk_topic_name("0011", 0), topic_conf); | |||
| topicSuffix = (char *)malloc(topicSuffixLength + 1 * sizeof(char)); | |||
There was a problem hiding this comment.
This change can be removed as it's not necessary for reproducing the issue
There was a problem hiding this comment.
We need topicSuffix because with the combination of both clientID and topicSuffix we were able to see this border condition in the segment.
There was a problem hiding this comment.
I had verified it happens with just that client id, because the segment is removed when the varint is changed without reducing the erased bytes
tests/0011-produce_batch.c
Outdated
| @@ -178,6 +194,9 @@ static void test_single_partition(void) { | |||
| TEST_SAY("Destroying kafka instance %s\n", rd_kafka_name(rk)); | |||
| rd_kafka_destroy(rk); | |||
|
|
|||
| free(clientId); | |||
| free(topicSuffix); | |||
|
There also a problem in My proposed solution is to add a |
Makes sense. I'll create a separate PR to handle that. |
|
@anchitj better to do it here, as it's basically the same thing that is missing |
After the implementation of KIP-320, TopicCnt was sent as a varint in the metadata request. This change caused an overflow issue due to the use of
rd_buf_eraseto erase the remaining bytes. The overflow occurred in the calculationrbuf->rbuf_size - (rbuf->rbuf_len + rbuf->rbuf_erased)becauserbuf->rbuf_erasedwas non-zero when the buffer was almost full, leading to an overflow condition.As a result, for certain configurations that caused the buffer to be nearly full, the client was unable to send metadata requests, and it also caused crashes in the .NET client.