Bug: The data can be normally parsed, but the downstream cannot retrieve data from the queue.

[hj2016]

Running script:

{
"version": "1.4.0",
"trace": 0,
"source": [
{
"alias": "S1",
"name": "O112A",
"reader": {
"type": "online",
"user": "olruser",
"password": "olruser2023_#",
"server": "//xxx.xxx.xxx.xxx:1521/DBCENTER"
},
"format": {
"type": "json"
},
"state": {
"type": "disk",
"path": "/opt/OpenLogReplicator/checkpoint"
},
"memory-min-mb": 64,
"memory-max-mb": 1024,
"filter": {
"table": [
{"owner": "KLBR", "table": "HUDI_TEST1","key":"ID,TEST_STR,TEST_DOUBLE,TEST_TIMESTAMP,SOURCE_LASTUPDATEDTTM"}
]
}
}
],
"target": [
{
"alias": "K1",
"source": "S1",
"writer": {
"type": "kafka",
"topic": "open_log_replicator_topic",
"properties": {
"bootstrap.servers": "xxx.xxx.xxx.xxx:xxx"
}
}
}
]
}

Problem phenomenon: I found that after simulating OpenLogReplicator's abnormal exit and restarting from the checkpoint, it runs for a while and I found that the upstream can parse the data normally, but the downstream queue cannot get data to send to Kafka. I debugged and found that the upstream is parsing normally, as shown in the figure below:

The downstream sendMessage breakpoint cannot be entered and there are no related logs, and Kafka also does not have related data.

Hasn't it already been pushed to the queue here, there might be a problem somewhere?

It seems that there is some additional data following the JSON data.

[hj2016]

I'm a Java programmer, and I don't quite understand the use of C++ queues. Could you please help clarify, maybe I can fix this problem.

[hj2016]

I understand now that the code is because the Kafka method doesn't have a callback, and the queue and memory are not released. But the parsed JSON still has garbled text, which is a problem.

[bersler]

Hi, @hj2016 I need to in-house reproduce this bug. Then make a test for that and make a fix. I did not have time for that. I will try to look at it the next few days. Thank you for reporting!

[bersler]

Can you please provide reproduction code:

1. take docker image for OLR + Oracle (preferred: https://github.com/bersler/OpenLogReplicator-tutorials so that the fault can be reproduced on Oracle XE)
2. set configuration of OLR to ... (preferred: json file)
3. run some SQL commands, restart, whatever else
4. achieve fault

or alternatively for other errors:

1. OLR configuration file
2. OLR checkpoint file set
3. set of redo log files which contain error
4. achieve fault

[bersler]

I still am unable to reproduce the issue, I plan to close this issue if nothing changes till end of may.