-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
COLLECT_LIST in KSQL returning duplicate results OR retaining previous results #10168
Comments
@AleksandarTokarev This looks like duplicates that happen due to the at-least-once semantics (ALOS) KSQL guarantees by default. Under ALOS, when a failure happens records might be written multiple times to an aggregate or to an output topic [1]. To solve your issue, you could try to run the application under exactly-once semantics (EOS). I am wondering whether in your specific it would suffice to use [1] https://docs.ksqldb.io/en/latest/operate-and-deploy/exactly-once-semantics/#at-least-once-semantics |
@cadonna Is it possible that the items are being retained with Also these ALOS semantics would be on the application level that writes to the |
Now, I realized that
|
Yes, time
These |
At the moment, I do not understand why the |
They do not exist because I have not put them above (and they do not exist in the beginning) - with time the items are being updated in |
I see. We need a minimal example to reproduce the situation, otherwise it is just guessing. I tried to repro it but was not able. |
What else can I provide in order to be more helpful? |
ok @cadonna some more followup on the issues.
It seems there is one more table that is backing up this
Also it seems that these issues have occurred when there was OOM in the KSQLDB - here is one of the logs when OOM occurred -
Few things that come on top of my mind:
Thanks a lot |
@AleksandarTokarev The best thing would be, if you could provide a reproduction with a docker-compose setup. You can find a docker-compose setup that you can re-use under the following link: https://ksqldb.io/quickstart.html#quickstart-content |
@cadonna we have added exactly once as a server parameter and it was good for like 2 weeks. Today we upgraded to ARM instances and during the upgrade there was an issue with the volumes/resources and the bug occurred again. Also i am not sure if i understand - but how are the |
@cadonna I think i have finally found a way how to reproduce this. docker compose file
Steps:
c) Use a
Do some insertions in the
After the last one - i seem to have ended with 2 Potatoes records in the resulting table - which is not what I want
DISCLAIMER: This does not happen always and for the same records - it is kinda intermittent. UPDATE 1: It seems this happens when same message key goes into different partitions. I wonder why and if this could happen in the failure scenarios (like OOM, volume issues, etc) |
Describe the bug
We have a use case where we have one KSQL Table and we have a push query - which queries the table continously and creates/feeds another table from it
In the table
ITEMS_TABLE
, we have the items using theid
as the primary key, while in the new push query, we want to create a table where we will group the items byuserId
- and the result will be a list.What we have noticed is - when everything works good after setup - there are no issues. But when KSQLDB has glitches - goes down and comes back up, or there are issues with it- we are getting duplicate (stale) items in the
ITEMS_PER_USER_TABLE
(we are not exactly sure why and how this happens).To Reproduce
Steps to reproduce the behavior, include:
0.29
In the table
ITEMS_PER_USER_TABLE
- we have the following itemsIn the
ITEMS_PER_USER_TABLE
- when all is good - we have the followingBut when things go wrong (eg. KSQL being down or some issues with it) - a bug occurs - and duplicate items start piling up (and they dont get cleared when
ITEMS_TABLE
gets more records.Expected behavior
The function
COLLECT_LIST
should not return duplicate items OR retain previous results - because in the previous table theid
is the key.Actual behaviour
Duplicate/Previous retained items are returned as part of the
COLLECT_LIST
. This should not be the case.It is interesting to note that not all the items are being duplicated/retained - sometimes some of them are - sometimes all of them.
This leads me to believe that the
COLLECT_LIST
works - but it retains some items that were previously in the list - instead of replacing them.The text was updated successfully, but these errors were encountered: