Add support for preserving source message timestamp #687
Conversation
This feature allows Brooklin to make use of Kafka message's source timestamp when sending the event to destination cluster. If the mirrored topic is configured with message.timestamp.type set to CreateTime, the intent is to have the timestamp supplied by the application. This feature will help achieving this intention.
|
@celiakung @ahmedahamid could you please review this PR? |
|
@sanjay24 Will do. |
|
thanks @ahmedahamid @somandal |
…entSourceTimestampFeature
|
What is the expected timeline for to review this change @ahmedahamid, @somandal? It would be a very useful feature. |
|
Will look into it by end of next week. @mateuszmrozewski |
datastream-kafka/src/main/java/com/linkedin/datastream/kafka/KafkaTransportProvider.java
Outdated
Show resolved
Hide resolved
datastream-kafka/src/main/java/com/linkedin/datastream/kafka/KafkaTransportProvider.java
Outdated
Show resolved
Hide resolved
…entSourceTimestampFeature
datastream-kafka/src/main/java/com/linkedin/datastream/kafka/KafkaTransportProvider.java
Outdated
Show resolved
Hide resolved
datastream-kafka/src/test/java/com/linkedin/datastream/kafka/TestKafkaTransportProvider.java
Outdated
Show resolved
Hide resolved
Added the following test-cases: - When preserveEventSourceTimestamp is True - When preserveEventSourceTimestamp is False - When preserveEventSourceTimestamp is not configured
|
@celiakung could you please check & approve the changes? Thanks |
|
There is still some more work required on this. I'll get back with my discovery and changes |
…entSourceTimestampFeature
…enabled Current implementation passes event's source timestamp only when its timestamp type is LogAppendTime, else it passes event's read time. This breaks the features. This commit changes the following: - Use event's source timestamp when the feature is enabled - Resort to default behavior when feature is disabled - Added associated test cases
|
@ahmedahamid @celiakung @somandal KafkaMirrorMakerConnectorTask was translating the records to pass the source event's timestamp only when its timestamp type was "LogAppendTime". Do you know why was it done? |
|
@ahmedahamid could we get moving on the review? Also, when is the next Brooklin release planned? |
|
@ahmedahamid Pinging you to get this reviewed and move ahead on this! |
This feature allows Brooklin to make use of Kafka message's source timestamp when sending the
event to destination cluster. If the mirrored topic is configured with message.timestamp.type
set to CreateTime, the intent is to have the timestamp supplied by the application. This feature
will help achieving this intention.