Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing few mirrormaker basics #633

Open
configOps opened this issue Aug 13, 2019 · 3 comments
Open

Missing few mirrormaker basics #633

configOps opened this issue Aug 13, 2019 · 3 comments
Labels

Comments

@configOps
Copy link

I have few issues :

  1. Topic metadata does not replicate to destination when brooklin datastream is setup. And neither it replicate modifications in same. What I can configure to do same ?
  2. How can I configure change in topic name in destination? Fo example, in source , I have a topic name first-topic and I have a datastream setup to mirror it in destination cluster with topic name as first-topic.global ?
@upendra1024
Copy link

upendra1024 commented Aug 20, 2019

I have few issues :

  1. Topic metadata does not replicate to destination when brooklin datastream is setup. And neither it replicate modifications in same. What I can configure to do same ?
  2. How can I configure change in topic name in destination? Fo example, in source , I have a topic name first-topic and I have a datastream setup to mirror it in destination cluster with topic name as first-topic.global ?

As @configOps mention bove in 1st point, I found/have some more issues/doubt with KafkaMirror :

  1. topic partitions are not creating on destination. I used REST endpoint to create a datastream with metadata "system.destination.identityPartitioningEnabled": "true".
  2. after creating datastream, new messages only replicating to destination.
  3. if partition increased on source side then destination should also get this change.
  4. auto topic creation is disabled in our datacenter. so on destination need to create topic first before pushing data.
  5. Will the kafkaMirroring feature support kafka version 0.10.0.1? Is there any limitation due to this old verssion?

This project is really nice, and lot of things we are learning from it.... Thanks a lot.

@OneCricketeer
Copy link

MirrorMaker 1 doesn't replicate topic metadata either (configs or partitions). It requires destination topic manually be made to match (which is easily done programmatically on your own).

I'm not aware of any rename feature in Brooklin, but I would like to think it does exist in some manner. What research have you done for the same?

Will the kafkaMirroring feature support kafka version 0.10.0.1? Is there any limitation due to this old version?

Can you please clarify your concerns here? What happens when you try it?

@sgoncharov1
Copy link

Hi,
First - thank you for this project, it is really great.

As a follow up to this question i wanted to ask if there are plans to add the ability of changing topic names (to mirror topics to different topic names in destination cluster).
I tried lookin at the documentation and the code, but did not manage to find such ability.
We currently need to use additional clusters to overcome this issue, and it would be great if we can do this with configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants