-
Notifications
You must be signed in to change notification settings - Fork 619
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Embed the connector into EventMesh Server for DTS #4787
Comments
good idea, connector can run in multi-mode for different scenarios. |
Design details Add InnerLocalServer as an embedded local Server whose primary role is to create an IInnerPubSubService while obtaining all dependent eventmesh-connectors So all the modification points are as follows:
Required further modification points:
Use method after modification: 设计详情 新增 InnerLocalServer 作为内嵌的本地Server ,其主要作用是创建一个IInnerPubSubService,同时获取所有依赖的 eventmesh-connectors 这样所有的修改点如下:
所需要的继续修改点:
修改完成后的使用方法: i will pr later |
Don’t rush to submit the PR. We need to discuss it well, define the interfaces and concepts, and design the architecture. |
This PR is not for merger, it's for discussion, otherwise I'm afraid I won't make it clear |
Design direction for this issue at regular community meetings: https://docs.qq.com/doc/DQkdlV0ZhdWZGRXFB, 2024-03-14 Point 7 |
I'm sorry, I didn't understand some descriptions. Isn't there three existing nettyserver? One implementation of tcp, one implementation of http, and another implementation of grpc, [http server can be transformed to the connector, etc.], this is not understood, [connector is not limited to the process level, can be thread level] I think this design is very reasonable, I think the design is not perfect. [connector currently passes through too many nodes, considering to optimize memory mode storage], the existing connector does not pass tcp to runtime, and then the runtime carries out subsequent processing. My solution is to change the original network communication to method call, that is, memory processing. As for switching to distruptor, don't you need to write an eventmesh-storage-plugin? Or does the connector put in the distruptor and the runtime gets it from the distruptor? I haven't learned [filter\transformer] here, [connector(as a standalone function)], isn't this issue about how to integrate connector into runtime,How come independent again? |
Search before asking
Feature Request
Currently, the
connector
andeventmesh-runtime
run separately. At the same time, theconnector
integrateseventmesh-sdk
to communicate witheventmesh-runtime
, and the data synchronization process will go through aneventmesh storage
. The synchronized data passes through multiple nodes and undergoes multiple serialization and deserialization operations, which has a great loss on the efficiency of synchronized data. We hope to embed theconnector
into theeventmesh-runtime
and optimize theeventmesh-runtime
memory storage mode. The source/sink connector can complete data flow and synchronization through memory storage ineventmesh-runtime
, improving the efficiency of synchronizing data.Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: