[Feature][Connector-V2] Support multi-table sink feature for paimon #5652 #6449

dailai · 2024-03-05T09:42:04Z

Purpose of this pull request

[Feature][Paimon] Support cdc write of paimon sink #6427
[Feature][Paimon] Support the streaming mode for sink #6342
[Feature][Paimon] Support save mode
[Feature][Paimon] Support multi-table sink feature #5652

Does this PR introduce any user-facing change?

no

How was this patch tested?

add new test.

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

Hisoka-X · 2024-03-05T10:25:35Z

Please open CI on your fork repository.

dailai · 2024-03-05T11:27:17Z

Please open CI on your fork repository.

done

Hisoka-X · 2024-03-06T03:36:27Z

.github/workflows/build_main.yml

    permissions:
      packages: write
    name: Run
-    uses: ./.github/workflows/backend.yml
+    uses: ./.github/notify_test_workflow.yml


Hi @dailai , please do not change the workflow file. It should work fine.

Yes, it was a mistake

Hisoka-X · 2024-03-06T07:25:04Z

Please try merge from dev to retrigger CI.

Hisoka-X · 2024-03-06T07:32:38Z

...on/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/data/PaimonTypeMapper.java

+
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class PaimonTypeMapper {


Please extends TypeConverter.

Please extends TypeConverter.

ok

Hisoka-X · 2024-03-06T07:34:20Z

...r-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/sink/PaimonSink.java

+            // Auto create if not exists the database and table for paimon
+            catalog.createDatabase(database, true);
+            TableSchema tableSchema = catalogTable.getTableSchema();
+            this.seaTunnelRowType = tableSchema.toPhysicalRowDataType();
+            Schema paimonTableSchema = SchemaUtil.toPaimonSchema(tableSchema);
+            catalog.createTable(identifier, paimonTableSchema, true);


Please move create table logic to SaveModeHandler part. You can refer

seatunnel/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SupportSaveMode.java

Line 23 in 6ec16ac

public interface SupportSaveMode {

Please move create table logic to SaveModeHandler part. You can refer

seatunnel/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SupportSaveMode.java

Line 23 in 6ec16ac

public interface SupportSaveMode {

done, please review!

Hisoka-X · 2024-03-06T07:35:02Z

seatunnel-shade/seatunnel-hadoop3-3.1.4-uber/pom.xml

+        <dependency>
+            <groupId>org.xerial.snappy</groupId>
+            <artifactId>snappy-java</artifactId>
+            <version>1.1.10.4</version>
+        </dependency>


Any reason for add this?

Any reason for add this?
The version of snnapy in seatunnel-shade/seatunnel-hadoop3-3.1.4-uber is confict with the version of sannpy built info flink-dis(flink1.16). If you submit tasks through flink mode, may cause this error.

I think we shoud not put seatunnel-hadoop3-3.1.4-uber into flink libs. It only used for zeta. With flink, we should use their own hadoop jar.

I think we shoud not put seatunnel-hadoop3-3.1.4-uber into flink libs. It only used for zeta. With flink, we should use their own hadoop jar.

I did not put the seatunnel-hadoop3-3.1.4-uber into flink libs. If we submit a job by the seatunnel's flink script , the flink will load the seatunnel-hadoop3-3.1.4-uber in classpath. For example, exec "start-seatunnel-flink-15-connector-v2.sh --config xxxx.job"(The source is mysql-cdc and sink is paimon )

@Hisoka-X please recheck

Thanks for ping me. I will check it this weekend.

[Feature][Paimon] Support cdc write of paimon sink apache#6427 [Feature][Paimon] Support auto create db and table for paimon sink

hailin0 · 2024-03-11T03:41:00Z

...el-connector-v2-e2e/connector-paimon-e2e/src/test/resources/mysql_cdc_sink_paimon_case1.conf

+}
+
+source {
+  MySQL-CDC {


You can use FakeSource instead of MySQL-CDC, the test case is more stable, please refer to https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-clickhouse-e2e/src/test/resources/clickhouse_sink_cdc_changelog_case2.conf#L27C3-L27C13

https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-clickhouse-e2e/src/test/resources/clickhouse_sink_cdc_changelog_case2.conf#L27C3-L27C13

Good idea! Ok, I will do it.

hailin0 · 2024-03-11T03:42:23Z

Please change the docs

dailai · 2024-03-11T08:50:25Z

Please change the docs
done

hailin0 · 2024-03-11T10:10:45Z

...seatunnel-connector-v2-e2e/connector-paimon-e2e/src/test/resources/mysql/server-gtids/my.cnf

@@ -0,0 +1,65 @@
+#


remove this file？

hailin0 · 2024-03-11T10:10:50Z

...unnel-e2e/seatunnel-connector-v2-e2e/connector-paimon-e2e/src/test/resources/mysql/setup.sql

@@ -0,0 +1,24 @@
+--


remove this file？

hailin0

LGTM
cc @TyrantLucifer

Carl-Zhou-CN · 2024-03-12T03:13:00Z

...r-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/sink/PaimonSink.java

-import static org.apache.seatunnel.connectors.seatunnel.paimon.config.PaimonConfig.HDFS_SITE_PATH;
-import static org.apache.seatunnel.connectors.seatunnel.paimon.config.PaimonConfig.TABLE;
-import static org.apache.seatunnel.connectors.seatunnel.paimon.config.PaimonConfig.WAREHOUSE;
+import static org.apache.seatunnel.api.table.factory.FactoryUtil.discoverFactory;

 @AutoService(SeaTunnelSink.class)


Do we need to remove this annotation?

Do we need to remove this annotation?

I have removed it.

Carl-Zhou-CN · 2024-03-12T03:18:46Z

seatunnel-e2e/seatunnel-connector-v2-e2e/connector-paimon-e2e/pom.xml

+        <dependencies>
+            <dependency>
+                <groupId>org.apache.seatunnel</groupId>
+                <artifactId>connector-jdbc</artifactId>


Can this dependency be removed?

Carl-Zhou-CN · 2024-03-12T03:24:25Z

...ctor-paimon-e2e/src/test/java/org/apache/seatunnel/e2e/connector/paimon/PaimonSinkCDCIT.java

+                "sh", "-c", "cd " + CATALOG_ROOT_DIR + " && tar -zxvf " + NAMESPACE_TAR);
+        try {
+            Process process = processBuilder.start();
+            // 等待命令执行完成


Please use English comments.

Carl-Zhou-CN · 2024-03-12T03:26:19Z

docs/zh/connector-v2/sink/Paimon.md

+- [x] [exactly-once](../../concept/connector-v2-features.md)
+
+## 连接器选项
+


There seem to be some problems with the format

There seem to be some problems with the format

I've already previewed it and there don't seem to be any formatting issues here.

Carl-Zhou-CN

LGTM

EricJoy2048 closed this Mar 5, 2024

EricJoy2048 reopened this Mar 5, 2024

dailai force-pushed the dev-support-cdc-on-paimon-connector branch from 2f7ccda to 013d68e Compare March 5, 2024 11:24

dailai force-pushed the dev-support-cdc-on-paimon-connector branch from 5e3fe5e to 7650e3f Compare March 6, 2024 03:35

Hisoka-X reviewed Mar 6, 2024

View reviewed changes

dailai force-pushed the dev-support-cdc-on-paimon-connector branch from 7650e3f to 23dbea9 Compare March 6, 2024 04:01

dailai requested a review from Hisoka-X March 6, 2024 04:12

Hisoka-X added the feature New feature label Mar 6, 2024

Hisoka-X reviewed Mar 6, 2024

View reviewed changes

dailai force-pushed the dev-support-cdc-on-paimon-connector branch from 23dbea9 to d1925e8 Compare March 6, 2024 08:12

dailai requested a review from Hisoka-X March 7, 2024 06:00

dailai force-pushed the dev-support-cdc-on-paimon-connector branch from 2e0943b to 6d44826 Compare March 7, 2024 09:12

dailai added 3 commits March 8, 2024 10:25

[Feature][Paimon] Support the streaming mode for sink apache#6342

2510eac

[Feature][Paimon] Support cdc write of paimon sink apache#6427 [Feature][Paimon] Support auto create db and table for paimon sink

[Improve][Paimon] PaimonTypeMapper implements TypeConverter

7b29d10

[Bufgix][Paimon] change catalog by paimon sink config

a70d696

dailai force-pushed the dev-support-cdc-on-paimon-connector branch from a003e52 to cd7c21a Compare March 8, 2024 02:26

[Feature][Paimon] paimon sink support savemode

5b32488

dailai force-pushed the dev-support-cdc-on-paimon-connector branch from cd7c21a to 5b32488 Compare March 8, 2024 02:44

[Feature][Paimon] Support multi-table sink

244d927

dailai changed the title ~~[Feature][Paimon] Support the streaming mode for sink #6342~~ [Feature][Connector-V2] Support multi-table sink feature #5652 Mar 11, 2024

dailai changed the title ~~[Feature][Connector-V2] Support multi-table sink feature #5652~~ [Feature][Connector-V2] Support multi-table sink feature for paimon #5652 Mar 11, 2024

hailin0 reviewed Mar 11, 2024

View reviewed changes

dailai requested a review from hailin0 March 11, 2024 08:50

dailai force-pushed the dev-support-cdc-on-paimon-connector branch from 317712e to bf918e4 Compare March 11, 2024 08:57

hailin0 reviewed Mar 11, 2024

View reviewed changes

dailai force-pushed the dev-support-cdc-on-paimon-connector branch from bf918e4 to 937c7aa Compare March 11, 2024 10:47

dailai requested a review from hailin0 March 11, 2024 10:48

hailin0 reviewed Mar 11, 2024

View reviewed changes

hailin0 previously approved these changes Mar 11, 2024

View reviewed changes

github-actions bot added approved reviewed labels Mar 11, 2024

Carl-Zhou-CN reviewed Mar 12, 2024

View reviewed changes

dailai dismissed hailin0’s stale review via 905eb44 March 12, 2024 04:35

github-actions bot removed approved reviewed labels Mar 12, 2024

dailai requested review from Carl-Zhou-CN and hailin0 March 12, 2024 04:45

dailai force-pushed the dev-support-cdc-on-paimon-connector branch 6 times, most recently from 8a1d736 to b5474a9 Compare March 12, 2024 07:35

[Improve][Paimon] remove unwanted dependencies

408784f

dailai force-pushed the dev-support-cdc-on-paimon-connector branch from b5474a9 to 408784f Compare March 12, 2024 08:09

[Improve][Paimon] remove unwanted code

b57b9cc

Carl-Zhou-CN approved these changes Mar 12, 2024

View reviewed changes

github-actions bot added approved reviewed labels Mar 12, 2024

Hisoka-X approved these changes Mar 19, 2024

View reviewed changes

Hisoka-X merged commit b0abbd2 into apache:dev Mar 19, 2024
8 checks passed

dailai deleted the dev-support-cdc-on-paimon-connector branch March 19, 2024 06:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Connector-V2] Support multi-table sink feature for paimon #5652 #6449

[Feature][Connector-V2] Support multi-table sink feature for paimon #5652 #6449

dailai commented Mar 5, 2024 •

edited

Hisoka-X commented Mar 5, 2024

dailai commented Mar 5, 2024

Hisoka-X Mar 6, 2024

dailai Mar 6, 2024

Hisoka-X commented Mar 6, 2024

Hisoka-X Mar 6, 2024

dailai Mar 6, 2024

Hisoka-X Mar 6, 2024

dailai Mar 8, 2024

Hisoka-X Mar 6, 2024

dailai Mar 6, 2024 •

edited

Hisoka-X Mar 6, 2024

dailai Mar 6, 2024 •

edited

hailin0 Mar 14, 2024

Hisoka-X Mar 14, 2024

hailin0 Mar 11, 2024

dailai Mar 11, 2024

hailin0 commented Mar 11, 2024

dailai commented Mar 11, 2024

hailin0 Mar 11, 2024

dailai Mar 11, 2024

hailin0 Mar 11, 2024

dailai Mar 11, 2024

hailin0 left a comment

Carl-Zhou-CN Mar 12, 2024

dailai Mar 12, 2024

Carl-Zhou-CN Mar 12, 2024

dailai Mar 12, 2024

Carl-Zhou-CN Mar 12, 2024

dailai Mar 12, 2024

Carl-Zhou-CN Mar 12, 2024

dailai Mar 12, 2024 •

edited

Carl-Zhou-CN left a comment


		import java.util.concurrent.atomic.AtomicInteger;

		public class PaimonTypeMapper {

		- [x] [exactly-once](../../concept/connector-v2-features.md)

		## 连接器选项

[Feature][Connector-V2] Support multi-table sink feature for paimon #5652 #6449

[Feature][Connector-V2] Support multi-table sink feature for paimon #5652 #6449

Conversation

dailai commented Mar 5, 2024 • edited

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Hisoka-X commented Mar 5, 2024

dailai commented Mar 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X commented Mar 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dailai Mar 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dailai Mar 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 commented Mar 11, 2024

dailai commented Mar 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hailin0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dailai Mar 12, 2024 • edited

Choose a reason for hiding this comment

Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

dailai commented Mar 5, 2024 •

edited

dailai Mar 6, 2024 •

edited

dailai Mar 6, 2024 •

edited

dailai Mar 12, 2024 •

edited