New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-35293][hive] Hive source supports dynamic parallelism inference #24764
[FLINK-35293][hive] Hive source supports dynamic parallelism inference #24764
Conversation
@zhuzhurk Could you please help to review this pr in your free time? Thanks. |
263d2ba
to
bffd8b1
Compare
@flinkbot run azure |
4ad7ba4
to
8d867f7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for creating this PR! @SinBex
I have a few comments. PTAL.
...nectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/HiveOptions.java
Show resolved
Hide resolved
...nectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/HiveOptions.java
Outdated
Show resolved
Hide resolved
...tor-hive/src/main/java/org/apache/flink/connectors/hive/HiveSourceDynamicFileEnumerator.java
Outdated
Show resolved
Hide resolved
...ve/src/main/java/org/apache/flink/connectors/hive/HiveStaticParallelismInferenceFactory.java
Outdated
Show resolved
Hide resolved
...e/src/main/java/org/apache/flink/connectors/hive/HiveDynamicParallelismInferenceFactory.java
Show resolved
Hide resolved
c5fd4e4
to
f99c425
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for addressing all the comments. @SinBex
The change looks quite good to me now. Nice work!
Could you squash the commits? I will merge the squashed commit once the CI gives green.
f99c425
to
ad9d5d0
Compare
What is the purpose of the change
FLIP-379 introduces dynamic source parallelism inference, which, compared to static inference, utilizes runtime information to more accurately determine the source parallelism. The FileSource already possesses the capability for dynamic parallelism inference. As a follow-up task to FLIP-379, this FLIP plans to implement the dynamic parallelism inference interface for HiveSource, and also switches the default static parallelism inference to dynamic parallelism inference.
Brief change log
Verifying this change
Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (no)Documentation