Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-34977][API] Introduce State Access on DataStream API V2 #24725

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

jeyhunkarimov
Copy link
Contributor

@jeyhunkarimov jeyhunkarimov commented Apr 26, 2024

What is the purpose of the change

This PR introduces a series of sub-PRs to support state access on Datastream API v2 (FLIP-433)

Brief change log

  • Introduce generic type descriptors. They are used to generate TypeInformation instances
  • Introduce state declarations. They represent parametrized declaration of different states, such as List, Map, Broadcast state
  • Introduce interfaces for declaring (via usesStates()) of their first input and for their both first and second input (via usesStatesForSideInput())
  • Bridge StateManager getState() with underlying state infrastructure
  • Support compile-time checks for state incompatibilities

Verifying this change

This change is covered by tests:

  • TypeDescriptionUtilsTest
  • DefaultStateManagerTest
  • BroadcastStreamImplTest
  • GlobalStreamImplTest
  • KeyedPartitionStreamImplTest
  • NonKeyedPartitionStreamImplTest

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)

@flinkbot
Copy link
Collaborator

flinkbot commented Apr 26, 2024

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@jeyhunkarimov jeyhunkarimov force-pushed the FLINK-34977 branch 10 times, most recently from 24ece2b to 8dcaa92 Compare May 3, 2024 11:36
@jeyhunkarimov jeyhunkarimov force-pushed the FLINK-34977 branch 4 times, most recently from 01c49c9 to 186e36a Compare May 4, 2024 06:34
@jeyhunkarimov jeyhunkarimov changed the title [WIP][FLINK-34977][API] Reorganize classes between core and core-api modules [WIP][FLINK-34977][API] Introduce State Access on DataStream API V2 May 8, 2024
@WencongLiu
Copy link
Contributor

WencongLiu commented May 13, 2024

Hello @jeyhunkarimov, have you completed this pull request?I'd like to review it. BTW, you should add the introduction of this pr.

@jeyhunkarimov
Copy link
Contributor Author

Hello @jeyhunkarimov, have you completed this pull request?I'd like to review it. BTW, you should add the introduction of this pr.

Hi @WencongLiu the PR is theorethically ready for review. I did not marked as ready since it needs to wait until FLIP-410 is merged (that is why I rebased it on top of FLIP-410 and not master).
Would be great if you have time to review this PR. Thanks

@jeyhunkarimov jeyhunkarimov changed the title [WIP][FLINK-34977][API] Introduce State Access on DataStream API V2 [FLINK-34977][API] Introduce State Access on DataStream API V2 May 13, 2024
@jeyhunkarimov jeyhunkarimov marked this pull request as ready for review May 13, 2024 12:16
@jeyhunkarimov jeyhunkarimov changed the title [FLINK-34977][API] Introduce State Access on DataStream API V2 [WIP][FLINK-34977][API] Introduce State Access on DataStream API V2 May 22, 2024
@jeyhunkarimov jeyhunkarimov changed the title [WIP][FLINK-34977][API] Introduce State Access on DataStream API V2 [FLINK-34977][API] Introduce State Access on DataStream API V2 May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants