Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create new client API for atomic counters based on Segment Attributes #7302

Open
RaulGracia opened this issue Oct 31, 2023 · 0 comments
Open
Assignees

Comments

@RaulGracia
Copy link
Contributor

Is your feature request related to a problem? Please describe.
We propose adding a new API methods in the client for allowing users to create and manage atomic counters associated to a stream. As an example of when this could be useful, in some processing scenarios, there could be processing tasks that use as input the output of other tasks (i.e., pipelines). If we think in this scenario in the context of Pravega, there would be writer and reader tasks writing and consuming data from a stream. However, given that we do not necessarily refer to stream processing tasks, readers may need to know when writers have completed their work so they can stop reading further events from a stream and complete their processing. In this scenario, having a "counter" that writers and readers could update and check, would be interesting to provide a simple way of synchronizing groups of tasks. Note that this is just an example, but there could be other cases in which atomic counters could be useful as a client-facing primitive.

Describe the solution you'd like
Segment Attributes can offer multiple types of updates at the Segment Store level. Once of them is the Accumulate update type, which adds the value passed in the WireCommand to the one existing in the server, if any. This primitive is exactly what we need to implement a client-facing atomic counter with minimal changes at the server side.

Describe alternatives you've considered
Certainly, the proposed feature could be implemented using the StateSychronizer. However, thanks to the attribute update types, we could implement counters in a much simpler way, both for the client and the server perspective. For simplicity, just exposing attribute update types at the WireCommand level solves the problem, and opens up for adding more attribute update types that could be automatically exposed as well.

Additional context
The need for this feature appeared when doing experiments with Pravega and serverless frameworks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant