Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add enhanced options and documentation for streaming data generation #115

Open
ronanstokes-db opened this issue Oct 4, 2022 · 0 comments · May be fixed by #127
Open

Add enhanced options and documentation for streaming data generation #115

ronanstokes-db opened this issue Oct 4, 2022 · 0 comments · May be fixed by #127
Assignees
Labels
Milestone

Comments

@ronanstokes-db
Copy link
Contributor

ronanstokes-db commented Oct 4, 2022

Issue to track:

1 - changes to streaming documentation to add
a) Delta Live Tables integration information,
b) example of generation of sliding event time windows
c) example of generation of simple IOT data with timestamps
d) example of generation of late arriving data with the above

2 - add options to simplify generation of the above
The above behavior is supported in the current version, but it would be useful to include options to simplify the generation of streaming data sets

2a - add options for
- ageLimit - ignore messages older than n seconds. This helps when benchmarking with trigger once using a rate
stream when significant time has elapsed between runs (otherwise there can be a large message backlog)
3 - support rate-micro-batch source

@ronanstokes-db ronanstokes-db added the enhancement New feature or request label Oct 4, 2022
@ronanstokes-db ronanstokes-db self-assigned this Oct 4, 2022
@ronanstokes-db ronanstokes-db linked a pull request Oct 21, 2022 that will close this issue
12 tasks
@ronanstokes-db ronanstokes-db added this to the v0.2.2 milestone Oct 22, 2022
@ronanstokes-db ronanstokes-db modified the milestones: v0.2.2, v0.3.1 Nov 29, 2022
@ronanstokes-db ronanstokes-db modified the milestones: v0.3.1, v0.4.0 Mar 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant