Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destinations: Refreshes: Track stream statuses in async framework #38075

Draft
wants to merge 1 commit into
base: edgao/catalog_parser_improvements
Choose a base branch
from
Draft

Destinations: Refreshes: Track stream statuses in async framework #38075

wants to merge 1 commit into from
+382 −132

Conversation

edgao
Copy link
Contributor

@edgao edgao commented May 8, 2024 •
edited

@vercel Vercel
Copy link

vercel bot commented May 8, 2024 •
edited

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview May 20, 2024 8:24pm

@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label May 8, 2024
@edgao Graphite App
Copy link
Contributor Author

edgao commented May 8, 2024 •
edited

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @edgao and the rest of your teammates on Graphite Graphite

@edgao edgao force-pushed the destinations_refreshes_cdk_changes branch from 50c1e4b to e857ce6 Compare May 8, 2024 21:13
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from fdf57a0 to 4514243 Compare May 8, 2024 21:13
@edgao edgao mentioned this pull request May 8, 2024
Draft
@edgao edgao force-pushed the destinations_refreshes_cdk_changes branch from e857ce6 to 982948e Compare May 8, 2024 21:21
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch 2 times, most recently from 7eb8203 to 2a3f5d9 Compare May 8, 2024 21:38
@edgao edgao force-pushed the destinations_refreshes_cdk_changes branch from 982948e to 0fd05a7 Compare May 8, 2024 21:39
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 2a3f5d9 to 7753251 Compare May 8, 2024 21:39
@edgao edgao force-pushed the destinations_refreshes_cdk_changes branch from 0fd05a7 to 0d4af89 Compare May 8, 2024 21:42
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 7753251 to 757297c Compare May 8, 2024 21:42
@edgao edgao force-pushed the destinations_refreshes_cdk_changes branch from 0d4af89 to 52b9185 Compare May 8, 2024 21:43
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 757297c to 21d308b Compare May 8, 2024 21:43
@edgao edgao force-pushed the destinations_refreshes_cdk_changes branch from 52b9185 to ccf601a Compare May 8, 2024 21:45
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 21d308b to 34d6f0b Compare May 8, 2024 21:46
@edgao edgao force-pushed the destinations_refreshes_cdk_changes branch from ccf601a to 6839b3f Compare May 8, 2024 21:51
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 34d6f0b to cf93b76 Compare May 8, 2024 21:52
edgao
edgao commented May 8, 2024
View reviewed changes
...src/main/kotlin/io/airbyte/cdk/integrations/destination/async/model/PartialAirbyteMessage.kt
@@ -26,6 +27,12 @@ class PartialAirbyteMessage {
@JsonProperty("state")
var state: PartialAirbyteStateMessage? = null

@get:JsonProperty("trace")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea if these are even necessary, literally just copypasting from other fields. Feels like they shouldn't be? since trace doesn't have any spaces/underscores/etc

@edgao edgao force-pushed the destinations_refreshes_cdk_changes branch from 6839b3f to d361d45 Compare May 8, 2024 22:00
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from cf93b76 to 5878c44 Compare May 8, 2024 22:00
@edgao edgao requested a review from gisripa May 8, 2024 23:00
@edgao edgao mentioned this pull request May 9, 2024
Closed
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 5878c44 to de8e132 Compare May 9, 2024 16:30
@edgao edgao changed the base branch from destinations_refreshes_cdk_changes to edgao/better_default_namespace_tracking May 9, 2024 16:31
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch 2 times, most recently from 136594e to 3c20271 Compare May 9, 2024 18:12
edgao
edgao commented May 9, 2024
View reviewed changes
...n/kotlin/io/airbyte/cdk/integrations/destination/buffered_stream_consumer/OnCloseFunction.kt
import io.airbyte.protocol.models.v0.StreamDescriptor

/**
* Interface allowing destination to specify clean up logic that must be executed after all
* record-related logic has finished.
*
* The map of StreamSyncSummaries MUST be non-null, but MAY be empty. Streams not present in the map
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was only needed for BufferedStreamConsumer, which should no longer be relevant to DV2 destinations...

@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 415f532 to 337722a Compare May 10, 2024 21:24
@edgao edgao force-pushed the destinations_refreshes_cdk_changes branch from 090fa6b to df8e461 Compare May 10, 2024 22:18
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 337722a to be07e84 Compare May 10, 2024 22:18
@edgao edgao force-pushed the destinations_refreshes_cdk_changes branch from df8e461 to af71cd6 Compare May 13, 2024 18:17
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch 3 times, most recently from 2d365dd to 9f9838b Compare May 17, 2024 18:42
@edgao edgao changed the base branch from destinations_refreshes_cdk_changes to edgao/catalog_parser_improvements May 17, 2024 18:42
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 355b8e5 to d5e885f Compare May 17, 2024 18:47
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 9f9838b to 161dfee Compare May 17, 2024 18:47
@gisripa gisripa force-pushed the edgao/catalog_parser_improvements branch from d5e885f to 02caf06 Compare May 17, 2024 20:08
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 02caf06 to 17ecb7f Compare May 17, 2024 21:10
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 161dfee to 6a02c16 Compare May 17, 2024 21:10
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 17ecb7f to 14806d4 Compare May 17, 2024 21:35
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 6a02c16 to e01ec42 Compare May 17, 2024 21:35
@edgao edgao mentioned this pull request May 17, 2024
Draft
2 tasks
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 14806d4 to 96f4b23 Compare May 17, 2024 22:30
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from e01ec42 to 0b1f7ad Compare May 17, 2024 22:30
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 96f4b23 to c0c6173 Compare May 17, 2024 23:21
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 0b1f7ad to e02cc57 Compare May 17, 2024 23:21
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from c0c6173 to 3498f16 Compare May 20, 2024 15:42
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch 2 times, most recently from fc44832 to 1f95ce3 Compare May 20, 2024 16:23
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 3498f16 to 4f56d93 Compare May 20, 2024 16:33
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 1f95ce3 to 6f4584b Compare May 20, 2024 16:33
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 4f56d93 to 0855f3e Compare May 20, 2024 18:46
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 6f4584b to 9613270 Compare May 20, 2024 18:46
@edgao edgao force-pushed the edgao/catalog_parser_improvements branch from 0855f3e to 4d48e79 Compare May 20, 2024 20:24
@edgao edgao force-pushed the edgao/async_tracking_stream_statuses branch from 9613270 to a906976 Compare May 20, 2024 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gisripa gisripa Awaiting requested review from gisripa

Assignees
No one assigned
Labels
CDK Connector Development Kit
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@edgao @gisripa @octavia-squidington-iii