Destinations: Refreshes: Track stream statuses in async framework #38075
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
Contributor Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
edgao commented May 8, 2024
...n/io/airbyte/cdk/integrations/destination/buffered_stream_consumer/BufferedStreamConsumer.kt Outdated Show resolved Hide resolved
edgao force-pushed the destinations_refreshes_cdk_changes branch from 50c1e4b to e857ce6 Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from fdf57a0 to 4514243 Compare edgao force-pushed the destinations_refreshes_cdk_changes branch from e857ce6 to 982948e Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch 2 times, most recently from 7eb8203 to 2a3f5d9 Compare edgao force-pushed the destinations_refreshes_cdk_changes branch from 982948e to 0fd05a7 Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from 2a3f5d9 to 7753251 Compare edgao force-pushed the destinations_refreshes_cdk_changes branch from 0fd05a7 to 0d4af89 Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from 7753251 to 757297c Compare edgao force-pushed the destinations_refreshes_cdk_changes branch from 0d4af89 to 52b9185 Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from 757297c to 21d308b Compare edgao force-pushed the destinations_refreshes_cdk_changes branch from 52b9185 to ccf601a Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from 21d308b to 34d6f0b Compare edgao force-pushed the destinations_refreshes_cdk_changes branch from ccf601a to 6839b3f Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from 34d6f0b to cf93b76 Compare edgao commented May 8, 2024
...src/main/kotlin/io/airbyte/cdk/integrations/destination/async/model/PartialAirbyteMessage.kt Outdated Show resolved Hide resolved
edgao force-pushed the destinations_refreshes_cdk_changes branch from 6839b3f to d361d45 Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from cf93b76 to 5878c44 Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from 5878c44 to de8e132 Compare edgao changed the base branch from destinations_refreshes_cdk_changes to edgao/better_default_namespace_tracking edgao force-pushed the edgao/async_tracking_stream_statuses branch 2 times, most recently from 136594e to 3c20271 Compare edgao commented May 9, 2024
Contributor Author
There was a problem hiding this comment.
This was only needed for BufferedStreamConsumer, which should no longer be relevant to DV2 destinations...
edgao force-pushed the edgao/catalog_parser_improvements branch from e29c5d9 to 1f11d82 Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from 221663a to 80bab4a Compare edgao force-pushed the edgao/catalog_parser_improvements branch from 1f11d82 to afef1a6 Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from 80bab4a to 96c5070 Compare 
gisripa reviewed Jun 3, 2024
| package io.airbyte.cdk.integrations.destination | ||
| | ||
| import java.util.* | ||
| import io.airbyte.protocol.models.v0.AirbyteStreamStatusTraceMessage.AirbyteStreamStatus |
Contributor
There was a problem hiding this comment.
for later: we need to clean up this v0 protocol from dependency before we introduce protocol versioning. most places we use classes from v0 and some non-v0
gisripa reviewed Jun 5, 2024
| AirbyteMessage.Type.STATE -> { | ||
| stateManager.trackState(message, sizeInBytes.toLong()) | ||
| } | ||
| else -> {} |
gisripa reviewed Jun 5, 2024
...ain/kotlin/io/airbyte/cdk/integrations/destination/async/deser/AirbyteMessageDeserializer.kt Outdated Show resolved Hide resolved
...src/main/kotlin/io/airbyte/cdk/integrations/destination/async/model/PartialAirbyteMessage.kt Outdated Show resolved Hide resolved
edgao force-pushed the edgao/catalog_parser_improvements branch from afef1a6 to 60fe677 Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch 4 times, most recently from 9c697a3 to 1be53b3 Compare 
gisripa force-pushed the edgao/catalog_parser_improvements branch from 60fe677 to 7910b55 Compare 2 tasks
gisripa approved these changes Jun 7, 2024
edgao force-pushed the edgao/catalog_parser_improvements branch from 7910b55 to 1638b6d Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from 1be53b3 to 600a84f Compare gisripa force-pushed the edgao/catalog_parser_improvements branch from 1638b6d to 7910b55 Compare edgao force-pushed the edgao/catalog_parser_improvements branch from 7910b55 to feba86e Compare edgao force-pushed the edgao/async_tracking_stream_statuses branch from 600a84f to 30466fd Compare 
edgao force-pushed the edgao/async_tracking_stream_statuses branch from 30466fd to c469716 Compare Contributor Author
| /publish-java-cdk
|
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
closes https://github.com/airbytehq/airbyte-internal-issues/issues/7608; closes #7621
overwriteFinalTableif the status was not COMPLETEthis PR has "safe" default behavior, i.e. if platform doesn't send a stream status message then we assume success. This is compatible with older platform versions. See #38067 for how we switch to the backwards-incompatible behavior (which is required for no data downtime).