🐛 bug(source-klaviyo): fix problem with pods running out of memory when syning events stream historical data

[aldogonzalez8]

What

Related to https://github.com/airbytehq/oncall/issues/6684

The last job in the ticket has been running since September 20th, 2024, and it is currently in attempt 10. It has been syncing data from 2015 and forward, running out of memory or having similar problems after ~4 days/145GB of reading. You can find a summary in this comment.

Connection schema is set for 3 streams:

• events
• metrics
• profile

The sync fails because the Events data takes too long; the other two generally continue from the last state.

So my intent to fix this is checkpointing, so even if the sync fails next attempt will start from a more advanced state rather than:

Setting state of SourceKlaviyo stream to {} 

How

Evens stream will use new base_incremental_checkpoint_stream with custom KlaviyoCheckpointDatetimeBasedCursor.

I have set a step of 1 month and a granularity of one second; the cursor uses the start and end date from the slices created to filter, the original KlaviyoDatetimeBasedCursor would only take start_date for this purpose.

Review guide

1. airbyte-integrations/connectors/source-klaviyo/source_klaviyo/manifest.yaml: new stream with checkpointing cursor and 1 month step and 1 second granularity.
2. airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/datetime_based_cursor.py: cursor with filter start and end date from slice created.

User Impact

User should be able to complete the sync job.

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 13, 2024 11:23pm

[natikgadzhi]

@aldogonzalez8 @bazarnov are you guys open to shipping this as a progressive rollout with @clnoll?

[aldogonzalez8]

@aldogonzalez8 @bazarnov are you guys open to shipping this as a progressive rollout with @clnoll?

Is ok with me, just need to fix baz comments and fix unit tests, already working on it but not sure of having it today, when you need it?

[bazarnov]

Approving, to not to block the fix! Thanks @aldogonzalez8

[aldogonzalez8]

Approving, to not to block the fix! Thanks @aldogonzalez8

Thanks @bazarnov !

@aldogonzalez8 @bazarnov are you guys open to shipping this as a progressive rollout with @clnoll?

@natikgadzhi @clnoll I think shipping as a progressive roll-out is okay. Can we include this workspace in the initial iteration? It's the one with long-running motivation for this PR.

If this is not possible, let me know so I can go ahead and pin the new version to the WS, I'm assuming "something" needs to be done before I click on squash and merge so I'm holding it on.

[aldogonzalez8]

/approve-regression-tests

Check job output.

✅ Approving regression tests

[aldogonzalez8]

There was a minimal diff in records in regression tests but I interpret this is because of the change on the queryset as now we use filter=greater-or-equal + less-or-equal for slices rather than just greater-than