✨29779 source postgres slow ctid read seen on customer connection #30125

rodireich · 2023-09-04T05:23:44Z

What

Postgres of versions before 14 are unable to run TID range scan queries that are running when we query with WHERE ctid > '(x,y)'.
As a result every time we query for a chunk of data postgres needs to run a full scan of the entire table. This leads to a considerably slow performance.
In order to avoid this issue we need to query postgres 12, 13 in a way that will do a TID scan (no "range")
See full discussion in this [doc]

How

Instead of querying for a range of ctid (WHERE ctid > '(x,y)'),
We are going to query for a list of ctid's.
Because there is no range of ctid, we need to know the last possible tuple in a page.

🚨 User Impact 🚨

User should see quicker initial sync of tables on all 3 sync modes (CDC, cursor, xmin).
Previous syncs that started with the old algorithm will seamlessly work faster upon another attempt.
No breaking changes.

# Conflicts: # airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresQueryUtils.java # airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java # airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/cdc/PostgresCdcCtidInitializer.java # airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/ctid/CtidGlobalStateManager.java # airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/ctid/CtidPerStreamStateManager.java # airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/ctid/CtidStateManager.java # airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/ctid/PostgresCtidHandler.java # airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/cursor_based/CursorBasedCtidUtils.java # airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/xmin/XminCtidUtils.java # airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/ctid/PostgresCtidHandlerTest.java

github-actions · 2023-09-04T05:24:06Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-09-04T05:51:21Z

Coverage report for source-postgres

File	Coverage [88.52%]	🍏
PostgresCtidHandler.java	94.78%	🍏
CtidUtils.java	91.74%	🍏
InitialSyncCtidIterator.java	89.83%	🍏
PostgresCdcCtidInitializer.java	89.61%	🍏
PostgresSource.java	88.55%	🍏
PostgresQueryUtils.java	84.63%	🍏
Ctid.java	82.42%	🍏

Total Project Coverage	71.68%	🍏

prateekmukhedkar

@rodireich i realized now you don't have changes to release a new version of source-postgres connector. Were you planning to do it in a different PR?

rodireich · 2023-09-20T21:56:03Z

@prateekmukhedkar did you mean an bumped version?
I always do that right before merging ☺️

…on-customer-connection

Tests like PostgresSourceAcceptanceLegacyCtidTest run the same test flows for with this new flow.

airbyte-oss-build-runner · 2023-09-20T22:26:41Z

source-postgres test report (commit `201421cffd`) - ❌

⏲️ Total pipeline duration: 28mn00s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-09-20T22:56:03Z

source-postgres test report (commit `ffe61c9921`) - ✅

⏲️ Total pipeline duration: 27mn38s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-09-20T23:02:55Z

source-postgres-strict-encrypt test report (commit `ffe61c9921`) - ✅

⏲️ Total pipeline duration: 06mn42s

Step	Result
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

airbyte-oss-build-runner · 2023-09-20T23:05:20Z

source-alloydb test report (commit `ffe61c9921`) - ❌

⏲️ Total pipeline duration: 02mn15s

Step	Result
Build connector tar	✅
Build source-alloydb docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-alloydb/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

airbyte-oss-build-runner · 2023-09-20T23:10:57Z

source-postgres test report (commit `f40e7032d8`) - ✅

⏲️ Total pipeline duration: 03mn39s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-oss-build-runner · 2023-09-20T23:12:45Z

source-alloydb test report (commit `f40e7032d8`) - ❌

⏲️ Total pipeline duration: 01mn38s

Step	Result
Build connector tar	✅
Build source-alloydb docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	❌
Validate airbyte-integrations/connectors/source-alloydb/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

rodireich · 2023-09-20T23:15:20Z

/approve-and-merge reason="AlloyDB acceptance test failing in CI"

airbyte-oss-build-runner · 2023-09-20T23:15:49Z

source-postgres-strict-encrypt test report (commit `f40e7032d8`) - ✅

⏲️ Total pipeline duration: 02mn53s

Step	Result
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

octavia-approvington · 2023-09-20T23:15:59Z

All in!!

…0125) Co-authored-by: subodh <subodh1810@gmail.com> Co-authored-by: subodh1810 <subodh1810@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com>

subodh1810 and others added 19 commits August 21, 2023 22:23

wip

ea02ea4

wip

b44af98

handle when no data available

5c9d11c

update filenode every time

0ff642d

Merge branch 'master' into postgres-validate-file-node-each-chunk

2493899

cleanup

3644565

minor simplification

9328866

check for null file node

190b10b

rename iterators for clarity

896334a

Merge branch 'master' into postgres-validate-file-node-each-chunk

dc6969b

address review comments

be6f71c

address review comments part 2

c0dbc58

Merge branch 'master' into postgres-validate-file-node-each-chunk

008e2be

Automated Commit - Formatting Changes

ae40ec1

fix log message

b4ca423

Merge branch 'master' into postgres-validate-file-node-each-chunk

7f3b586

address review comments

417a16f

initial commit

758c722

rodireich linked an issue Sep 4, 2023 that may be closed by this pull request

source-postgres: slow ctid read seen on customer connection #29779

Closed

1 task

octavia-squidington-iii added area/connectors Connector related issues connectors/source/postgres labels Sep 4, 2023

Automated Commit - Formatting Changes

cfb1c34

rodireich changed the base branch from master to postgres-validate-file-node-each-chunk September 4, 2023 05:47

rodireich and others added 4 commits September 3, 2023 23:11

initial commit

a1d2801

Automated Commit - Formatting Changes

466d5da

initial commit

6e4af73

Automated Commit - Formatting Changes

d97e257

prateekmukhedkar requested review from prateekmukhedkar and subodh1810 September 20, 2023 16:33

prateekmukhedkar approved these changes Sep 20, 2023

View reviewed changes

prateekmukhedkar reviewed Sep 20, 2023

View reviewed changes

Merge branch 'master' into 29779-source-postgres-slow-ctid-read-seen-…

201421c

…on-customer-connection

vercel bot deployed to Preview September 20, 2023 21:58 View deployment

rodireich added 2 commits September 20, 2023 15:06

reduce querying for version server

53f8727

Bump version. update changelog

ffe61c9

octavia-squidington-iii added area/documentation Improvements or additions to documentation connectors/source/postgres-strict-encrypt connectors/source/alloydb labels Sep 20, 2023

vercel bot deployed to Preview September 20, 2023 22:23 View deployment

rodireich changed the title ~~29779 source postgres slow ctid read seen on customer connection~~ ✨29779 source postgres slow ctid read seen on customer connection Sep 20, 2023

sanity

f40e703

vercel bot deployed to Preview September 20, 2023 22:57 View deployment

octavia-approvington approved these changes Sep 20, 2023

View reviewed changes

octavia-approvington merged commit a876e60 into master Sep 20, 2023

octavia-approvington deleted the 29779-source-postgres-slow-ctid-read-seen-on-customer-connection branch September 20, 2023 23:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

✨29779 source postgres slow ctid read seen on customer connection #30125

✨29779 source postgres slow ctid read seen on customer connection #30125

Uh oh!

rodireich commented Sep 4, 2023 •

edited

Loading

github-actions bot commented Sep 4, 2023 •

edited by rodireich

Loading

github-actions bot commented Sep 4, 2023 •

edited

Loading

prateekmukhedkar left a comment

rodireich commented Sep 20, 2023

airbyte-oss-build-runner commented Sep 20, 2023

airbyte-oss-build-runner commented Sep 20, 2023

airbyte-oss-build-runner commented Sep 20, 2023

airbyte-oss-build-runner commented Sep 20, 2023

airbyte-oss-build-runner commented Sep 20, 2023

airbyte-oss-build-runner commented Sep 20, 2023

rodireich commented Sep 20, 2023

airbyte-oss-build-runner commented Sep 20, 2023

octavia-approvington commented Sep 20, 2023

Labels

7 participants

✨29779 source postgres slow ctid read seen on customer connection #30125

✨29779 source postgres slow ctid read seen on customer connection #30125

Uh oh!

Conversation

rodireich commented Sep 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

github-actions bot commented Sep 4, 2023 • edited by rodireich Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

github-actions bot commented Sep 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report for source-postgres

prateekmukhedkar left a comment

Choose a reason for hiding this comment

rodireich commented Sep 20, 2023

airbyte-oss-build-runner commented Sep 20, 2023

source-postgres test report (commit 201421cffd) - ❌

airbyte-oss-build-runner commented Sep 20, 2023

source-postgres test report (commit ffe61c9921) - ✅

airbyte-oss-build-runner commented Sep 20, 2023

source-postgres-strict-encrypt test report (commit ffe61c9921) - ✅

airbyte-oss-build-runner commented Sep 20, 2023

source-alloydb test report (commit ffe61c9921) - ❌

airbyte-oss-build-runner commented Sep 20, 2023

source-postgres test report (commit f40e7032d8) - ✅

airbyte-oss-build-runner commented Sep 20, 2023

source-alloydb test report (commit f40e7032d8) - ❌

rodireich commented Sep 20, 2023

airbyte-oss-build-runner commented Sep 20, 2023

source-postgres-strict-encrypt test report (commit f40e7032d8) - ✅

octavia-approvington commented Sep 20, 2023

Labels

7 participants

rodireich commented Sep 4, 2023 •

edited

Loading

github-actions bot commented Sep 4, 2023 •

edited by rodireich

Loading

github-actions bot commented Sep 4, 2023 •

edited

Loading

source-postgres test report (commit `201421cffd`) - ❌

source-postgres test report (commit `ffe61c9921`) - ✅

source-postgres-strict-encrypt test report (commit `ffe61c9921`) - ✅

source-alloydb test report (commit `ffe61c9921`) - ❌

source-postgres test report (commit `f40e7032d8`) - ✅

source-alloydb test report (commit `f40e7032d8`) - ❌

source-postgres-strict-encrypt test report (commit `f40e7032d8`) - ✅