Skip to content

Source Zendesk Support: refactor and debug stuck syncs #20900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Jan 17, 2023
Merged

Source Zendesk Support: refactor and debug stuck syncs #20900

merged 7 commits into from
Jan 17, 2023

Conversation

@davydov-d
Copy link
Contributor

@davydov-d davydov-d commented Dec 28, 2022 •
edited
Loading

What

https://github.com/airbytehq/oncall/issues/1231

How

  • Reduce duplicate sleeps in a main thread
  • Reduce backoff time (for unknown reason it was doubled)
  • Added warning if a ProcessPoolExecutor is used instead of a ThreadPoolExecutor
  • Added more logs to understand what makes the sync stuck
  • Introduced ignore_pagination config variable to speed up SATs
@octavia-squidington-iv octavia-squidington-iv added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/zendesk-support labels Dec 28, 2022
@davydov-d
Copy link
Contributor Author

davydov-d commented Dec 28, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3795199632
✅ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3795199632
Python tests coverage:

Name Stmts Miss Cover -------------------------------------------------------- source_zendesk_support/__init__.py 2 0 100% source_zendesk_support/streams.py 360 31 91% source_zendesk_support/source.py 56 6 89% -------------------------------------------------------- TOTAL 418 37 91% Name Stmts Miss Cover Missing ---------------------------------------------------------------------------------- source_acceptance_test/base.py 12 4 67% 16-19 source_acceptance_test/config.py 140 5 96% 87, 93, 238, 242-243 source_acceptance_test/conftest.py 208 92 56% 36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353 source_acceptance_test/plugin.py 69 25 64% 22-23, 31, 36, 120-140, 144-148 source_acceptance_test/tests/test_core.py 402 115 71% 53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776 source_acceptance_test/tests/test_incremental.py 158 14 91% 52-59, 64-77, 240 source_acceptance_test/utils/asserts.py 39 2 95% 62-63 source_acceptance_test/utils/common.py 94 10 89% 16-17, 32-38, 72, 75 source_acceptance_test/utils/compare.py 62 23 63% 21-51, 68, 97-99 source_acceptance_test/utils/connector_runner.py 133 33 75% 24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208 source_acceptance_test/utils/json_schema_helper.py 107 13 88% 30-31, 38, 41, 65-68, 96, 120, 192-194 ---------------------------------------------------------------------------------- TOTAL 1603 336 79%

Build Passed

Test summary info:

=========================== short test summary info ============================ SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical. SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical. ================== 38 passed, 3 skipped in 661.48s (0:11:01) ===================
@davydov-d
Copy link
Contributor Author

davydov-d commented Jan 10, 2023 •
edited by github-actions bot
Loading

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3883200185
@davydov-d
Copy link
Contributor Author

davydov-d commented Jan 16, 2023 •
edited by github-actions bot
Loading

/test connector=connectors/source-zendesk-support

🕑 connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3933201233
✅ connectors/source-zendesk-support https://github.com/airbytehq/airbyte/actions/runs/3933201233
Python tests coverage:

Name Stmts Miss Cover -------------------------------------------------------- source_zendesk_support/__init__.py 2 0 100% source_zendesk_support/streams.py 380 37 90% source_zendesk_support/source.py 56 6 89% -------------------------------------------------------- TOTAL 438 43 90% Name Stmts Miss Cover Missing ---------------------------------------------------------------------------------- source_acceptance_test/base.py 12 4 67% 16-19 source_acceptance_test/config.py 141 5 96% 87, 93, 239, 243-244 source_acceptance_test/conftest.py 211 95 55% 36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358 source_acceptance_test/plugin.py 69 25 64% 22-23, 31, 36, 120-140, 144-148 source_acceptance_test/tests/test_core.py 402 115 71% 53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776 source_acceptance_test/tests/test_incremental.py 160 14 91% 58-65, 70-83, 246 source_acceptance_test/utils/asserts.py 39 2 95% 62-63 source_acceptance_test/utils/common.py 94 10 89% 16-17, 32-38, 72, 75 source_acceptance_test/utils/compare.py 62 23 63% 21-51, 68, 97-99 source_acceptance_test/utils/connector_runner.py 133 33 75% 24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208 source_acceptance_test/utils/json_schema_helper.py 107 13 88% 30-31, 38, 41, 65-68, 96, 120, 192-194 ---------------------------------------------------------------------------------- TOTAL 1609 339 79%

Build Passed

Test summary info:

=========================== short test summary info ============================ SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical. ================== 39 passed, 2 skipped in 390.89s (0:06:30) ===================
davydov-d
davydov-d commented Jan 16, 2023
View reviewed changes
airbyte-integrations/connectors/source-zendesk-support/unit_tests/test_futures.py
assert len(records) == records_count
end = datetime.datetime.now()
sleep_time = int(60 / x_rate_limit)
assert sleep_time - 1 <= (end - start).seconds <= sleep_time + 1
Copy link
Contributor Author

@davydov-d davydov-d Jan 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test shows 6 seconds sleeping against 44 on a previous version
bazarnov
bazarnov approved these changes Jan 17, 2023
View reviewed changes
Copy link
Contributor

@bazarnov bazarnov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move the common method from the stream class to their base class.
@davydov-d
Copy link
Contributor Author

davydov-d commented Jan 17, 2023

Please move the common method from the stream class to their base class.

@bazarnov unfortunately I can not do it because the base class has it's own implementation and some of the descendants inherit it
@davydov-d
Copy link
Contributor Author

davydov-d commented Jan 17, 2023 •
edited by github-actions bot
Loading

/publish connector=connectors/source-zendesk-support

🕑 Publishing the following connectors:
connectors/source-zendesk-support
https://github.com/airbytehq/airbyte/actions/runs/3940910725


Connector Did it publish? Were definitions generated?
connectors/source-zendesk-support ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets January 17, 2023 16:15 — with GitHub Actions Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets January 17, 2023 16:16 — with GitHub Actions Inactive
@github-actions
Copy link
Contributor

github-actions bot commented Jan 17, 2023

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage 26.59% 🍏
@davydov-d davydov-d merged commit f31d92c into master Jan 17, 2023
@davydov-d davydov-d deleted the ddavydov/#1231-oncall-source-zendesk-support-sync-stucks branch January 17, 2023 16:54
@marcosmarxm marcosmarxm mentioned this pull request Jan 23, 2023
Closed
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Jan 24, 2023
Merged
@maxi297 maxi297 mentioned this pull request Mar 20, 2025
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/zendesk-support

6 participants

@davydov-d @bazarnov @lazebnyi @octavia-squidington-iii @octavia-squidington-iv