Source Zendesk Support: Issue with more than 100k organizations (Missing Headers) #23580
Description
Environment
- Airbyte version: example is 0.40.32
- OS Version / Instance: AWS EC2
- Deployment: Docker
- Source Connector and version: Zendesk-Support 0.2.25
- Step where error happened: Sync job
Current Behavior
During the synchronization of the organization stream, I observe a random issue when querying pages above 1000 (Above 100k organizations, Zendesk has a different behaviour for pages above 1000 where you can query max 10 pages per minute and the response is different).
Traceback (most recent call last): File "/airbyte/integration_code/main.py", line 13, in <module> launch(source, sys.argv[1:]) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in launch for message in source_entrypoint.run(parsed_args): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 122, in run for message in generator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 134, in read raise e File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 120, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 189, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 255, in _read_incremental for message_counter, record_data_or_message in enumerate(records, start=1): File "/airbyte/integration_code/source_zendesk_support/streams.py", line 354, in read_records self._retry(request=request, retries=retries, original_exception=exc, **kwargs) File "/airbyte/integration_code/source_zendesk_support/streams.py", line 320, in _retry sleep_time = self.backoff_time(response) File "/airbyte/integration_code/source_zendesk_support/streams.py", line 132, in backoff_time retry_after = int(to_int(response.headers.get("Retry-After", 0))) AttributeError: 'NoneType' object has no attribute 'headers' I already fixed a similar issue in PR #19967 and it has come back here because it seems to not be a common issues as people using Airbyte do not seem to have more than 100k organizations.
The issue I had in this previous PR was the same : for this specific error, Zendesk doesn't fill a Header
Request URL: https://*******.zendesk.com/api/v2/organizations?start_time=978307200&page=1022, Response Code: 429, Response Text: Pagination limit reached for offset model Expected Behavior
I expect organizations to be correctly synchronized (it works for version Zendesk-Support 0.2.19).
Are you willing to submit a PR?
Yes I am.