Random timeouts in api calls to bigquery.googleapis.com #40
Description
After updating google-cloud-bigquery from version 1.19.0 to 1.24.0,
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Read timed out. (read timeout=11.0) started to pop up randomly.
This is likely related to #34
Environment details
- OS type and version:
Ubuntu 19.10 x64 - Python version:
3.7.5 - pip version:
pip 20.0.2 google-cloud-bigqueryversion:1.24.0
Steps to reproduce
The error is not deterministic, however we observed it on production environments (compute instances in GCP) and local developement environments.
For about 100 queries, at least 1-2 fails with this error (which makes it reproducible). This makes something like 1-2% of ALL requests to fail!
If I understand correctly, BQ API endpoint responsible for result() method will block for at most 10 seconds. There is also 1 second margin to neutralize network lags etc. No retry mechanism covering the timeout is present, so in case of delay of more than 1 second, the whole request will fail.
In my opinion, having 1.0s non-configurable timeout is not safe. Also not-mutating endpoints (like "is job finished") should automatically retry in case of timeout. This is not implemented at the moment.
We had to rollback to 1.19.0 to make everything stable again.
Code example
Nothing really helpful could be placed here.
The easiest way to reproduce this error is to run query that takes MORE than 10s in 100x loop.
Stack trace
The stack trace is always the same:
Traceback (most recent call last): File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 421, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 416, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.7/http/client.py", line 1344, in getresponse response.begin() File "/usr/lib/python3.7/http/client.py", line 306, in begin version, status, reason = self._read_status() File "/usr/lib/python3.7/http/client.py", line 267, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib/python3.7/socket.py", line 589, in readinto return self._sock.recv_into(b) File "/usr/lib/python3.7/ssl.py", line 1071, in recv_into return self.read(nbytes, buffer) File "/usr/lib/python3.7/ssl.py", line 929, in read return self._sslobj.read(len, buffer) socket.timeout: The read operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<proj>/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "<proj>/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 400, in increment raise six.reraise(type(error), error, _stacktrace) File "<proj>/venv/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise raise value File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen chunked=chunked, File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "<proj>/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout self, url, "Read timed out. (read timeout=%s)" % timeout_value urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Read timed out. (read timeout=11.0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 3196, in result super(QueryJob, self).result(retry=retry, timeout=timeout) File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 818, in result return super(_AsyncJob, self).result(timeout=timeout) File "<proj>/venv/lib/python3.7/site-packages/google/api_core/future/polling.py", line 122, in result self._blocking_poll(timeout=timeout) File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 3098, in _blocking_poll super(QueryJob, self)._blocking_poll(timeout=timeout) File "<proj>/venv/lib/python3.7/site-packages/google/api_core/future/polling.py", line 101, in _blocking_poll retry_(self._done_or_raise)() File "<proj>/venv/lib/python3.7/site-packages/google/api_core/retry.py", line 289, in retry_wrapped_func return retry_wrapped_func File "<proj>/venv/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target return target() File "<proj>/venv/lib/python3.7/site-packages/google/api_core/future/polling.py", line 80, in _done_or_raise if not self.done(): File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/job.py", line 3085, in done timeout=timeout, File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 1287, in _get_query_results retry, method="GET", path=path, query_params=extra_params, timeout=timeout File "<proj>/venv/lib/python3.7/site-packages/google/cloud/bigquery/client.py", line 556, in _call_api return call() File "<proj>/venv/lib/python3.7/site-packages/google/api_core/retry.py", line 289, in retry_wrapped_func return retry_wrapped_func File "<proj>/venv/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target return target() File "<proj>/venv/lib/python3.7/site-packages/google/cloud/_http.py", line 419, in api_request timeout=timeout, File "<proj>/venv/lib/python3.7/site-packages/google/cloud/_http.py", line 277, in _make_request method, url, headers, data, target_object, timeout=timeout File "<proj>/venv/lib/python3.7/site-packages/google/cloud/_http.py", line 315, in _do_request url=url, method=method, headers=headers, data=data, timeout=timeout File "<proj>/venv/lib/python3.7/site-packages/google/auth/transport/requests.py", line 317, in request **kwargs File "<proj>/venv/lib/python3.7/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "<proj>/venv/lib/python3.7/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "<proj>/venv/lib/python3.7/site-packages/requests/adapters.py", line 529, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Read timed out. (read timeout=11.0)