bridge_manager failed to push data due to TypeError #79
Description
Hello,
I have exactly the same issues it happen after debian 10 > 11 upgrade
What I tried so far, I completely purged all old amplify agent configs directories and re-install.
Using how to (agent was reinstalled), after initial one min, agent start throwing error not able to reach https://receiver.amplify.nginx.com:443
Versions:
dpkg-query -l |grep nginx nginx 1.20.2-1~buster amd64 high performance web server nginx-amplify-agent 1.8.0-1~bullseye amd64 Agent for NGINX Amplify monitoring platform nginx-common 1.10.3-1+deb9u3 all small, powerful, scalable web/proxy server - common files # Python root@nginx:~# python3 --version Python 3.9.2 # OS cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" # Kernel root@nginx:~# uname -a Linux nginx 5.10.0-13-amd64 nginxinc/nginx-amplify-agent-priv#1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux2022-03-28 12:24:54,721 [9328] nginx_metrics [None] get http://127.0.0.1:80/nginx_status 200 10 106 0.004 2022-03-28 12:24:54,721 [9328] nginx_metrics 34d39fc314628b9ff20bfb12138a4c0cb407d6395fa5512cef844b95d9467663 collect in 0.006 2022-03-28 12:24:54,831 [9328] nginx_meta 34d39fc314628b9ff20bfb12138a4c0cb407d6395fa5512cef844b95d9467663 collect in 0.075 2022-03-28 12:24:55,016 [9328] supervisor system objects: ['e5389f19c285e6b27ce9f3cb150d4a97bad0ab4fa472deff1a5cc5c7194c6ae7'] 2022-03-28 12:24:55,033 [9328] supervisor ps nginx output: [' 7533 1 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf', ' 7534 7533 nginx: worker process', ' 7535 7533 nginx: worker process', ' 7536 7533 nginx: cache manager process', ''] 2022-03-28 12:24:55,043 [9328] supervisor nginx objects: ['34d39fc314628b9ff20bfb12138a4c0cb407d6395fa5512cef844b95d9467663'] 2022-03-28 12:24:55,044 [9328] supervisor status objects: [] 2022-03-28 12:24:55,044 [9328] supervisor api objects: [] 2022-03-28 12:24:55,674 [9328] bridge_manager failed POST "https://receiver.amplify.nginx.com:443/1.4/XXXXXXXXXXXXX/update/", exception: "HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0)" 2022-03-28 12:24:55,675 [9328] bridge_manager Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in connect self.sock = ssl_wrap_socket( File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket ssl_sock = _ssl_wrap_socket_impl( File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 114, in wrap_socket return self.sslsocket_class( File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 312, in __init__ raise x File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 308, in __init__ self.do_handshake() File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 672, in do_handshake self._wait(self._read_event, timeout_exc=_SSLErrorHandshakeTimeout) File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch socket.timeout: The handshake operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 385, in _make_request self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 336, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/amplify/agent/common/util/http.py", line 94, in make_request r = self.session.post( File "/usr/lib/python3/dist-packages/requests/sessions.py", line 590, in post return self.request('POST', url, data=data, json=json, **kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3/dist-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0) 2022-03-28 12:24:55,765 [9328] bridge_manager 2022-03-28 12:24:55,766 [9328] bridge_manager [None] post https://receiver.amplify.nginx.com:443/1.4/5147cd2d2fd569c76dbe47a18e2de225/update/ 500 1447 0 7.809 2022-03-28 12:24:55,766 [9328] bridge_manager http delay set to 10 (fails: 1) 2022-03-28 12:24:55,766 [9328] bridge_manager failed to push data due to ReadTimeout 2022-03-28 12:24:55,766 [9328] bridge_manager additional info: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn conn.connect() File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 411, in connect self.sock = ssl_wrap_socket( File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket ssl_sock = _ssl_wrap_socket_impl( File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 114, in wrap_socket return self.sslsocket_class( File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 312, in __init__ raise x File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 308, in __init__ self.do_handshake() File "/usr/lib/python3/dist-packages/gevent/_ssl3.py", line 672, in do_handshake self._wait(self._read_event, timeout_exc=_SSLErrorHandshakeTimeout) File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch socket.timeout: The handshake operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 532, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise raise value File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 385, in _make_request self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 336, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=7.0)Connectivity test
curl -v https://receiver.amplify.nginx.com * Trying 35.155.143.94:443... * Connected to receiver.amplify.nginx.com (35.155.143.94) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: CN=receiver.amplify.nginx.com * start date: Mar 14 21:05:25 2022 GMT * expire date: Jun 12 21:05:24 2022 GMT * subjectAltName: host "receiver.amplify.nginx.com" matched cert's "receiver.amplify.nginx.com" * issuer: C=US; O=Let's Encrypt; CN=R3 * SSL certificate verify ok. > GET / HTTP/1.1 > Host: receiver.amplify.nginx.com > User-Agent: curl/7.74.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 404 Not Found < Server: nginx < Date: Mon, 28 Mar 2022 09:31:51 GMT < Content-Type: text/html < Content-Length: 146 < Connection: keep-alive < X-Amplify-ID: 2017ea69c9f904f966ae0433d6f205a0 < <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html> * Connection #0 to host receiver.amplify.nginx.com left intact