Invalid token in metadata exchange #346
Description
Bug Description
New connections to alloyDB stops working after 30hs of fastapi service running on cloud run.
The problem is the token sent in the metadata exchange is never refreshed.
The following code example is a simplified version of the actual connection code.
Example code (or command)
alloy_connector = None def getconn(): global alloy_connector alloy_connector = alloy_connector or AlloyConnector(refresh_strategy="lazy") conn = alloy_connector.connect( settings.ALLOY_DB_IAM_CONNECTION_NAME, "pg8000", user=settings.ALLOY_DB_IAM_USER db=db_name, enable_iam_auth=True ) engine = engine_new(getconn) Session = sessionmaker(autocommit=False, autoflush=False, bind=engine) def get_db(tenant_id: str = None): session = Session() try: yield session except Exception: session.rollback() finally: session.close() class StoreFactory: def __init__(self, StoreClass): self.store_class = StoreClass def __call__(self): return self.store_class(next(get_db())) @router.get('/foobar') @db_session def foobar(store: TestStore = Depends(StoreFactory(TestStore))): retval = store.get('foobar'): if retval is None: raise HTTPException(status_code=404) return retvalStacktrace
0: "Traceback (most recent call last):" 1: " File "/usr/src/app/core/storage/postgresql/store.py", line 126, in get_tenant_db_connection conn = alloy_connector.connect(" 2: " File "/opt/sorcero_env/.venv/lib/python3.9/site-packages/google/cloud/alloydb/connector/connector.py", line 135, in connect return connect_task.result()" 3: " File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result()" 4: " File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception" 5: " File "/opt/sorcero_env/.venv/lib/python3.9/site-packages/google/cloud/alloydb/connector/connector.py", line 208, in connect_async sock = await self._loop.run_in_executor(None, metadata_partial)" 6: " File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs)" 7: " File "/opt/sorcero_env/.venv/lib/python3.9/site-packages/google/cloud/alloydb/connector/connector.py", line 313, in metadata_exchange raise ValueError(" 8: "ValueError: Metadata Exchange request has failed with error: error checking IAM permissions: response code: 16, response message: Request had invalid authentication credentials. Expected OAuth 2 access token or self-signed JWT token. See https://developers.google.com/identity/sign-in/web/devconsole-project."Steps to reproduce?
- Run fastapi service in cloudrun with a global alloydb connector and session using SqlAlchemy
- Start making frequent request to an api that creates a new connection to alloydb
- Wait for a couple of days for connector token oauth0 token to expire
- Exception is raised on connector.connect() (See logs)
Environment
- OS type and version: Docker image based on python:3.9-slim on Cloud RUN
- Python version: 3.9.18
- AlloyDB Python Connector version: 1.2.0
Additional Details
No response