- Notifications
You must be signed in to change notification settings - Fork 16.2k
Handle trigger calls to get_connection #55799
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle trigger calls to get_connection #55799
Conversation
In 2.x sometimes get_connection (which goes to the database) might be called without wrapping in sync_to_async. This did not fail, though it was not good behavior, since it can block the event loop. In 3.0, since we now route db calls through an API, this triggers that do this fail. The reason is, the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop.
Yes but not quite. The issue is itit wasn't first wrapped in sync_to_async (which is what the async_to_sync is meant to "bubble out" of Most triggers are fine and so correctly call it in the right way. The ones they don't were always at risk of blocking the event loop (but in practice works have not blocked it does more than a few ms if they did. Most of the time) |
| @ashb re
isn't that what i wrote just above?
|
| Yes, I guess it is. Forgive me it's late 😴 |
This reverts commit 4f154a2.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail. (cherry picked from commit f5b1eb4)
In 2.x sometimes get_connection (which goes to the database) might be called without wrapping in sync_to_async. This did not fail, though it was not good behavior, since it can block the event loop. In 3.0, since we now route db calls through an API, triggers that do this fail. The reason is, the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. Related: apache#55568 (cherry picked from commit f5b1eb4)
In 2.x sometimes get_connection (which goes to the database) might be called without wrapping in sync_to_async. This did not fail, though it was not good behavior, since it can block the event loop. In 3.0, since we now route db calls through an API, triggers that do this fail. The reason is, the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. Related: apache#55568 (cherry picked from commit f5b1eb4)
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in #55799 for 3.1.0 but #56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: #55568 Fixes #57145
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in #55799 for 3.1.0 but #56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: #55568 Fixes #57145 (cherry picked from commit da32b68)
We added `greenback` code in Task SDK & Core Airflow in apache/airflow#55799 but only added dep in Core but not Task SDK. This fixes it. (cherry picked from commit 1b88626ac3904ac98a293bef934c8b136eb21b42) GitOrigin-RevId: 16569d9a83da03065142d459b4fd22b0d40ef265
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async.
This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error.
In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop.
What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
Related: #55568