Skip to content

Conversation

@dstandish
Copy link
Contributor

@dstandish dstandish commented Sep 17, 2025

Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async.

This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error.

In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop.

What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.

Related: #55568

In 2.x sometimes get_connection (which goes to the database) might be called without wrapping in sync_to_async. This did not fail, though it was not good behavior, since it can block the event loop. In 3.0, since we now route db calls through an API, this triggers that do this fail. The reason is, the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop.
@ashb
Copy link
Member

ashb commented Sep 17, 2025

The reason is, the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop.

Yes but not quite.

The issue is itit wasn't first wrapped in sync_to_async (which is what the async_to_sync is meant to "bubble out" of

Most triggers are fine and so correctly call it in the right way. The ones they don't were always at risk of blocking the event loop (but in practice works have not blocked it does more than a few ms if they did. Most of the time)

@dstandish
Copy link
Contributor Author

@ashb re

The reason is, the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop.

Yes but not quite.

The issue is itit wasn't first wrapped in sync_to_async (which is what the async_to_sync is meant to "bubble out" of

Most triggers are fine and so correctly call it in the right way. The ones they don't were always at risk of blocking the event loop (but in practice works have not blocked it does more than a few ms if they did. Most of the time)

isn't that what i wrote just above?

sometimes get_connection (which goes to the database) might be called without wrapping in sync_to_async.

@ashb
Copy link
Member

ashb commented Sep 17, 2025

Yes, I guess it is. Forgive me it's late 😴

@dstandish dstandish merged commit f5b1eb4 into apache:main Sep 18, 2025
92 checks passed
@dstandish dstandish deleted the handle-bad-trigger-get-connection-behavior branch September 18, 2025 19:00
kaxil added a commit to astronomer/airflow that referenced this pull request Sep 18, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
@kaxil kaxil added this to the Airflow 3.1.0 milestone Sep 18, 2025
kaxil added a commit that referenced this pull request Sep 18, 2025
We added `greenback` code in Task SDK & Core Airflow in #55799 but only added dep in Core but not Task SDK. This fixes it.
kaxil pushed a commit that referenced this pull request Sep 18, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail. (cherry picked from commit f5b1eb4)
kaxil added a commit that referenced this pull request Sep 18, 2025
We added `greenback` code in Task SDK & Core Airflow in #55799 but only added dep in Core but not Task SDK. This fixes it. (cherry picked from commit 1b88626)
dstandish added a commit to astronomer/airflow that referenced this pull request Sep 19, 2025
In 2.x sometimes get_connection (which goes to the database) might be called without wrapping in sync_to_async. This did not fail, though it was not good behavior, since it can block the event loop. In 3.0, since we now route db calls through an API, triggers that do this fail. The reason is, the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. Related: apache#55568 (cherry picked from commit f5b1eb4)
dstandish added a commit to astronomer/airflow that referenced this pull request Sep 19, 2025
In 2.x sometimes get_connection (which goes to the database) might be called without wrapping in sync_to_async. This did not fail, though it was not good behavior, since it can block the event loop. In 3.0, since we now route db calls through an API, triggers that do this fail. The reason is, the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. Related: apache#55568 (cherry picked from commit f5b1eb4)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 7, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 7, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 8, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 8, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 9, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 9, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 10, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 10, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 11, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 11, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 12, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 12, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 14, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 14, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 15, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 15, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 17, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 17, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 19, 2025
Some [incorrectly-written] triggers may call BaseHook.get_connection without wrapping with sync_to_async. This is naughty behavior because it will block the event loop. But in 2.x it would not cause an error. In 3.0, however, this results in an error. It fails in 3.0 because the code to hit the API wraps the get_connection call with async_to_sync, which is forbidden in the asyncio event loop. What we do here is try to detect when this happens and when it does, we run it through greenback portal which makes it not fail.
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 19, 2025
We added `greenback` code in Task SDK & Core Airflow in apache#55799 but only added dep in Core but not Task SDK. This fixes it.
kaxil added a commit to astronomer/airflow that referenced this pull request Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
kaxil added a commit to astronomer/airflow that referenced this pull request Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
kaxil added a commit to astronomer/airflow that referenced this pull request Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
kaxil added a commit to astronomer/airflow that referenced this pull request Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
kaxil added a commit that referenced this pull request Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in #55799 for 3.1.0 but #56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: #55568 Fixes #57145
kaxil added a commit that referenced this pull request Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in #55799 for 3.1.0 but #56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: #55568 Fixes #57145 (cherry picked from commit da32b68)
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 24, 2025
We added `greenback` code in Task SDK & Core Airflow in apache/airflow#55799 but only added dep in Core but not Task SDK. This fixes it. (cherry picked from commit 1b88626ac3904ac98a293bef934c8b136eb21b42) GitOrigin-RevId: 16569d9a83da03065142d459b4fd22b0d40ef265
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment