Follow zero internal warning in unit tests #38642
Description
Body
This is follow up task after #38504, we have quite a few tests which raise one of the internal deprecation warning:
airflow.exceptions.AirflowProviderDeprecationWarningairflow.exceptions.RemovedInAirflow3Warningairflow.utils.context.AirflowContextDeprecationWarning
All current exclusions which should be resolved stored into the tests/deprecations_ignore.yml in format test node id: {relative path to the test}::{ClassName}::{test_case} or {relative path to the test}::{test_case} without parametrised part
How to resolve specific test suite
Remove /commented out specific line in exclusion list a have a look on error by run removed node id,
root@e68ed83041ce:/opt/airflow# pytest tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3 ======================================================================= test session starts ======================================================================== platform linux -- Python 3.8.19, pytest-7.4.4, pluggy-1.4.0 -- /usr/local/bin/python cachedir: .pytest_cache rootdir: /opt/airflow configfile: pyproject.toml plugins: custom-exit-code-0.3.0, instafail-0.5.0, rerunfailures-14.0, asyncio-0.23.6, cov-5.0.0, xdist-3.5.0, timeouts-1.2.1, anyio-4.3.0, icdiff-0.9, mock-3.14.0, requests-mock-1.11.0, time-machine-2.14.1 asyncio: mode=strict setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s collected 1 item tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3 FAILED [100%] ============================================================================= FAILURES ============================================================================= ___________________________________________________________ TestPythonVirtualenvDecorator.test_python_3 ____________________________________________________________ self = <tests.decorators.test_python_virtualenv.TestPythonVirtualenvDecorator object at 0xffff855dd760> dag_maker = <tests.conftest.dag_maker.<locals>.DagFactory object at 0xffff8593da30> def test_python_3(self, dag_maker): @task.virtualenv(python_version=3, use_dill=False, requirements=["dill"]) def f(): import sys print(sys.version) try: {}.iteritems() except AttributeError: return raise Exception with dag_maker(): > ret = f() tests/decorators/test_python_virtualenv.py:171: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ airflow/decorators/base.py:372: in __call__ op = self.operator_class( airflow/models/baseoperator.py:483: in apply_defaults result = func(self, **kwargs, default_args=default_args) airflow/models/baseoperator.py:483: in apply_defaults result = func(self, **kwargs, default_args=default_args) airflow/decorators/python.py:52: in __init__ super().__init__( airflow/models/baseoperator.py:483: in apply_defaults result = func(self, **kwargs, default_args=default_args) airflow/decorators/base.py:257: in __init__ super().__init__(task_id=task_id, **kwargs_to_upstream, **kwargs) airflow/models/baseoperator.py:483: in apply_defaults result = func(self, **kwargs, default_args=default_args) airflow/operators/python.py:620: in __init__ warnings.warn( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <airflow.utils.decorators._autostacklevel_warn object at 0xffff85ebb820> message = 'Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.' category = <class 'airflow.exceptions.RemovedInAirflow3Warning'>, stacklevel = 2, source = None def warn(self, message, category=None, stacklevel=1, source=None): > self.warnings.warn(message, category, stacklevel + 2, source) E airflow.exceptions.RemovedInAirflow3Warning: Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead. airflow/utils/decorators.py:108: RemovedInAirflow3Warning ---------------------------------------------------------------------- Captured stdout setup ----------------------------------------------------------------------- ========================= AIRFLOW ========================== Home of the user: /root Airflow home /root/airflow Skipping initializing of the DB as it was initialized already. You can re-initialize the database by adding --with-db-init flag when running tests. [2024-03-31T10:06:44.945+0000] {dagbag.py:540} INFO - Filling up the DagBag from /dev/null ------------------------------------------------------------------------ Captured log setup ------------------------------------------------------------------------ INFO airflow.models.dagbag.DagBag:dagbag.py:540 Filling up the DagBag from /dev/null ===================================================================== short test summary info ====================================================================== FAILED tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3 - airflow.exceptions.RemovedInAirflow3Warning: Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.After that need to determine which kind of a problem here
Option 1: It might be test deprecation functional without explicit usage of pytest.warn. This case is pretty simple and required to add pytest.warn context manager wit appropriate matcher.
Option 2: Use deprecated functional in tests, time moves on and not all test updated. In this case need to rewrite test to avoid use deprecated part
Option 3: Use deprecated functional in tests utilities and fixtures, the same as option 2, however it might spread across multiple tests suits, replace/add functional to use not deprecated functional it this utilities/fixtures. This might required to use full test needed label in pr, If you not a Commiter or PMC please ask to add this label.
Option 4: Incorrect depreciation in code base. This one could be cause if one part of functional deprecated in code base, but the other still use it. There is no a universal solution for this situation, it might required to avoid use deprecated functional in codebase or even undeprecate.
Other cases: It might be other cases, feel free to share you experience with non listed above in this issue or into the Slack
Tip
Feel free to ask a suggestion/helps in slack channels #contributors or #new-contributors
Note
There main target of this task it resolve deprecation warnings, but not hide all of them under pytest.warn, use this context manager, as well as warnings.catch_warnings context manager only for test suites around deprecated functional
Warning
Please do not mixin changes in code base and only tests into the same PR. Better to split it into the separate PRs.
Committer
- I acknowledge that I am a maintainer/committer of the Apache Airflow project.