--stepwise-skip has no effect without --stepwise #9062
Description
- a detailed description of the bug or problem you are having
The flag --stepwise will "exit on test failure and continue from last failing test".
The flag --stepwise-skip is supposed to "ignore the first failing test but stop on the next failing test".
However, it does nothing unless used in conjunction with --stepwise. The combo requirement is not mentioned in the help text, seems redundant, and is surprising behavior.
I recommend that --stepwise-skip should act the same as --stepwise --stepwise-skip.
- output of
pip listfrom the virtual environment you are using
$ pip list Package Version ---------- ------- attrs 21.2.0 iniconfig 1.1.1 packaging 21.0 pip 21.2.4 pluggy 0.13.1 py 1.10.0 pyparsing 2.4.7 pytest 6.2.4 setuptools 46.4.0 toml 0.10.2 wheel 0.34.2- pytest and operating system versions
MacOS, Python 3.9.5, pytest-6.2.4
- minimal example if possible
Example test_step.py:
def test_one(): assert False def test_two(): assert False def test_three(): assert Falsethree failures:
$ pytest --tb=no test_step.py ========================= test session starts ========================== platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 rootdir: /Users/okken/projects/sw collected 3 items test_step.py FFF [100%] ======================= short test summary info ======================== FAILED test_step.py::test_one - assert False FAILED test_step.py::test_two - assert False FAILED test_step.py::test_three - assert False ========================== 3 failed in 0.01s =========================== --stepwise-skip has no effect:
$ pytest --tb=no --stepwise-skip test_step.py ========================= test session starts ========================== platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 rootdir: /Users/okken/projects/sw collected 3 items test_step.py FFF [100%] ======================= short test summary info ======================== FAILED test_step.py::test_one - assert False FAILED test_step.py::test_two - assert False FAILED test_step.py::test_three - assert False ========================== 3 failed in 0.01s ===========================--stepwise works as expected, stopping after first failure:
$ pytest --tb=no --stepwise test_step.py ========================= test session starts ========================== platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 rootdir: /Users/okken/projects/sw collected 3 items stepwise: no previously failed tests, not skipping. test_step.py F ======================= short test summary info ======================== FAILED test_step.py::test_one - assert False !!!! Interrupted: Test failed, continuing from this test next run. !!!!! ========================== 1 failed in 0.07s ===========================--stepwise-skip only works with --stepwise to stop after second failure:
$ pytest --tb=no --stepwise --stepwise-skip test_step.py ========================= test session starts ========================== platform darwin -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 rootdir: /Users/okken/projects/sw collected 3 items stepwise: skipping 0 already passed items. test_step.py FF ======================= short test summary info ======================== FAILED test_step.py::test_one - assert False FAILED test_step.py::test_two - assert False !!!! Interrupted: Test failed, continuing from this test next run. !!!!! ========================== 2 failed in 0.07s ===========================I believe the above behavior, the combo of the flags, should work like that even if only --stepwise-skip is used.