Repository navigation
8.0.0 reverses order of parametrize when calling individual tests #11937
Description
Summary
Marking a test with @pytest.mark.parametrize('i', [1, 2, 3, 4]), I expect the test to run 1, 2, 3, 4.
As a file, pytest does this as expected.
Running individual tests, though, results in 4, 3, 2, 1.
This is surprising behavior and new to 8.0.0.
(foo) $ pytest test_param.py::test_param -v
...
platform win32 -- Python 3.11.7, pytest-8.0.0, ...
...
test_param.py::test_param[4] PASSED [ 25%]
test_param.py::test_param[3] PASSED [ 50%]
test_param.py::test_param[2] PASSED [ 75%]
test_param.py::test_param[1] PASSED [100%]
...
Version 7.4.4 runs the test cases in the expected 1, 2, 3, 4 order.
(foo) $ pytest test_param.py::test_param -v
...
platform win32 -- Python 3.11.7, pytest-7.4.4, ...
...
test_param.py::test_param[1] PASSED [ 25%]
test_param.py::test_param[2] PASSED [ 50%]
test_param.py::test_param[3] PASSED [ 75%]
test_param.py::test_param[4] PASSED [100%]
...
Detail
-
a detailed description of the bug or problem you are having
When running individual parametrized tests, the order is reversed.
This is new in 8.0.0, and not the case in 7.4.4
Since 8.0.0 doesn't guarantee backward compatibility, I'm not sure if this is a bug or a feature. :)
But I didn't see it in the notes, so I'm guessing this is an unintended change, and it is an unexpected, surprising behavior. -
output of
pip listfrom the virtual environment you are using
$ pip list
Package Version
---------- -------
colorama 0.4.6
iniconfig 2.0.0
packaging 23.2
pip 23.3.1
pluggy 1.3.0
pytest 8.0.0
setuptools 65.5.0
- pytest and operating system versions
platform win32 -- Python 3.11.7, pytest-8.0.0, pluggy-1.3.0
- minimal example if possible
$ cat test_param.py
import pytest
@pytest.mark.parametrize('i', [1, 2, 3, 4])
def test_param(i):
...
Output from 8.0.0, shows expected 1,2,3,4 order with file invocation, but reverse order with individual test, 4, 3, 2, 1
(foo) $ pytest test_param.py -v
============================= test session starts =============================
platform win32 -- Python 3.11.7, pytest-8.0.0, pluggy-1.3.0 -- C:\Users\okken\projects\foo\venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\okken\projects\foo
configfile: pytest.ini
collecting ... collected 4 items
test_param.py::test_param[1] PASSED [ 25%]
test_param.py::test_param[2] PASSED [ 50%]
test_param.py::test_param[3] PASSED [ 75%]
test_param.py::test_param[4] PASSED [100%]
============================== 4 passed in 0.01s ==============================
(foo) $ pytest test_param.py::test_param -v
============================= test session starts =============================
platform win32 -- Python 3.11.7, pytest-8.0.0, pluggy-1.3.0 -- C:\Users\okken\projects\foo\venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\okken\projects\foo
configfile: pytest.ini
collecting ... collected 4 items
test_param.py::test_param[4] PASSED [ 25%]
test_param.py::test_param[3] PASSED [ 50%]
test_param.py::test_param[2] PASSED [ 75%]
test_param.py::test_param[1] PASSED [100%]
============================== 4 passed in 0.02s ==============================
Output from 7.4.4, shows expected 1,2,3,4 order with both file and individual test invocation.
(foo) $ pytest test_param.py -v
============================= test session starts =============================
platform win32 -- Python 3.11.7, pytest-7.4.4, pluggy-1.3.0 -- C:\Users\okken\projects\foo\venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\okken\projects\foo
configfile: pytest.ini
collecting ... collected 4 items
test_param.py::test_param[1] PASSED [ 25%]
test_param.py::test_param[2] PASSED [ 50%]
test_param.py::test_param[3] PASSED [ 75%]
test_param.py::test_param[4] PASSED [100%]
============================== 4 passed in 0.02s ==============================
(foo) $ pytest test_param.py::test_param -v
============================= test session starts =============================
platform win32 -- Python 3.11.7, pytest-7.4.4, pluggy-1.3.0 -- C:\Users\okken\projects\foo\venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\okken\projects\foo
configfile: pytest.ini
collecting ... collected 4 items
test_param.py::test_param[1] PASSED [ 25%]
test_param.py::test_param[2] PASSED [ 50%]
test_param.py::test_param[3] PASSED [ 75%]
test_param.py::test_param[4] PASSED [100%]
============================== 4 passed in 0.00s ==============================
Activity
The-Compiler commentedon Feb 6, 2024
Bisected to 385796b:
cc @bluetech
main: fix reversed collection order in Session
bluetech commentedon Feb 9, 2024
Thanks @okken for the clear report and @The-Compiler for bisecting. Submitted a fix in #11957.
okken commentedon Feb 9, 2024
@The-Compiler I really need to lean on bisect more often. You're an inspiration.
@bluetech That's awesome that you found a fix so quickly. You rock.
The-Compiler commentedon Feb 9, 2024
Hah, the moment I found out about
git bisect, it felt like magic!For reference, for something like this, what I do is something like:
pip install -e .git bisect startgit checkout 8.0.0and make sure I can reprogit bisect badgit checkout 7.4.4and make sure it works finegit bisect goodgit bisect goodorgit bisect badas necessary.After the initial range, I needed to test < 10 times to find the culprit, so it's something that can usually be done in minutes once there is a nice fast reproducer.
main: fix reversed collection order in Session