Repository navigation
Issue with duplicates handling in Pytest 8 #12083
Description
Activity
Thanks for the bug report @nonatomiclabs, I agree the new behavior is buggy. I will take a look.
The duplicate handling is a bit of a headache and as you said it was quite broken also before #11646.
Reacted by Jean Cruypenynck and YaniceI started looking at this, but it's surprisingly tricky to come up with a self-consistent, intuitive and reasonably backward-compatible logic for the duplicate handling, if you think about it. Though I'll keep trying :)
Reacted by Jean CruypenynckFirst, some explanations of the details that are relevant to duplicate handling, then some thoughts on how it should work.
Collection arguments
The collection arguments are the inputs that pytest starts collecting from. These are usually the positional command line arguments but can also be
testpathsand others - doesn't matter.A collection arg has two parts - the path and (optionally) parts within the file.
All of the collection args are given to
Session.collect()which collects them and yields the initial set of nodes.Session.collect()Suppose the collection arguments are
a/aa/aaa.py,/ab/. ThenSession.collect()will produce two nodes (the order is important):0: <Module a/aa/aaa.py> 1: <Dir a/ab>But remember that nodes form a tree, so to get the full picture we need to look at the parents of the nodes:
<Dir a> <Dir a/aa> <Module a/aa/aaa.py> (0) <Dir a/ab> (1)Note that the
<Dir a>parent is the same for both collection args.genitemsAfter
Session.collect()takes the collection arguments and returns the initial nodes, the functiongenitemstakes each node and recursively expands by callingcollect()on each collector node and yielding item nodes (the leaves, i.e. the tests).So it can look something like this:
genitems(<Module a/aa/aaa.py>) <Module a/aa/aaa.py>.collect() -> <Function test_it>, <Class TestCls> genitems(<Function test_it>) yield <Function test_it> genitems(<Class TestCls>) <Class TestCls>.collect() -> <Function test_meth1>, <Function test_meth2> genitems(<Function test_meth1>) yield <Function test_meth1> genitems(<Function test_meth2>) yield <Function test_meth2>keep-duplicatesflagPytest has a
--keep-duplicatesflag (off by default) documented here but is mostly unspecified.I think we can ignore whatever it does currently and make it mean what we want.
How should duplicates work?
I think first we need to decide on the semantics. I quickly wrote a test case with some scenarios, it is incomplete but can be a basis for discussion. It includes roughly the behavior that I think it should have but it will definitely change with more consideration (and there are some more interesting cases I should add).
Test cases
def test_duplicate_handling(pytester: Pytester) -> None: pytester.makepyfile( **{ "top1/__init__.py": "", "top1/test_1.py": ( """ def test_1(): pass class TestIt: def test_2(): pass def test_3(): pass """ ), "top1/test_2.py": ( """ def test_1(): pass """ ), "top2/__init__.py": "", "top2/test_1.py": ( """ def test_1(): pass """ ), }, ) result = pytester.runpytest_inprocess("--collect-only", ".") result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top1>", " <Module test_1.py>", " <Function test_1>", " <Class TestIt>", " <Function test_2>", " <Function test_3>", " <Module test_2.py>", " <Function test_1>", " <Package top2>", " <Module test_1.py>", " <Function test_1>", "", ], consecutive=True, ) result = pytester.runpytest_inprocess("--collect-only", "top2", "top1") result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top2>", " <Module test_1.py>", " <Function test_1>", " <Package top1>", " <Module test_1.py>", " <Function test_1>", " <Class TestIt>", " <Function test_2>", " <Function test_3>", " <Module test_2.py>", " <Function test_1>", "", ], consecutive=True, ) result = pytester.runpytest_inprocess("--collect-only", "top1", "top1/test_2.py") result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top1>", " <Module test_1.py>", " <Function test_1>", " <Class TestIt>", " <Function test_2>", " <Function test_3>", " <Module test_2.py>", " <Function test_1>", " <Module test_2.py>", " <Function test_1>", "", ], consecutive=True, ) result = pytester.runpytest_inprocess("--collect-only", "top1/test_2.py", "top1") result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top1>", " <Module test_2.py>", " <Function test_1>", " <Module test_1.py>", " <Function test_1>", " <Class TestIt>", " <Function test_2>", " <Function test_3>", "", ], consecutive=True, ) result = pytester.runpytest_inprocess( "--collect-only", "--keep-duplicates", "top1/test_2.py", "top1" ) result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top1>", " <Module test_2.py>", " <Function test_1>", " <Module test_1.py>", " <Function test_1>", " <Class TestIt>", " <Function test_2>", " <Function test_3>", " <Module test_2.py>", " <Function test_1>", "", ], consecutive=True, ) result = pytester.runpytest_inprocess( "--collect-only", "top1/test_2.py", "top1/test_2.py" ) result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top1>", " <Module test_2.py>", " <Function test_1>", " <Module test_2.py>", " <Function test_1>", "", ], consecutive=True, ) result = pytester.runpytest_inprocess("--collect-only", "top2/", "top2/") result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top2>", " <Module test_1.py>", " <Function test_1>", " <Package top2>", " <Module test_1.py>", " <Function test_1>", "", ], consecutive=True, ) result = pytester.runpytest_inprocess( "--collect-only", "top2/", "top2/", "top2/test_1.py" ) result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top2>", " <Module test_1.py>", " <Function test_1>", " <Package top2>", " <Module test_1.py>", " <Function test_1>", " <Module test_1.py>", " <Function test_1>", "", ], consecutive=True, ) result = pytester.runpytest_inprocess( "--collect-only", "top1/test_1.py", "top1/test_1.py::test_3" ) result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top1>", " <Module test_1.py>", " <Function test_1>", " <Class TestIt>", " <Function test_2>", " <Function test_3>", " <Function test_3>", "", ], consecutive=True, ) result = pytester.runpytest_inprocess( "--collect-only", "top1/test_1.py::test_3", "top1/test_1.py" ) result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top1>", " <Module test_1.py>", " <Function test_3>", " <Function test_1>", " <Class TestIt>", " <Function test_2>", "", ], consecutive=True, ) result = pytester.runpytest_inprocess( "--collect-only", "--keep-duplicates", "top1/test_1.py::test_3", "top1/test_1.py", ) result.stdout.fnmatch_lines( [ "<Dir *>", " <Package top1>", " <Module test_1.py>", " <Function test_3>", " <Function test_1>", " <Class TestIt>", " <Function test_2>", " <Function test_3>", "", ], consecutive=True, )
Here are some guidelines I based my expected outcomes on (these are debatable):
- The order of the collection args matters and should be respected as much as possible
- A collection arg specified explicitly (i.e. not a descendant of another arg) should always be duplicated
- Otherwise whether or not to duplicate should depend on
--keep-duplicates - Emitting multiple items with the same nodeid is fine, but emitting the same Item object itself multiple times should never happen
- Collectors should be shared as much as possible, unless explicitly specified requires duplication
Hi, any plan to work on this, someone?
Reacted by cidlik and reuvenstr- marked pytest ignores test file if a single test method is called from the same test file #13240 as a duplicate of this issue
on Mar 9, 2025 - marked When both a directory and a file are passed on the CLI, only the file is run #13494 as a duplicate of this issue
on Jun 27, 2025 - added 5 commits that reference this issue
on Sep 6, 2025 - marked
testpaths = A A/Bdoesn't test folder A (used to work in v7.4.4) #12605 as a duplicate of this issueon Nov 11, 2025 - unmarked
testpaths = A A/Bdoesn't test folder A (used to work in v7.4.4) #12605 as a duplicate of this issueon Nov 23, 2025 - added 2 commits that reference this issue
on Feb 4, 2026 - added a commit that references this issue
on Apr 1, 2026 - added 4 commits that reference this issue
on Apr 27, 2026 - added a commit that references this issue
on Apr 30, 2026 - added a commit that references this issue
on Apr 30, 2026 - added a commit that references this issue
on Apr 30, 2026 - added a commit that references this issue
on Jul 2, 2026 - added a commit that references this issue
on Jul 4, 2026 - added a commit that references this issue
on Jul 19, 2026 - added a commit that references this issue
on Aug 31, 2026
Following an upgrade to Pytest 8, we are seeing a change in the way duplicate items are handled, which does not seem logical/expected to me.
Current behavior
Let's assume we have the following directory structure:
With Pytest 7, if we call
pytest test_one.py tests --collect-only, it returns 3tests:
With Pytest 8, we get only one:
After looking a bit at Pytest's internals, the change looks related to the refactoring done in #11646, where the duplicates handling logic was moved away from the former
_collectfilemethod (which, I guess, operate on a file-per-file basis), to be consolidated ingenitems()(in which a single duplicate in a node will result in the complete node being ignored).Expected behavior
I would expect
test_one.pynot to prevent the further collection of tests insubdirectory, even though it's a duplicate.To be honest, I'm also a bit puzzled by the previous behavior in Pytest 7, as I wouldn't have expected to see
tests/test_one.pytwice (as I didn't pass the--keep-duplicatesoption).Are my expectations correct in the first place, or did I misunderstand the changes to the test collection and the behavior is expected?
Additional information