Quadratic blowup when using TestCase subtests in python 3.10

Running the following test using pytest 9.0.0 in python 3.10 leads to a quadratic blowup:

from unittest import TestCase

N = 1  # 1, 2, 3, 4, ...

class TestQuadraticBlowup(TestCase):
    def test_quadratic_blowup(self) -> None:
        for i in range(1000 * N):  # multiplying by 1000 to make bigger steps
            with self.subTest():
                pass

For a given N, it takes O(N^2) time to complete. Here are a few samples:

N=10 -> 10s
N=20 -> 35s
N=30 -> 73s
N=40 -> 130s

When running this test using python -m unittest test.py, it takes less than 0.5s for N=40.

I traced this test a little:

TestCaseFunction.runtest() is called (unittest.py#360)
it calls testcase(result=self) (unittest.py#389) which is TestCase.run() (case.py#557)
it constructs outcome = _Outcome(result) (case.py#582) and assigns to self a line later
the actual test is executed with self._callTestMethod(testMethod) (case.py#591)
the test repeatedly calls with self.subTest() which is TestCase.subTest() (case.py#480)
TestCase.subTest() constructs _SubTest (case.py#495)
it enters that subtest with with self._outcome.testPartExecutor(self._subtest, isTest=True): yield (case.py#497) which is _Outcome.testPartExecutor() (case.py#55)
when the subtest finishes successfully, that _Subtest instance is appended to the _Outcome.errors list with None as error (case.py#79)
after all subtests finish, TestCase.run() calls self._feedErrorsToResult(result, outcome.errors) (case.py#599)
- note that outcome.errors has 1000 * N elements
TestCase._feedErrorsToResult() iterates over all those errors (case.py#511)
- this gives the first, outer O(N)
for each error, it calls result.addSubTest(test.test_case, test, exc_info) (case.py#513) which is TestCaseFunction.addSubTest() (unittest.py#404)
TestCaseFunction.addSubTest() eventually iterates over self.instance._outcome.errors (unittest.py#455) which is the same list as outcome.errors above (1000 * N elements)
- this gives us the second, nested O(N), which in total gives us O(N^2) for the entire test

There might be similar issues with skipped tests (unittest.py#450). Another potential hot spot is unittest.py#318.

pip list output:

Package           Version
----------------- -------
exceptiongroup    1.3.0
iniconfig         2.3.0
packaging         25.0
pip               23.0.1
pluggy            1.6.0
Pygments          2.19.2
pytest            9.0.1
setuptools        65.5.0
tomli             2.3.0
typing_extensions 4.15.0

Python: 3.10.17

Interesting. From your analysis of the code, do you think there is some workaround that pytest can do on Python 3.10 only to fix it?

(I was going to suggest a workaround of disabling the subtests plugin using -p no:subtests, but that doesn't work currently because the unittest plugin does subtest stuff even if the subtests plugin is disabled, we might want to fix that)

I'm not familiar with the codebase to be able to suggest a well-thought-out solution, and I didn't see any quick and obvious workarounds.

My ideas all revolve around some kind of caching of the values non_subtest_skip (unittest.py#448) and subtest_errors (unittest.py#453). I crafted a somewhat hacky solution that replaces the original _Outcome class with a class that caches those values, and invalidates the cache when _Outcome.skipped or _Outcome.errors is modified (either replaced completely, or some elements are removed/added/replaced). This patch makes the test with N=100 in python 3.10 finish in roughly the same amount of time (6~8s) as in python 3.11. Precisely, it runs ~1.25 times longer, but this might be because the code performs additional checks in the first place in python 3.10.

Let me know if this kind of approach seems reasonable.

Please open a draft pr for easy review

Fix quadratic-time behavior when handling `unittest` subtests in Pyth…

chore(deps): update pytest (major)

Merge branch 'renovate/pytest-9.x' into 'main'

🔒️ Update dependency pytest to v9 [SECURITY] (#16)

[SECURITY] Update dependency pytest to v9 [SECURITY] (#2)

Update dependency pytest to v9 (#35)

Navigation Menu

Repository navigation

Quadratic blowup when using TestCase subtests in python 3.10 #13965

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Quadratic blowup when using TestCase subtests in python 3.10 #13965

Description

Activity

bluetech commented on Nov 14, 2025

AkaZecik commented on Nov 15, 2025

RonnyPfannschmidt commented on Nov 15, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions