Thread safety and Pytest

@ngoldbaum

(This is a high-level issue that proposes making pytest more thread safe; I'm definitely looking for feedback, discussion, and confirmation this is a good direction here.)

Background

This issue comes out of a brief conversation on the pytest discord:

[@ngoldbaum] out of curiosity, if there were funding for work on improving pytest’s thread safety, is that something that the project might be interested in? For the work we’ve been doing so far we’ve contracted David Woods to work on Cython, David Hewitt to work on PyO3, and Liam DeVoe to work on Hypothesis. Is there an opportunity for pytest as well?

[@The-Compiler] Personally I'm not too interested in it being thread-safe, but I'd absolutely support using our Opencollective funds to make it happen

[@RonnyPfannschmidt] my sentinent is the same - but i also want to note that i'd prefer thread safety in a manner that uses queues and non-shared state for fixtures & co (aka xdist-alike, just without separate node objects) - the key contention would be log/stdio capture state + managing pytest stores on nodes in a controlled manner

I don't presume to know the best direction for thread-safety in pytest, so I'm opening this issue to get some feedback on my understanding.

While this may be obvious to all involved, I want to at least give a brief motivation for thread-safety here. Python is working towards removing the global interpreter lock, in a python build version called free-threading. As free-threading progresses, it seems likely that more people will want to run multiple tests simultaneously in separate threads. In particular, it would be nice for free-threading development if pytest-xdist supported threading (in place of multiprocessing) as its worker primitive, so existing test suites can be run with pytest-xdist in threading mode to flush out concurrency bugs.

Initial proposal

Proposal

I would summarize all of pytest as two (very complex) steps:

Test collection.
Test execution.

with some calls to pluggy thrown in during both.

From talking with @Zac-HD, I understand there's some appetite for parallelizing collection in pytest itself using threads, perhaps as a consequence of folding pytest-xdist into pytest. This would require making (1) test collection thread safe. I think that parallelized test collection is "merely" a performance improvement, so I'm not proposing it in this issue, though it may be something to keep in mind.

Instead I want to focus on (2); making test execution thread-safe. Here is my understanding of the required work there:

Make pluggy thread-safe, as hooks will be called from multiple threads.
- It's possible this work is either minimal or zero, if pluggy is already thread safe.
Guarantee all (feasible) built-in fixtures are thread-safe.
- I don't (yet) know pytest internals enough to judge how difficult this will be.
- Some fixtures, like monkeypatch and capsys, don't seem possible to make thread-safe. Perhaps someone knows an arcanely cursed way to achieve this?
Make pytest data collection during tests thread-safe.
- the tracking of the number of xfails, successes, failures, etc.
Document the thread-safety guarantees, including what is and is not safe.
Add an opt-in execution model to pytest-xdist which uses threads instead of processes.
- If we're done our job right up to this point, this should be conceptually simple, though it may require some involved changes to pytest-xdist to support this.

And finally;

Test the heck out of this.
- I put a substantial amount of effort into testing thread-safety when migrating Hypothesis: Add threading CI job HypothesisWorks/hypothesis#4489. I would expect to do so again here, though I don't yet know the best way to do so.

Open questions

If we do leave fixtures like monkeypatch thread-unsafe, how should this interact with pytest-xdist's proposed threading mode? A few options: do nothing; print a warning; error; or skip the test. I don't like half of these!

cc: @ngoldbaum, @lysnikolaou

Refined Proposal

separate SetupState from Session
- one Session collection-tree should be shared among all threads
- each thread gets a separate SetupState and therefore a separate instance of each fixture
audit Stash uses for thread-safety
audit cacheprovider for thread-safety
make capsys thread-aware, with integrations for the per-thread SetupState

Issues and PRs

there are multiple missunderstandings here

pluggy is threadsafe
nomal fixtures shouldnt have to care about thread-safety unless multi thread fixtures are introduced
monkeypatch will never be threadsafe, 🤷
capsys cannot be threadsafe as is as posix file descriptors are not thread isolated
- multithraded pytest will need stand-in objects with context locals
collection thread safety is not necessarily required, its ok to leave it as a secondary concern
xdist should move focus to multi process/multi computer/running in remote environments not gain threads
sheduler protocols in process can be more powerful than xdist due to direct data access
execution thread safety will require a more elaborate sheduling mechanism than modifyitems
- the goal would be to have independent runtestloops with distinct setupstates
node objects and their stores/stashes need thread safety concerns

so my primary concerns would be detangling setup-state, enabling thread safety in node objects (in particular higher scope nodes)
then would be a follow-up with details like monkeypatch crossover (very hellish, most likely unsafe), log/stdio capture with python level standings & co

fixtures wouldn't need direct thread safety due to explicitly distinct thread safety

but we do need a plan for fixtures that mess with shared state (like modules/monkepatch/fdcapure)

we also need to investigate if subinterpreters will help

also additional considerations are needed for fun bits like gui framework eventloops going broke on non-main threads and other curious ecosystem traps

@ngoldbaum

This broadly makes sense, but I wanted to clarify a few things:

xdist should move focus to multi process/multi computer/running in remote environments not gain threads
sheduler protocols in process can be more powerful than xdist due to direct data access

I had the impression that pytest-xdist gaining an optional threading execution model (pytest -n 8 -> pytest -n 8 --use-threads, pending naming) was a desirable outcome from all parties here. Is that not the case on pytest's end?
If not, do you see any place in pytest itself to ship such a builtin threaded execution model? Or would you prefer to simply focus on making threaded execution possible to implement, by guaranteeing thread-safety in pytest, and leave implementing it to third-party plugins?
It seems you want the latter, but possibly adding or changing lifecycle hooks to support the implementation of this by a plugin? (this is my reading of "execution thread safety will require a more elaborate sheduling mechanism than modifyitems").

execution thread safety will require a more elaborate sheduling mechanism than modifyitems

the goal would be to have independent runtestloops with distinct setupstates

Repeating back in my own words: "fixture instances should not be shared across threads; rather, each thread should maintain its own SetupState which pushes/pops/instantiates fixtures for that thread. Fixtures which inherently modify shared state (monkeypatch etc) will need special consideration." This makes sense to me, and I think thread-isolated fixtures is actually what @ngoldbaum recommended initially, and I misread.

the core focus of pytest should be making it possible

we also have to consider async task models as possible

xdist has a focus on different processes, execnet and different nodes alltogether

we have had multiple experiments of having a simplefied xdist and i think its going to be something along the lines of splitting it up into scheduling concerns - thread concerns, async concerns - running in remote infra concerns

@RonnyPfannschmidt

@RonnyPfannschmidt would you be able to write up your proposal for what Pytest should look like in the free-threaded future? Once we have agreement on that, we can work out how to get there incrementally.

For example, I think most people only use xdist to get multi-core local execution, and we could (imo should eventually) obsolete that with native multithreading in core pytest (with a pull-from-queue efficient scheduler). Then xdist can be multi-computer stuff only, though tbh I think this is usually a mistake and people should do independent runs then combine reports.

But the blocker to all of this seems to be having an agreed list of thread-safety-improving tasks which people can work on!

we are probably able to do a rough proposal - but i suspect quite a bit of the details will change as we work towards it and learn

some key elements that come to mind off hand are:

context aware capture for stdio/logging with the awareness that posix level capture is never thread-safe (this might also help to resolve some long standing developer experience bugs as people keep misunderstanding correct capture configuration for stdio and logging)
a strategy for monkeypatch - while none may get away with patching some objects , patching the environment, global things and cwd is a potential disaster
un-tangleing setupstate from nodes/session to enable having more than one per pytest session (this also would be great strides towards a more maintainable fixture system and eventually support for considering type annotations)
validating if/how the store objects in nodes need to accommodate multi threading
evaluate if we can do a sensible worker pool version of running collectors
evaluate scheduling and setup/teardown lineup - in particular wrt expensive higher scope fixtures as

Regarding monkeypatching etc, I'd say some tests are never going to be thread safe. We'd need a way to mark tests as not thread safe, or more likely the other way around.

I think I'd want to be able to explicitly mark fixtures as safe, unsafe or hazard plus propagate implications to unmarked
Fixtures

I agree we'll need that eventually, but I'd start by working on the minimal core, e.g. pytest --capture=no -p no:monkeypatch without use of fixtures. Once we get multiple pytest sessions and node-stores to be threadsafe and tested as such, we can incrementally add all the other stuff users need, and finally the (very, perhaps fundably) nice-to-haves like parallel collection.

I expect that we'll also see some improvements elsewhere in the ecosystem which make this easier, e.g. here's a proposal for contextlib.redirect_stdout() to become thread-aware which we'd be able to use (eventually).

Well never be able to use those contextmanagers unless python fundamentally changes io redirection

For thread/logging safety Well have to essentially have thread context aware replacements that include pytest integration

Well never be able to use those contextmanagers unless python fundamentally changes io redirection

I was also worried about warnings, but it seems that catch_warnings is handled in Python 3.14, so pytest's warnings plugin should work at least.

un-tangleing setupstate from nodes/session to enable having more than one per pytest session

Do I understand correctly that the idea here is that collection is done in a non-parallel phase on the main thread, and the same collection tree is shared among multiple parallel runners? As opposed to xdist which runs collection per worker.

I was also worried about warnings, but it seems that catch_warnings is handled in Python 3.14, so pytest's warnings plugin should work at least.

And in-practice I haven't seen any thread safety issues due to uses of pytest.warns in 3.14, so any remaining thread safety issues in pytest itself should be hard to trigger, if there are any.

@ngoldbaum

Thanks for the discussion here. Here's a more refined todolist:

separate SetupState from Session
- one Session collection-tree should be shared among all threads
- each thread gets a separate SetupState and therefore a separate instance of each fixture
audit Stash uses for thread-safey
make capsys thread-aware, with integrations for the per-thread SetupState

The working goal in my mind is to enable pytest support for the following implementation of a parallel pytest_runtestloop:

from queue import Queue, Empty
from concurrent.futures import ThreadPoolExecutor

n = 2

def pytest_runtestloop(session):
    queue = Queue()
    for item in session.items:
        queue.put(item)

    def worker():
        try:
            item = queue.get_nowait()
        except Empty:
            return

        while item is not None:
            try:
                next_item = queue.get_nowait()
            except Empty:
                next_item = None

            item.config.hook.pytest_runtest_protocol(
                item=item, nextitem=next_item
            )
            item = next_item

    with ThreadPoolExecutor() as executor:
        futures = [executor.submit(worker) for _ in range(n)]
        for future in futures:
            future.result()

    return True

which could be implemented by e.g. pytest-run-parallel or a simplified pytest-xdist as they see fit. (I thought implementing this parallelism in a plugin might require additional pytest hooks, but I think pytest_runtestloop covers it.)

I don't think Quansight or @ngoldbaum necessarily sees it as a requirement that pytest itself ship such a parallel execution model, merely that pytest makes it possible to implement one, though feel free to correct me. As a result, I don't think we yet have to address concerns like "monkeypatch is inherently thread-unsafe", because that can be a caveat/requirement of writing such a parallel pytest_runtestloop. But if we did, then an extension is:

figure out what to do about inherently-unsafe fixtures like monkeypatch
- possibly by requiring each fixture to opt-{in, out} to thread safety

[I also forgot to mention in the original post, but #13679 would be wonderful to have merged for pytest-run-parallel!]

Do I understand correctly that the idea here is that collection is done in a non-parallel phase on the main thread, and the same collection tree is shared among multiple parallel runners? As opposed to xdist which runs collection per worker.

This is my understanding yes, with a possible eventual goal to make collection itself parallel as an extension to this issue.

note that this very naive testloop implementation wrecks havoc wrt sensible reorder to minimize setup/teardown

one would want shared setup/teardowns in "runs" that minimize the need to setup/teardown specific fixtures

aka try to make high scope fixtures sticky to specific treads so we remove the need for setup/teardown cycles

oh - and one extra fun consideration - specific event loops will break outside of the main thread (i recall at least QT and gtk)

context aware capture for stdio/logging with the awareness that posix level capture is never thread-safe (this might also help to resolve some long standing developer experience bugs as people keep misunderstanding correct capture configuration for stdio and logging)

We ran into this working on pytest-run-parallel as well. See this comment in the CPython repo for an outline of a way to do this.

EDIT: And I see Zac posted about this later in the thread...

@bwhitt7

Liam's summary seems like an excellent plan and would substantially help ecosystem efforts to improve multithreaded test coverage. Currently there's lots of manual patching that's needed to work around pytest thread safety issues that Liam's plan would fix. If each thread had a local fixture and if setup and teardown happened in a thread-safe manner, @bwhitt7's recent work on using pytest-run-parallel in the NumPy CI would have been a lot easier, just as a concrete example. Pairing pytest-run-parallel with compiler sanitizers is already pointing out real bugs.

Thanks all for your active participation and ideas on this thread, I'm really excited about where this can go.

@RonnyPfannschmidt

To limit the scope of this, I think we should limit the usage of --threads (just gonna assume that will be the flag 😀 ) to the freethreaded build. The reasons being:

Without free threading the feature is probably not so useful (not useless, but less useful for the main purpose of speeding up the test suite).
It allows us to focus on a new Python baseline (e.g. no Python 3.10 headaches).
The free threaded build enables two very relevant flags by default, sys.flags.thread_inherit_context and sys.flags.context_aware_warnings. See here (python docs are currently 503 so linking to github instead).

With sys.flags.thread_inherit_context I think we can make a usable capsys replacement using ContextVar. Using threading.local alone wouldn't be good since it will fail to capture in tests which themselves launch threads.

Using ContextVar is problematic as well, tests which launch threads or tasks without inheriting the context will then fail to capture. For threads with the sys.flags.thread_inherit_context I'd be less worried. Async tests probably do this more, although I think even asyncio.run inherits context from the running thread, so maybe it's fine.

Regarding fixtures @RonnyPfannschmidt wrote:

aka try to make high scope fixtures sticky to specific treads so we remove the need for setup/teardown cycles

@Liam-DeVoe wrote:

each thread gets a separate SetupState and therefore a separate instance of each fixture

We should decide what we want to do here.

Liam's idea to have entirely separate SetupState's is the simplest way, similar to how xdist works. Is it workable in practice?

What I think Ronny is aiming at is for threads to share higher-scope fixtures. This will bring a significant advantage over xdist, but will no longer be a "shared nothing" architecture, so a significant up in the complexity. Also due to setup/teardown expectations, it will have heavy synchronization overhead and basically serialize the threads in some cases, unless the scheduler (what assigns tests to threads) is smart enough to avoid that. (And don't forget higher-scoped fixtures can be parametrized, meaning setup/torndown multiple times.)

@RonnyPfannschmidt did I understand correctly?

Surely a simple "fixture is higher scope => fixture is sticky to one thread" wouldn't work, since a single session-scoped fixture will force all tests to a single thread.

Stickyness is about parameterized fixtures

We should cluster based on sheduling setup/teardown

No actual instances should share for now

To limit the scope of this, I think we should limit the usage of --threads (just gonna assume that will be the flag 😀 ) to the freethreaded build

Your reasons make sense, but since it's possible to opt into threads_inherit_context and context_aware_warnings on the GIL-enabled build and opt out on the free-threaded build, maybe instead just make it conditional on the value of the startup flags?

I've found it very useful in-practice to check whether a bug reproduces on the free-threaded build and the GIL-enabled build or just the free-threaded build. Often also it helps to set the thread switch interval to a very small value on the order of a microsecond to trigger multithreaded bugs more often on the GIL-enabled build.

although I think even asyncio.run inherits context from the running thread

yup!

Details

Python 3.12.10 (main, Apr 28 2025, 11:48:32) [Clang 17.0.0 (clang-1700.0.13.3)]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
Tip: `?` alone on a line will brings up IPython's help

In [1]: import asyncio
   ...: import contextvars
   ...: import contextlib
   ...:
   ...: v = contextvars.ContextVar("test")
   ...:
   ...:
   ...: async def main():
   ...:     print(v.get())
   ...:
   ...:
   ...: @contextlib.contextmanager
   ...: def test_context(value):
   ...:     token = v.set(value)
   ...:     try:
   ...:         yield
   ...:     finally:
   ...:         v.reset(token)
   ...:
   ...: with test_context("context variable has a value!"):
   ...:     asyncio.run(main())
   ...:
context variable has a value!

a threaded mode can also make sense on non-free-threading

any acceptance test thats network bound on something like selenium/playwright or rest services can gain wins not dissimilar from xdist (with the caveat that the only win is intersparsed concurrency

but its also abundantly clear that an cpu-bound execution will not take a win home there

An update - here's my current progress on thread safety: https://github.com/pytest-dev/pytest/compare/main...Liam-DeVoe:pytest:threading-work?expand=1.

This branch requires each thread that runs runtestprotocol to follow a pytest-thread-{n} naming convention, which pytest uses to track that thread as a "pytest thread". This branch uses ContextVar and assumes thread_inherit_context=1 is set, so that threads spawned by tests inherit knowledge of the current "pytest thread" and integrate accordingly.

In this branch, each pytest thread gets its own copy of each fixture's value. Fixtures are effectively thread-local. (*they are really pytest-thread-local, as threads spawned inside tests still use that pytest thread's fixture value).

Note that I think this is a somewhat less structural refactoring than the separation of SetupState and Session which has been suggested in this thread. I'm curious to hear thoughts on this, and whether this should go further in refactoring pytest internals.

I'm pretty excited to report that this branch lets me run most of the Hypothesis test suite in parallel, when combined with the appropriate conftest.py:

Details

import threading
from concurrent.futures import ThreadPoolExecutor
from queue import Empty, Queue

n = 3


def pytest_runtestloop(session):
    queue = Queue()
    for item in session.items:
        queue.put(item)

    def worker(worker_id):
        threading.current_thread().name = f"pytest-thread-{worker_id}"
        try:
            item = queue.get_nowait()
        except Empty:
            return

        while item is not None:
            try:
                next_item = queue.get_nowait()
            except Empty:
                next_item = None

            item.config.hook.pytest_runtest_protocol(item=item, nextitem=next_item)
            item = next_item

    with ThreadPoolExecutor() as executor:
        futures = [executor.submit(worker, i) for i in range(n)]
        for future in futures:
            future.result()

    return True

Stumbling blocks are that it currently requires --capture no -p no:cacheprovider. The former will definitely require some work to fix, whereas I'm merely somewhat skeptical of the latter's thread-safety, and am keeping it out of an abundance of caution for now.

Please open a draft pr for easy review

There's a number of bugfix prs surrounding fixtures si theres conflict potential

As for capture i believe we need threadmapped fd capture that has context aware replacements for the streams (bur that idea is not yet batle trsted)

Refactor update_manifest_for_old_job cli tests

Navigation Menu

Repository navigation

Thread safety and Pytest #13768

Background

Initial proposal

Proposal

Open questions

Refined Proposal

Issues and PRs

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Thread safety and Pytest #13768

Description

Background

Initial proposal

Proposal

Open questions

Refined Proposal

Issues and PRs

Activity

RonnyPfannschmidt commented on Sep 30, 2025

RonnyPfannschmidt commented on Sep 30, 2025

Liam-DeVoe commented on Sep 30, 2025

RonnyPfannschmidt commented on Sep 30, 2025

Zac-HD commented on Sep 30, 2025

RonnyPfannschmidt commented on Oct 1, 2025

bluetech commented on Oct 3, 2025

RonnyPfannschmidt commented on Oct 3, 2025

Zac-HD commented on Oct 3, 2025

RonnyPfannschmidt commented on Oct 4, 2025

bluetech commented on Oct 4, 2025

ngoldbaum commented on Oct 4, 2025

Liam-DeVoe commented on Oct 5, 2025

RonnyPfannschmidt commented on Oct 5, 2025

RonnyPfannschmidt commented on Oct 5, 2025

ngoldbaum commented on Oct 13, 2025

ngoldbaum commented on Oct 13, 2025

bluetech commented on Nov 21, 2025

RonnyPfannschmidt commented on Nov 21, 2025

ngoldbaum commented on Nov 21, 2025

RonnyPfannschmidt commented on Nov 21, 2025

Liam-DeVoe commented on Jan 21, 2026

RonnyPfannschmidt commented on Jan 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions