Indirect session fixture reordering not working as expected

@dina809

There seems to be a problem with our reordering code that optimizes higher-level fixtures' execution order.

import pytest

@pytest.fixture(scope="session")
def prepare(request):
    print("\nprepare setup with {!r}".format(request.param))

@pytest.mark.parametrize('prepare', ["dina"], indirect=True, scope="session")
def test_1(prepare):
    print("Test1 done\n")

@pytest.mark.parametrize('prepare', ["mor"], indirect=True, scope="session")
def test_2(prepare):
    print("Test2 done\n")

@pytest.mark.parametrize('prepare', ["dina"], indirect=True, scope="session")
def test_3(prepare):
    print("Test3 done\n")

I get this output:

λ pytest .tmp\test_fix_session.py -sq

prepare setup with 'dina'
Test1 done

.
prepare setup with 'mor'
Test2 done

.
prepare setup with 'dina'
Test3 done

.
3 passed in 0.02s

We would expect it to reorder the tests to [test_1, test_3, test_2] so the fixture would execute only twice, instead of 3 times as it is doing now, so definitely looks like a bug (unless I'm missing something).

Thanks @dina809 for the original report (#8328).

Originally posted by @nicoddemus in #8328 (comment)

I'm gonna have a look into this one next on my hit list, I've read the other thread for all the context and will start debugging soon

Consider param index and value for item reordering

@nicoddemus

@nicoddemus This issue hit me as well, hopefully I found the root cause and a possible fix, see this commit

Reordering test items only took parameter indexes into account. This
works fine for direct parametrization. But for indirect parametrization,
the parameter index is always 0, and therefore every item went into the
same argkey group.

Now also add the parameter value to the argkey. This is enough
information to establish different argkey groups and make the grouping
algorithm work.

The output from your minimal reproducer is now as expected

prepare setup with 'dina'
Test1 done
Test3 done
prepare setup with 'mor'
Test2 done

but the change makes 3 tests fail (test_parameters_without_eq_semantics, test_parameters_without_eq_semantics, test_parametrize_issue634). I've not yet looked at them in detail.

Could you please give some short feedback? If you think I'm on the right track, I'll try to understand and fix the regression and open a PR here.

@haxtibal

Hi @haxtibal, that's great! I looked at the code and it makes sense.

The failing tests are about making sure that pytest works even if the argvalues can't compare with == (raise an error for example).

I think a solution is to return a wrapper instead of the actual object, something like:

@attr.s(autoattribs=True, eq=False)
class SafeEqWrapper:

    obj: object

    def __eq__(self, other: object) -> Any:
        try:
            res = self.obj == other
            bool(res)
            return res
        except Exception:
            # perhaps return False?
            return id(self.obj) == id(other)

And then use it like this:

key: _Key = (argname, param_index, SafeEqWrapper(param_value))

Feel free to open a PR even if incomplete, we can discuss this over the code. 👍

Fix reordering of indiretly parameterized fixture

Fix test reordering for indirect parameterization

@nicoddemus

Thank's for your suggestions @nicoddemus ! We can switch over to #9350 now.

Fix test reordering for indirect parameterization

Add and adapt tests

Fix test reordering for indirect parameterization

Add and adapt tests

Navigation Menu

Repository navigation

Indirect session fixture reordering not working as expected #8914

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Indirect session fixture reordering not working as expected #8914

Description

Activity

Formartha commented on Jul 15, 2021

symonk commented on Jul 31, 2021

haxtibal commented on Nov 27, 2021

nicoddemus commented on Nov 28, 2021

haxtibal commented on Nov 29, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions