Repository navigation
Regression in 7.1 ? Windows CI started to fail with "assert mod not in mods" since 7.1 #9765
Description
Now sorry I know this is not a minimal reproducer, but I just want to log this here, and will investigate later:
You can see one Failure here
ERROR collecting test session
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\runner.py:338: in from_call
result: Optional[TResult] = func()
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\runner.py:369: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\main.py:7[23](https://github.com/ipython/ipython/runs/5535021197?check_suite_focus=true#step:8:23): in collect
for x in self._collectfile(pkginit):
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\main.py:575: in _collectfile
ihook = self.gethookproxy(fspath)
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\main.py:539: in gethookproxy
my_conftestmodules = pm._getconftestmodules(
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\config\__init__.py:579: in _getconftestmodules
mod = self._importconftest(conftestpath, importmode, rootpath)
C:\hostedtoolcache\windows\Python\3.9.10\x64\lib\site-packages\_pytest\config\__init__.py:6[24](https://github.com/ipython/ipython/runs/5535021197?check_suite_focus=true#step:8:24): in _importconftest
assert mod not in mods
E AssertionError
Now I'm not sure what this does, but my quick understanding is that it checks whether conftest.py itself is not in the list of collected modules ? We might definitely do something wrong on IPython side, but it's strange as other non-windows CI are passing and regardless of whether this is something I did wrong, can the assert get an explanation messages as second argument ?
Downgrading to <7.1 fixes the issue.
Again I'll try to get a MVP/small example, but it may take me some time, and I guessed maybe one of you will immediately know why this is happening.
- a detailed description of the bug or problem you are having
- output of
pip listfrom the virtual environment you are using - pytest and operating system versions
- minimal example if possible
Activity
bluetech commentedon Mar 14, 2022
Thanks for the report, I will take a look when I get a chance. 7.1 had some internal conftest refactorings. If you manage to provide a reproduction that'd be great.
The-Compiler commentedon Mar 14, 2022
One thing that seems somewhat suspicious is how IPython's
pytest_ipdoctest.pyseems to have some custom code around importing conftest files:https://github.com/ipython/ipython/blob/d7cc335a36cdfe4159c9b88c94311866c923a547/IPython/testing/plugin/pytest_ipdoctest.py#L635-L646
Carreau commentedon Mar 14, 2022
Yeah, it might be something on our end, we have to do some pretty weird things seeing we also an interpreter. So I would not be surprised if the problem is from our plugins.
Carreau commentedon Mar 14, 2022
It does seem like removing the plugin fixes the issue,
I'm not quite sure why it's windows only, and I haven't written it, so I'm not sure how I can fix it.
Feel free to close, worse case I'll run the plugin only on linux CI.
Avoid regression on pytest 7.1
gonzalocasas commentedon Mar 14, 2022
Not sure if it helps, I know it's not a slim repro setup, but this repo was just pinned to
<7.1because it was consistently reproducible in there: https://github.com/compas-dev/compasCI: use pytest<7.1 on Windows
bashtage commentedon Mar 18, 2022
Seeing same in statsmodels on Windows.
https://dev.azure.com/statsmodels/statsmodels-testing/_build/results?buildId=4410&view=logs&j=9c2ba6b6-9ef1-5300-06cd-4db9cc34ebf3&t=f07685ef-4dee-5864-e861-bb3b9ab4fc6b
Carreau commentedon Mar 18, 2022
@gonzalocasas and @bashtage I see both your CI install IPython in them, could you try pinning IPython to
<8to see if it's a problem with the IPython doctest plugin itself (that should not be present in 7.x)bashtage commentedon Mar 20, 2022
@Carreau I get the same error when I
pip install ipython<8. I also get the same error when Ipip uninstall ipython. Works find with 7.0.x, broken in 7.1.x.bashtage commentedon Mar 20, 2022
I can also get this error when I reun specific tests:
jgb commentedon Apr 8, 2022
I've got the same error, but on linux! Can't run any test on 7.1.x, reverting back to 7.0.x fixes the issue:
bnavigator commentedon Apr 26, 2022
FYI: I also get this in an rpmbuild of asdf for openSUSE Linux. I attribute it to some import mismatch for the conftest.py installed into the sitelib.
Fails:
Have asdf installed in sitelib but collect from asdf in source directory. This worked with pytest < 7
Works:
move the asdf source directory away and directly test with
--pyargs asdf60 remaining items
lesteve commentedon Jan 8, 2024
@pllim this seems very similar to the Scipy one,
os.path.normcaseissue, capitalLLib(inmod.__file__) vs smallllib(inconftestpath):I think this could be fixed by #11708. Hopefully we manage to push the PR forward tomorrow.
pllim commentedon Jan 8, 2024
Thanks, @lesteve ! I will be happy to test that PR when it is ready.
lesteve commentedon Jan 9, 2024
@pllim and maybe @h-vetinari or @tylerjereddy, can you try installing the code from #11708 and see whether that fixes your issue 🤞?
pllim commentedon Jan 9, 2024
pllim commentedon Jan 9, 2024
@lesteve , I applied the patch from fcharras to 8.0rc1 directly at https://github.com/pllim/pytest/tree/debug-8.0rc1 (pllim@22949f2). It went further this time but errored out because it cannot find test fixtures. We have a root level
root/conftest.pyand aroot/package/conftest.py. The first one is so that tox picks up the pytest header stuff. The second one is where all the fixtures are and we need it at the package level so we can runpackage.test()command for any installed package (this uses the astropy test runner machinery).Log: https://github.com/spacetelescope/jdaviz/actions/runs/7465954413/job/20316247352?pr=2654
For completeness, I can confirm that this problem only still happens on Windows. OSX passed and Linux failed with some transient remote data access problem that is unrelated to pytest itself.
lesteve commentedon Jan 9, 2024
OK, so at least you avoid the
assert mod not in modsand it goes a bit further, so I would count that as a win for the PR 😉 !Your debug branch looks fine to me. Right now, I don't really understand how such a change could cause some fixtures to not be registered.
It could potentially be another pytest 8.0rc1 regression, hard to tell since I am by no means a pytest expert ...
pllim commentedon Jan 9, 2024
Yes, I agree that the "assert mod not in mods" problem is solved in #11708
pllim commentedon Jan 9, 2024
I also just found out that this "mod not in mods" problem on Windows is not a problem at all if I switch from Python 3.10 to Python 3.11 🤪 🤷♀️
Normalize conftest paths
woutdenolf commentedon Jan 14, 2024
For reference, in case this needs to be reproduced (Windows 11, python 3.8.10):
This works fine (well there is an issue with the
papermilldependency but that's not related to pytest)lesteve commentedon Jan 15, 2024
@woutdenolf this is similar to
scipyandjdavizsee #9765 (comment),lib(smalll) vsLib(capitalL).I think this is fixed in
main(since Saturday January 13 ~9PM Grenoble time) and in has been back-ported in the8.0.xbranch.I double-checked and I don't get the
assert mod not in modson Windows and Python 3.8, when doing this:bluetech commentedon Jan 15, 2024
For reference, the simple "Solution 3" normalize-as-early-as-possible approach (implemented here) does not work -- the problem is that, while
Foo/Mod.pyandfoo/mod.pyare equivalent paths in a case-insensitive filesystem,Foo.Modandfoo.modare not equivalent Python module names.woutdenolf commentedon Jan 15, 2024
The link is broken. What do you mean by "normalize"? Normalize what and how?
I don't see the problem. You cannot import
Foo/Mod.pyby doingimport foo.mod. Could you create an example?I'm asking because I made a proposal that normalizes
anymodule.__file__but preserves symlinks: #11821. The tests pass so if there is a problem with this, it is not covered by the tests.lesteve commentedon Jan 16, 2024
The correct link for the commit is 203889b