KeyError fired from importlib during collection when running with --import-mode=importlib on a directory containing a same-named one

@criemen

A detailed description of the bug or problem you are having

When creating the following structure:

x/
├─ y/
│  ├─ y/
│  ├─ test_y.py

then pytest --import-mode=importlib fails with the following cryptic message:

================================= ERRORS =================================
_____________________ ERROR collecting x/y/test_y.py _____________________
<frozen importlib._bootstrap_external>:1448: in find_spec
    ???
<frozen importlib._bootstrap_external>:1222: in __init__
    ???
<frozen importlib._bootstrap_external>:1238: in _get_parent_path
    ???
E   KeyError: 'x'

To trigger this bug:

the directory containing the test needs to be within at least another directory, it can't be directly underneath the root path
the directory containing the test must in turn contain another directory with the same base name

Running with --pdb gives a slightly more useful backtrace:

  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/runner.py(341)from_call()
-> result: Optional[TResult] = func()
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/runner.py(389)collect()
-> return list(collector.collect())
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(548)collect()
-> self._register_setup_module_fixture()
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(561)_register_setup_module_fixture()
-> self.obj, ("setUpModule", "setup_module")
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(287)obj()
-> self._obj = obj = self._getobj()
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(545)_getobj()
-> return importtestmodule(self.path, self.config)
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/python.py(492)importtestmodule()
-> mod = import_path(
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/pathlib.py(565)import_path()
-> mod = _import_module_using_spec(
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/pathlib.py(660)_import_module_using_spec()
-> parent_module = _import_module_using_spec(
  /home/redsun82/.cache/pypoetry/virtualenvs/foo-4uHc0o_q-py3.10/lib/python3.10/site-packages/_pytest/pathlib.py(633)_import_module_using_spec()
-> spec = meta_importer.find_spec(module_name, [str(module_location)])
  <frozen importlib._bootstrap_external>(1448)find_spec()
  <frozen importlib._bootstrap_external>(1222)__init__()
> <frozen importlib._bootstrap_external>(1238)_get_parent_path()

Output of `pip list` from the virtual environment you are using

I tried this on a minimal poetry env with just pytest

-------------- -------
exceptiongroup 1.2.1
iniconfig      2.0.0
packaging      24.1
pip            24.0
pluggy         1.5.0
pytest         8.2.2
setuptools     70.0.0
tomli          2.0.1

pytest and operating system versions

pytest 8.2.2. Encountered this both with python 3.10 and python 3.12, so the python version doesn't seem to be a factor.

Running on Ubuntu 22.04.4 LTS.

cc @criemen who encountered this

Feels like this might be an upstream issue; we should try to get a reproducer using stdlib code only.

Fix: Using `--import-mode=importlib`, a directory with the same name …

Fix KeyError with importlib mode (directories with same name) (#12752)

Fix KeyError with importlib mode (directories with same name) (#12752) (

Fix KeyError with importlib mode (directories with same name) (pytest…

cheers for fixing this, folks! I noticed a side-effect of this though: if pytest is running a test on a module that imports e.g.

from cow.moo.farm import udder

and there is an /cow/moo/moo.py, so module moo.py is in a dir identically called moo and there is no __init__.py in /moo then the test fails with an ImportError - not a biggie since IMHO since one uses package-level imports, then there must be init files all over the place where modules live, it's all good, but just wanted to let you know if yous get some others complaining about it 🍺

Seeing a similar issue as #12592 (comment)

Perhaps it's my misunderstanding of namespace packages, but I typically don't have any __init__.py files in my repo. And I'm seeing an AttributeError for my module now:

 AttributeError: module 'third_party.my_protobuf_utils.my_protobuf_utils' has no attribute '_read_varint'

Automatically Upgrade Dependencies (#171)

@valeriupredoi

Hi, @valeriupredoi @michael-christen
Sorry for seeing these messages so late. Can you please provide a simple reproducible example?

I tried to reproduce an example, Is it the same as this? Please feel free to provide additional examples

https://github.com/dongfangtianyu/pytest_importlib_issue

Fix `ImportError` crash when using `--import-mode=importlib` (#13053)

@valeriupredoi

Hi, @valeriupredoi @michael-christen Sorry for seeing these messages so late. Can you please provide a simple reproducible example?

I tried to reproduce an example, Is it the same as this? Please feel free to provide additional examples

https://github.com/dongfangtianyu/pytest_importlib_issue

Yep, that's the exact case I have, thanks @dongfangtianyu 🙏

Fix `ImportError` crash when using `--import-mode=importlib` (#13053) (…

Navigation Menu

Repository navigation

`KeyError` fired from `importlib` during collection when running with `--import-mode=importlib` on a directory containing a same-named one #12592

A detailed description of the bug or problem you are having

Output of `pip list` from the virtual environment you are using

pytest and operating system versions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

KeyError fired from importlib during collection when running with --import-mode=importlib on a directory containing a same-named one #12592

Description

A detailed description of the bug or problem you are having

Output of pip list from the virtual environment you are using

pytest and operating system versions

Activity

Zac-HD commented on Jul 21, 2024

valeriupredoi commented on Dec 2, 2024

michael-christen commented on Dec 3, 2024

dongfangtianyu commented on Dec 11, 2024

michael-christen commented on Dec 12, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`KeyError` fired from `importlib` during collection when running with `--import-mode=importlib` on a directory containing a same-named one #12592

Output of `pip list` from the virtual environment you are using