Repository navigation
importlib.reload not updated when only change a little bit code layout #120932
Description
Bug report
Bug description:
import os
import importlib
TEST_MODULE_NAME = "tmod"
def WriteCodeContext(text: str, filePath):
with open(filePath, "w") as f:
f.write(text)
f.flush()
scriptDir = os.path.abspath(os.path.dirname(__file__))
testFilePath = os.path.join(scriptDir, f"{TEST_MODULE_NAME}.py")
source = """
a = 1
"""
target = """
a = 2
"""
WriteCodeContext(source, testFilePath)
mod = importlib.import_module(f"{TEST_MODULE_NAME}")
old = mod.a
WriteCodeContext(target, testFilePath)
importlib.reload(mod)
new = mod.a
assert old != new # failed, 1 != 1When I added a small number of comments, the results changed
# ... ...
target = """
a = 2 # some comments
"""
# ... ...
assert old != new # pass, 1 != 2CPython versions tested on:
3.8
Operating systems tested on:
Windows
Activity
ZeroIntensity commentedon Jul 11, 2024
This is confirmed on Python 3.12 and the CPython main branch. Here's a smaller reproducer:
At first glance, this looks like a problem related to cache invalidation inside
importlib, which would explain why it doesn't occur when comments are added. I'm guessing it does this by the size of the file, since this problem only occurs when the original file size is equivalent to the new one (which is rare, but still a bug).I played around with this a little bit, and it seems likely that the problem is with
SourceLoader.get_code-- only the file is updated, not the compiledpycfile.get_codethen reads that old compiled file, and ignores the updatedpyfile.I'll submit a patch for this ASAP.
ZeroIntensity commentedon Jul 11, 2024
The problem is here. The repro updates the file too quickly for the number to be different, and when the size is the same, it doesn't get caught there either.
I'm not all too sure this is worth fixing, considering it will probably have negative performance benefits, and only happens in very extreme cases. It might be worth documenting, though. What do you think, @serhiy-storchaka (pinging you considering you helped on the previous
zipimportissue)?serhiy-storchaka commentedon Jul 11, 2024
This is not related to
zipimport, but it looks like the same issue as #121376 (there may be other open issues for the same problem).pythongh-120932: Chack the pyc file mtime at import
pythongh-120932: Check the pyc file mtime at import
pythongh-120932: Check the pyc file mtime at import
serhiy-storchaka commentedon Jul 11, 2024
Repeating myself from #121376 (comment):
This is possible, but this may be not so simple. Currently the code compares the timestamp of the
.pyfile (truncated to seconds) with the timestamp saved in the.pycfile. They are supposed to be equal. You need a third timestamp to handle the case when the.pyfile was changed within a second..pycfile. It needs to be strongly larger than the timestamp of the.pyfile (truncated to seconds) to ensure that the.pycfile is not outdated. But additionalstat()call adds an overhead. We are trying to minimize the number of system calls for import.stat()is significant, we can use the current time, and only if the difference between timestamps is small, use the timestamp of the.pycfile. It will be not easy to test.serhiy-storchaka commentedon Jul 11, 2024
#121620 implements this idea.
ZeroIntensity commentedon Jul 11, 2024
Right, I was just assuming you had knowledge of the import system :)