Uses a higher granularity than seconds for .pyc file invalidation

Bug report

Bug description:

reproduction:

#!/bin/sh 
 
mkdir -p src 
rm -fr src/__pycache__ 
rm -fr __pycache__ 
 
echo 'from src.test import f' > main.py 
echo 'print(f())' >> main.py 

echo 'def f():' > src/test.py 
echo '    return "Hello!"' >> src/test.py 
# prints "Hello!"
python3 main.py 

echo 'def f():' > src/test.py 
echo '    return "Goodbye!"' >> src/test.py 
# also prints "Hello!", but note the file does not say that anymore
python3 main.py

(this is probably sensitive to how fast your computer is)

this seems to stem from (i guess) that the timestamps are 32-bit, and write the current second (unix time, as 32-bit). and then, when the file is edited and the python script runs again, it gets a cache hit (same timestamp as file 32-bit mtime) and yields the old content.

is this known/intended? is there a mitigation that isn't just "delete the pycache"? i tried looking if anyone else ran into this and could not find anything.

i ran into this as part of a workflow that first edited a python script via shell (edits mtime), then invoked python on it (updates pyc to that mtime), then that python invocation updates the same script a second time and those changes were weirdly lost and confused me for a while. after having to delete the pycache a few times, i looked into it and ended up at (seemingly) this issue.

also (not relevant), having not looked at the code, what happens in 2038 when 64-bit file timestamps go over the 32-bit value? will the pyc just wrap around or is there some other planned handling?

maybe this format should be updated to 64-bit milliseconds (this seems.. difficult, maybe a new magic?) (and technically milliseconds can also race, though python is usually not fast enough to hit that), or perhaps the pycache should be invalidated when the mtime matches (but this would loop a bunch of times when the script is ran multiple times on the same second until current time is 1 second later than the filetime)

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

gh-121376: Switch from stat.st_mtime to stat.mtime_ns to achieve more accuracy #121502

@Eclips4

@Eclips4 This bug has been confirmed, It is rare but actuially exist. Would you mind help me to checkout which label is suitable for this issue?

@Eclips4

@Eclips4 This bug has been confirmed, It is rare but actuially exist. Would you mind help me to checkout which label is suitable for this issue?

I guess the already attached type-bug label is the right label for this issue.

@Eclips4

@Eclips4 This bug has been confirmed, It is rare but actuially exist. Would you mind help me to checkout which label is suitable for this issue?

I guess the already attached type-bug label is the right label for this issue.

I think we need a discussion here. But I don't know who should be mentioned in this case

@Fidget-Spinner

Hi @Fidget-Spinner @brandtbucher ... Sorry for bother you. You guys are the only two core developers I might know. And I don't know who is in charge of this part now.

I think this issue is worth to dive more deeper. Should we change the cache timestamp to more accurate number to avoid this circumtance.

@Zheaoli

Sorry @Zheaoli for the late response. I actually do not know who to ping on this issue. I think this should definitely be fixed. I can reproduce it on my machine, and it is clearly a bug.

@Zheaoli

Sorry @Zheaoli for the late response.

Don't mind bro(lol

I think this should definitely be fixed. I can reproduce it on my machine, and it is clearly a bug.

Same thought, I think we need to make the identify of the cache file more accurate

@Zheaoli

@Zheaoli maybe the import machinery people would be interested? https://devguide.python.org/core-developers/experts/index.html

@brettcannon

If @brettcannon has time, I would like to hear his opinion about this.

This issue cannot be fixed without changing the format of the .pyc files. This is a large issue which perhaps requires a PEP. And if we change the format for timestamp, we can also also make other changes: include larger stable signature, Python version, etc. This is a large issue.

it can be worked around by considering the .py file over bytecode when the timestamp second value is the same, at the cost of sometimes not using bytecode when it could have been

@Fidget-Spinner

https://discuss.python.org/t/change-pyc-file-format-to-record-more-accurate-timestamp-and-other-information/57815

I propose an idea in the discuss.python.org. We can discuss here. If we actually need a PEP finally, I would like to write it.

cc @Fidget-Spinner @serhiy-storchaka @q66 @Eclips4

As Serhiy said, this is a complex issue. I've opened #121502 as a PoC to confirm that this approach can theoretically be used, but in reality it doesn't solve the problem, it just makes it harder to reproduce.

I have changed this to a feature request.

is this known/intended?

Yes, for over a decade.

is there a mitigation that isn't just "delete the pycache"?

If you needed more granular .pyc invalidation there is hash-based validation. You can also just turn off use of .pyc files.

what happens in 2038 when 64-bit file timestamps go over the 32-bit value? will the pyc just wrap around or is there some other planned handling?

The timestamp is an equality check of two ints, so the wrapping isn't really a concern.

If you needed more granular .pyc invalidation there is hash-based validation.

hm. i see that compileall has options for this, but i cant seem to figure out how to set this for the __pycache__ that is automatically generated when the interpreter runs.

currently i basically do

if sys.pycache_prefix is None:
    cbp = os.path.dirname(os.path.realpath(__file__))
    sys.pycache_prefix = f"{cbp}/__pycache__"

and this has the effect of making a single __pycache__ dir in the repo root, as the interpreter caches the files. is there some setting that then also sets the interp to use hash-based pycache instead of the time based one, accessible from either inside python itself or some cli option? using compileall first as a separate step would be clunky in this case

i cant seem to figure out how to set this for the __pycache__ that is automatically generated when the interpreter runs.

https://docs.python.org/3/using/cmdline.html#cmdoption-check-hash-based-pycs

ah, i thought that only affects checking hash based that happen to exist pycs and doesn't change what is generated. sure :)

ah, i thought that only affects checking hash based that happen to exist pycs and doesn't change what is generated. sure :)

I would double-check I'm actually right. 😅

it can be worked around by considering the .py file over bytecode when the timestamp second value is the same, at the cost of sometimes not using bytecode when it could have been

This is possible, but this may be not so simple. Currently the code compares the timestamp of the .py file (truncated to seconds) with the timestamp saved in the .pyc file. They are supposed to be equal. You need a third timestamp to handle the case when the .py file was changed within a second.

Use the timestamp of the .pyc file. It needs to be strongly larger than the timestamp of the .py file (truncated to seconds) to ensure that the .pyc file is not outdated. But additional stat() call adds an overhead. We are trying to minimize the number of system calls for import.
Use the current time. The problem is that on the FAT filesystem the file timestamp has granularity of 2 seconds. So the timestamp of just written file will often be different from the current time truncated to seconds.. And you cannot easily say what is the granularity of the file timestamp. There is also lesser problem of double rounding on filesystems with more precise file timestamps.
Hybrid approach. If the cost of stat() is significant, we can use the current time, and only if the difference between timestamps is small, use the timestamp of the .pyc file. It will be not easy to test.

#121620 implements this idea.

Navigation Menu

Repository navigation

Uses a higher granularity than seconds for `.pyc` file invalidation #121376

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Uses a higher granularity than seconds for .pyc file invalidation #121376

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

Zheaoli commented on Jul 6, 2024

Eclips4 commented on Jul 6, 2024

Zheaoli commented on Jul 6, 2024

Zheaoli commented on Jul 8, 2024

Eclips4 commented on Jul 8, 2024

Zheaoli commented on Jul 8, 2024

Fidget-Spinner commented on Jul 8, 2024

Eclips4 commented on Jul 8, 2024

serhiy-storchaka commented on Jul 10, 2024

q66 commented on Jul 10, 2024

Zheaoli commented on Jul 10, 2024

Eclips4 commented on Jul 10, 2024

brettcannon commented on Jul 10, 2024

nekopsykose commented on Jul 10, 2024

brettcannon commented on Jul 10, 2024

nekopsykose commented on Jul 10, 2024

brettcannon commented on Jul 10, 2024

serhiy-storchaka commented on Jul 11, 2024

serhiy-storchaka commented on Jul 11, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Uses a higher granularity than seconds for `.pyc` file invalidation #121376