Repository navigation
Reduce false-positive modification warnings using file size and hash - #270
Merged
Merged
Conversation
The is_modified() method on NXroot previously compared only the filesystem mtime to detect external changes. This caused false positives whenever a process opened the file in write mode without writing anything (e.g., nxcheck -a, h5py.File(path, 'a').close()), since the HDF5 library updates the superblock consistency flags — and therefore st_mtime — on every write-mode open. HDF5 provides no reliable internal API for detecting actual data changes: h5py.h5o.get_info().mtime always returns 0, and ctime tracks object creation only (and only if track_times=True was set at file creation). The fix adds a three-stage check to is_modified(): 1. If st_mtime has not advanced, return False immediately (unchanged fast path). 2. If st_mtime has advanced but st_size is unchanged, read the first 64 KB of the file and compare an Adler-32 checksum against a baseline recorded at last open/close. If the hash matches, treat the mtime bump as a false positive and return False. 3. Only if st_mtime advanced *and* either the size or the hash differs is the file reported as modified. New additions to support the check: - NXFile.size property (st_size from the same stat() call as mtime) - NXFile.file_hash(nbytes=65536) — zlib.adler32 of leading file bytes - NXroot._file_size and NXroot._file_hash, recorded alongside _mtime at every open(), close(), reload(), and nxfile setter call - NXroot.serialize() persists _file_size; deserialize() restores it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
rayosborn
force-pushed
the
fix-is-modified
branch
from
September 29, 2026 20:10
d8d7897 to
5e0140a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This improves the accuracy of the
NXroot.is_modified()function, whichpreviously compared only the filesystem
st_mtimeto detect externalchanges. The HDF5 library updates
st_mtimewhenever a file is openedin write mode — even if nothing is written — and provides no internal API
for detecting actual data changes.
is_modified()now uses a three-stage check:st_mtimehas not advanced, returnFalseimmediately (unchangedfast path, no overhead).
st_mtimehas advanced butst_sizeis unchanged, read the first64 KB of the file and compare an Adler-32 checksum against a baseline
recorded at last open/close. If the hash matches, treat the mtime bump
as a false positive and return
False.st_mtimeadvanced and either the size or the hash differsis the file reported as modified.
New additions to support the check:
NXFile.sizeproperty —st_sizefrom the samestat()call alreadymade by
mtimeNXFile.file_hash(nbytes=65536)—zlib.adler32over the leading bytesof the file; reads 64 KB, well under 1 ms on local storage
NXroot._file_sizeandNXroot._file_hash, recorded alongside_mtimeat every
open(),close(),reload(), andnxfilesetter callNXroot.serialize()persists_file_size;deserialize()restores itvia
.get()so existing serialised sessions are handled gracefully