Improve validation robustness and add nxlint / nxvalidate tools - #271
Merged
Merged
Conversation
The is_modified() method on NXroot previously compared only the filesystem mtime to detect external changes. This caused false positives whenever a process opened the file in write mode without writing anything (e.g., nxcheck -a, h5py.File(path, 'a').close()), since the HDF5 library updates the superblock consistency flags — and therefore st_mtime — on every write-mode open. HDF5 provides no reliable internal API for detecting actual data changes: h5py.h5o.get_info().mtime always returns 0, and ctime tracks object creation only (and only if track_times=True was set at file creation). The fix adds a three-stage check to is_modified(): 1. If st_mtime has not advanced, return False immediately (unchanged fast path). 2. If st_mtime has advanced but st_size is unchanged, read the first 64 KB of the file and compare an Adler-32 checksum against a baseline recorded at last open/close. If the hash matches, treat the mtime bump as a false positive and return False. 3. Only if st_mtime advanced *and* either the size or the hash differs is the file reported as modified. New additions to support the check: - NXFile.size property (st_size from the same stat() call as mtime) - NXFile.file_hash(nbytes=65536) — zlib.adler32 of leading file bytes - NXroot._file_size and NXroot._file_hash, recorded alongside _mtime at every open(), close(), reload(), and nxfile setter call - NXroot.serialize() persists _file_size; deserialize() restores it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Three null-safety issues caused crashes on valid but edge-case NXDL
structures:
- GroupValidator.__init__() did not initialise self.symbols before the
conditional block, so accessing it for a group whose nxclass is None
(i.e. a group with no NX_class attribute) raised AttributeError.
Fixed by setting self.symbols = {} unconditionally in __init__().
- GroupValidator.get_xml_dict() and ApplicationValidator.load_application()
accessed xml_dict['symbols']['symbol'] directly. A <symbols> block
that contains only a <doc> child (and no <symbol> elements) is valid
NXDL but produced a KeyError. Fixed by using .get('symbol', {}).
- GroupValidator.validate() accessed valid_groups[name]['@type'] without
checking for its presence. Groups defined in a base class without an
explicit type attribute caused a KeyError. Fixed by using .get('@type')
and guarding the class-mismatch check with 'if cls is not None'.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A link L1 that resolves to a second link L2 which itself is broken (its target no longer exists) was previously reported as valid because item.exists() returned True for L1. The validator now looks one level deeper for internal links: if the resolved object is another NXlink and that link does not exist, the validator logs a descriptive error naming both the intermediate link target and the dangling endpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
lint_nxdl() is a new function that validates the structural integrity of an NXDL file. It uses lxml to parse the file and walks the element tree to detect nested <field> elements, which are a common authoring mistake that confuses the validator. If the bundled nxdl.xsd schema is present, it also runs a full XSD validation and reports any additional errors with namespace prefixes stripped for readability. validate_application() is updated in three ways: - An upfront Path.exists() check gives a clear error message instead of letting nxopen() raise an obscure exception for missing files. - The entire open/validate block is now wrapped in a try/except NeXusError so file-level errors (corrupt HDF5, permission denied, etc.) are logged and return gracefully instead of propagating. - Before validating, lint_nxdl() is called on the application definition; if it finds issues a warning is logged advising the user to run 'nxlint <definition>' for details. Also fixes inspect_base_class(): when the NXDL file for a class does not exist, the fallback message incorrectly referenced validator.filepath (which is also None in that case) instead of validator.definitions. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
nxlint validates the structural integrity of one or more NXDL files and reports any errors found by lint_nxdl(). It accepts the same -d/ --definitions, -i/-w/-e verbosity, and -v/--version flags used by the other nxcheck-family scripts. nxvalidate is a dedicated CLI wrapper around validate_application(). It validates a NeXus file against an application definition, with the same interface as the -a flag of nxcheck but as a standalone command with cleaner help text. The application can be specified explicitly with -a, or read from the 'definition' field in the file's NXentry group. pyproject.toml changes: - Add lxml as a formal dependency (required by lint_nxdl). - Register nxlint and nxvalidate as installed console scripts. - Add 'nexusformat.definitions': ['*.xsd'] to package-data so that nxdl.xsd is included in the installed package and lint_nxdl can perform full XSD validation against it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
lxml represents XML comments and processing instructions with a callable tag (e.g. lxml.etree.Comment) rather than a string. The walk() helper inside lint_nxdl() applied '}}' in child.tag to every child node, which raises TypeError when child.tag is a callable. Fix by guarding both the element dispatch and the inner field-child loop with a callable(tag) check, mirroring the standard lxml idiom for skipping non-element nodes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
FileValidator.__init__ was already guarded with a try/except NeXusError, but validator.validate(path) was not. Files that open successfully at the Python level but fail when h5py tries to read them (e.g. HDF4 files, which h5py cannot open) raised an unhandled NeXusError from inside validate(). Wrapping the validate() call in the same pattern logs a clean error message and returns instead of propagating the exception. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes three crash-level bugs in
validate.pytriggered by valid butedge-case NXDL structures, adds detection of broken chained internal
links, introduces a new
lint_nxdl()function for structural NXDLvalidation, and ships two new CLI scripts:
nxlintandnxvalidate.Bug fixes in validate.py
AttributeError: 'GroupValidator' has no attribute 'symbols'—self.symbolswas only assigned inside theelsebranch ofGroupValidator.__init__(), so groups with noNX_classattribute(
nxclass=None) never set it. Fixed by initialisingself.symbols = {}unconditionally before the conditional block.
KeyError: 'symbol'—GroupValidator.get_xml_dict()andApplicationValidator.load_application()accessedxml_dict['symbols']['symbol']directly. A<symbols>block thatcontains only a
<doc>child is valid NXDL but has nosymbolkey.Fixed by using
.get('symbol', {})in both places.KeyError: '@type'—GroupValidator.validate()accessedvalid_groups[name]['@type']without checking for its presence. Groupsdefined in a base class without an explicit
typeattribute caused acrash. Fixed using
.get('@type')with aNoneguard.Wrong attribute in
inspect_base_class()— When the NXDL file for aclass does not exist, the fallback message referenced
validator.filepath(which is
Nonein that case) instead ofvalidator.definitions.New: broken chained link detection
Validator.check_link()now looks one level deeper for internal links. Ifa link L1 resolves to a second link L2 whose target no longer exists, a
descriptive error is logged naming both the intermediate target and the
dangling endpoint.
New:
lint_nxdl()and improvedvalidate_application()lint_nxdl()validates the structural integrity of an NXDL file usinglxml. It detects nested<field>elements (a common authoring error thatconfuses the validator) and, when
nxdl.xsdis present in the definitionsdirectory, runs a full XSD validation with namespace prefixes stripped from
error messages.
validate_application()gains an upfrontPath.exists()check (clearerror for missing files), a top-level
try/except NeXusErrorwrapper(graceful exit on corrupt HDF5 or permission errors), and a pre-validation
call to
lint_nxdl()that warns the user if the application definitionitself has structural issues.
New CLI scripts
nxlint— lints one or more NXDL files and reports structuralerrors. Accepts
-d/--definitions,-i/-w/-everbosity,-v/--version.nxvalidate— dedicated wrapper aroundvalidate_application();validates a NeXus file against an application definition read from the
file or supplied with
-a. Same interface asnxcheck -abut as astandalone command with cleaner help text.
pyproject.tomladdslxmlas a formal dependency, registers the two newconsole scripts, and adds
nxdl.xsdtopackage-dataso that installedpackages can run full XSD validation.
🤖 Generated with Claude Code