Repository navigation
Preserve dimensions for empty orthogonal indexing - #1499
Conversation
Retain selected dimensions and scalar squeeze markers without allocating Cartesian chunk grids for empty results. Preserve broadcastable empty writes and automatic decoding, using the selected multifile components and existing slice/fancy-index dtype rules.
|
The current head |
|
Thanks for detecting and fixing this. I'm a bit concerned about the amount and opacity of code that the agent created to solve the problem - however, the indexing code is already pretty opaque so perhaps I shouldn't be surprised. |
|
Please create a Changelog entry (after merging updates from master). |
Sync upstream 7b4d837 as requested for Unidata#1499. Preserve the existing contribution and history; no unrelated feature changes.
Document preserved dimensions and decoded types after merging current upstream master. This commit changes only the requested release note.
|
Thanks for reviewing and merging this, @jswhit. I appreciate your guidance on the implementation. |
Empty integer indices and all-false masks can collapse surviving dimensions: an empty selection on a
(24, 100)variable returns(0, 1)instead of(0, 100). This reproduces in netCDF4 1.7.4 and explains the backend behavior reported in pydata/xarray#11601.Preserve selected lengths and scalar squeeze markers when an orthogonal selection is empty, without constructing a Cartesian product of fancy indices. Skip native reads for empty output, preserve automatic decoding and character-conversion eligibility, and keep broadcastable empty writes as no-ops. Multi-file empty reads derive their decoded dtype from the selected files.
The final diff contains the two indexing implementation files,
test/test_empty_indexing.py, and the requestedChangelogentry. Upstream master is integrated, including its benchmark typing, numeric-attribute byte-order and command-line corrections. The indexing correction and regression edits are unchanged from the previously validated revision; complete source files also contain those upstream changes.Regressions cover scalar/slice/array/mask combinations, orthogonal indexing, strings and variable-length data, unlimited dimensions, empty writes, bounded metadata, and multi-file packing/unsigned conversion.
Validation at final head
9a6bc82405d3ef53e704e6b45e71796e2d242f61:Earlier defect validation used a rebuilt wheel installed outside the checkout on macOS ARM64, Python 3.12.3, NumPy 2.5.3, netCDF-C 4.9.3 and HDF5 1.14.6:
For the indexing review,
_StartCountStrideretains one compact chunk when any selected axis is empty, so surviving lengths and scalar squeeze markers remain available. Nonempty selections still expand to the existing orthogonal chunk grid. The emptyMFDatasetpath retains selected-file decoding: slices promote the contributing files' decoded dtypes, non-slice record chunks retain their final record's dtype, and a selection with no records follows the first file's convention. Tests exercise these behaviors with packed/unsigned data and character conversion as well as ordinary numeric variables.