Skip to content

[Sad Catapella B3][Performance] Misaligned Zarr block reads may decode the same chunk repeatedly #78

Open

Description

@guijacquemet

Baseline: d7a538e56d41ac1093ba79adcad397c21b3dfd32. Classification: confirmed sub-chunk repeated-slice access pattern; decompression/caching cost is an unmeasured performance hypothesis.

Evidence: _choose_block() lines 1021–1058 permits partial rows; _ranges() makes multiple sub-chunk ranges; _zarr_layout() lines 1338–1385 always enables partial_rows=True; _zarr_plane() lines 1392–1404 executes a separate Zarr array[selection] for each range. On compressed Zarr stores, each sub-chunk selection may trigger another whole-chunk decode absent effective caching. The same blocks are revisited across histogram passes via the pinned normalization brick (compute_block_histogram). Test asserts pixel correctness of sub-chunk reads, not decode counts or throughput.

Impact: Very poor I/O/decompression efficiency when chunk Y is large relative to affordable block rows, especially remote/object stores and robust histogram passes. No measured slowdown is asserted.

Fix: instrument actual store get/decompress calls; compare chunk-aligned reads, cache-by-chunk with explicit memory bound, and rechunking/preprocessing options. Avoid assuming smaller NumPy selection means smaller decompression. Include 1/2/4-D chunk shapes spanning channels and planes; compression=none/lz4/zstd; local SSD and network object store where available; RSS, decompressed bytes, storage requests and wall time. No Numba role for codec/I/O itself.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions