Skip to content

[Sad Catapella B3][Performance] TIFF-to-Zarr conversion rereads the converted pixels to compute histograms #79

Closed as not planned

Description

@guijacquemet

Baseline: d7a538e56d41ac1093ba79adcad397c21b3dfd32. Classification: confirmed redundant I/O by design; benefit of a fused pass is a performance hypothesis.

Evidence: _CopyStaging.convert() lines 885–897 reads TIFF planes and writes Zarr. run_experiment() lines 600–610 then invokes _read_file() on the new Zarr copy for histogram calculation. Existing test asserts each TIFF is read once for conversion and the Zarr copy is read again for histograms: intended behavior. It should not be altered without preserving the invariant that the table describes the persisted converted pixels.

Optimization plan: investigate a fused, chunk-aware stream that feeds level-0 writer and histogram accumulator from the same canonical pixel blocks, with a post-write integrity check to ensure copied data and histograms match. Consider direct concurrent accumulation and conversion only if retry/rollback, channel-ordering, clipping, endianness and range-check multi-pass behavior can remain exactly equivalent. For 32/64-bit and float robust range, extra quantile passes are still necessary unless spooling/compact intermediate state is used. Benchmark source TIFF reads, Zarr reads/decode calls, disk traffic, wall time, peak RSS and end-to-end CSV equivalence. Do not add Numba to the I/O orchestrator.

Activity

  1. guijacquemet commented on Nov 2, 2026

    @guijacquemet
    ContributorAuthor

    Obsolete in v0.4.0: convert_tiffs, zarr_export and OME-Zarr copy reuse were removed in PP-2 (#66, commit 3c2802d; plan question 46). Named-axis copies of images with axis errors are now the OME-TIFF convert step (PP-11, #124, merged as 8db1f4a). Closing as not planned (owner-approved triage, 2026-11-02).

    After conversion the run reads the copies themselves and recomputes every row (design §5).


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions