Skip to content

[Sad Catapella B3][Memory] Blockwise pyramid region selector does not enforce a viable minimum budget #76

Closed as not planned

Description

@guijacquemet

Baseline: d7a538e56d41ac1093ba79adcad397c21b3dfd32. Classification: confirmed failure to enforce supplied memory bound at region selection; actual peak RSS not measured.

Evidence: OmeZarrWriter._region_side() lines 393–403 reduces side until it reaches 2, but does not check whether the estimated cost at 2 still exceeds memory_limit. _write_blocks() then reads, float64-converts and writes these regions. The estimate adds five full chunks; even if the region is 2×2, a Zarr partial write can involve full chunk processing. Existing test constructs a writer with memory_limit=1 and expects _region_side()==2, without exercising the read/write under that bound. Current ConversionSettings permits chunk sides from 16 to 8192.

Impact: The advertised memory limit can be violated during TIFF→OME-Zarr conversion, even when check_planes() approves source plane reading. Some configurations may exhaust RAM without a clear PlaneMemoryError.

Fix/acceptance: compute the minimum feasible region/chunk memory footprint before writing (including input chunk decode, output chunk RMW, float64 intermediates and compression buffers), and fail fast with a named, actionable error when the requested budget is impossible. Either implement a genuinely bounded alternative or demand larger memory/smaller chunks; never silently exceed the limit. Test positive/negative boundaries, large chunks with tiny budgets, full and blockwise source reading, and peak RSS. No Numba fix: this is allocation/I/O planning.

Activity

  1. guijacquemet commented on Nov 2, 2026

    @guijacquemet
    ContributorAuthor

    Obsolete in v0.4.0: convert_tiffs, zarr_export and OME-Zarr copy reuse were removed in PP-2 (#66, commit 3c2802d; plan question 46). Named-axis copies of images with axis errors are now the OME-TIFF convert step (PP-11, #124, merged as 8db1f4a). Closing as not planned (owner-approved triage, 2026-11-02).

    PP-11 checks every plane against the memory limit before anything is written and reads larger planes in blocks; an impossible budget fails fast.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions