Skip to content

Conversion reopens TIFF headers for pixel reads and memory checks (performance follow-up of #96) #141

Open

Description

@guijacquemet

Context

#96 is closed. The consistency problem it raised is fixed: the conversion no longer mixes metadata from different points in time, and it checks each source's fingerprint before and after it writes the copy (conversion_write._write_partial). The conversion still opens each TIFF header again more than once, and nobody has measured what that costs. The external review of v0.4.0rc4 found this. This issue tracks it as a performance item only.

What still reopens a header (main e24f334)

  • conversion_plan._cannot_convert → planes.check_planes(root, mappings, memory_limit=...) opens every TIFF to be converted, to check that its planes fit the memory limit. Preflight has already inspected those headers.
  • ome_tiff_copy (iter_planes(source, mapping, memory_limit=...), line 126) opens the source again to read the pixels. ome_tiff_copy line 134 opens a tifffile.TiffFile again for the copy's layout.

So one converted file costs several header opens, IFD reads and OME-XML parses on top of the preflight inspection.

What to do

  1. Benchmark first. Count TiffFile opens and measure elapsed time and peak memory for 100, 1k and 10k TIFFs: small ones, files with many pages, files with heavy OME-XML, on a local disk and on a network or Drive mount. Compare the output byte for byte: copies, NGFF axis scales and units.
  2. Only if the numbers justify it, reuse one validated header/IFD descriptor across the memory check, the layout and the pixel read. That descriptor must stay bound to the fingerprint taken with the header ([Sad Catapella B5][Scientific integrity] Source files may change between fingerprint capture and histogram publication #90). Metadata epochs must never mix, and nothing is cached across a change to the source.
  3. Never claim a speed-up without a benchmark whose outputs are identical.

Numba gives no benefit here. The work is metadata parsing and I/O.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions