[guijacquemet]

What and why

Two worker-side changes to how an image input is read, from the findings of #123 (docs/PERF_SHARED_MEMORY.md sections 2, 4.5 and 6).

1. Shared-memory order (correctness). The worker ignored a descriptor's order, so an F-order block (what an ImgLib2-style [X, Y] host naturally has) was read transposed with no error: wrong science. Now C/C_ORDER (or no key) and F/F_ORDER are honoured: the block is read with the strides of the declared order and the tool gets the same logical array, as a C-ordered copy. Any other value ("A", "K", "f", a number, a list, null) is refused with unreadable_image and a message saying send 'C' or 'F'. I chose honour-and-test because it is one order= argument; unknown values are refused either way. Documented in docs/PROTOCOL.md.

2. Memory-mapped TIFF input. An image input given as a TIFF file of at least 8 MiB that is clearly the simple case (one series, one level, uncompressed, untiled, every page's strips in one contiguous run, native byte order, kind generic/ImageJ/shaped, not cut short) is mapped with np.memmap(mode="c") instead of read and copied.

Behaviour changes

• Tool-visible semantics are unchanged: same shape, dtype and values as tifffile.imread, a plain writable numpy.ndarray. Decision: copy-on-write, not a read-only view. A read-only view would make a tool that today does image[mask] = 0 on its input raise; copy-on-write keeps it working, and the first write copies the page privately so the file (the host's export or a person's own) never changes.
• Fallback to the existing tifffile.imread path for compressed, tiled, predictor, multi-series, pyramid, non-contiguous, OME, non-native byte order, small (< 8 MiB), cut-short files, and when LC_TIFF_MEMMAP=0 is set in the worker's environment.
• Found while testing: tifffile's series.dataoffset trusts the layout of a series from its first pages, so a stack whose later pages lie elsewhere would have been mapped to the wrong pixels. Every page's offsets are now checked (_pages_are_one_block; about 65 ms for a 5000-page stack), with a test on a patched file.
• Lifetime: the worker calls convert.release_mapped_inputs() before every terminal message, so a failed tool whose traceback or a reference cycle holds the input does not keep the file locked (Windows) after FAILURE. Documented limits in PROTOCOL.md: a tool that keeps its input in a global, and a host that truncates the file while a task runs (a mapped page past the new end ends the worker).
• Timing at 512 MiB (uint16, warm cache, Linux sandbox, median of 5), benchmarks/tiff_memmap.py from Research: shared memory for big images (issue #66), benchmarks and findings #123 and the real worker (load_inputs_s of COMPLETION):

	before (imread)	after (mapped)
#123 script, load	0.202 s	0.001 s
#123 script, load + one full pass	0.243 s	0.040 s
#123 script, cold cache, load + pass	0.298 s	0.218 s (the read moves into the first pass)
real worker, load_inputs_s	0.191 s	0.001 s
real worker, task wall time (load + max())	0.232 s	0.050 s

Tests

• tests/test_input_loading.py (+ tests/input_app/): F order read with a non-symmetric array in 2D and 3D, both spellings, C/absent, 10 refused values, through a real worker; mapped cases (u8/u16/i16/u32/f32/f64, stack, RGB, ImageJ hyperstack, BigTIFF, many strips) equal to imread in shape/dtype/values and plain ndarray type; every fallback (zlib, tiled, predictor, a stack with a gap, two series, OME, opposite byte order, pyramid, small, cut-short, env switch) with the same result as imread; a tool writing to its input (in process and through a worker): file hash and mtime unchanged, file deletable afterwards; after both a completed and a failing task the worker has no mapping of the file (/proc/self/maps) and the file can be removed.
• Sabotage (tests/test_sabotage.py, 6 new breaks, all caught): order ignored, unknown order read as C, mapping opened r+, the page-gap check removed, mapping not released, mapping never used. Run locally with LC_SABOTAGE_ONLY.
• Local: full suite 635 tests, 1 failure that also fails on main here (see below); black, ruff, mypy, vulture as in the lint job.

Not verified

• Windows and macOS behaviour of the mapping (file locking after the task, copy-on-write, os.remove after FAILURE) is only exercised by the CI cells; nothing here ran on those systems. See CI below.
• Cold-cache numbers are from drop_caches in a sandbox VM, not a spinning or network disk; no timing on real host machines or with Defender.
• Hosts: nothing was changed or run in the Napari, Fiji or QuPath hosts. Fiji writes TIFFs with ImageJ that are covered by the ImageJ case only as written by tifffile (imagej=True).
• test_tools_features.py::test_doctor_prints_them_and_does_not_call_an_upgrade_stale: not a failure on main in CI (green, full matrix, run 38356428622) nor in a clean virtualenv here. It fails when run with this sandbox's system Python: that interpreter has an older editable install of Tools (/tmp/wt_versions, same version string, without retention_prune) which the doctor subprocess imports instead of the checkout, so the printed feature list differs. An environment artefact, not changed here. (test_hardening...shared_entry_is_skipped fails on main too when run as root in this sandbox; it is skipped on CI runners.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y2SN8SuEakTdiWy3F2WUqN

---

Generated by Claude Code