From 29ac14d4dfd84859a076d6bedde561757b2c2cb2 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 8 Oct 2026 11:53:57 +0000 Subject: [PATCH] Spec Wave 2: time-lapse and 3D axes, FigureOut, Open() (proposal, docs only) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Y2SN8SuEakTdiWy3F2WUqN --- docs/SPEC_WAVE2.md | 336 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 336 insertions(+) create mode 100644 docs/SPEC_WAVE2.md diff --git a/docs/SPEC_WAVE2.md b/docs/SPEC_WAVE2.md new file mode 100644 index 0000000..951db3c --- /dev/null +++ b/docs/SPEC_WAVE2.md @@ -0,0 +1,336 @@ +# Spec (proposal): Wave 2 of the manifest roadmap + +Status: **for review, nothing is implemented.** Same rules as Wave 1 ([SPEC_WAVE1.md](SPEC_WAVE1.md)): the manifest stays small and organised in four families ([MANIFEST.md](MANIFEST.md)); everything is additive within `protocol: 1`; a host that does not know a hint shows a plain, working form. Out of scope: `TracksOut` (parked) and GEFF (maybe later; nothing here blocks it). + +| # | Item | Family | New names | Hosts | +|---|---|---|---|---| +| 1 | Time-lapse and 3D images | Types + Links + Behaviour | `Axes("TZYX")` (extended), `TimeIntervalOf`, `ZStepOf`, `PerPlane`, `t` / `z` columns and properties | Napari, Fiji, QuPath (+ CLI, notebook) | +| 2 | Figures and reports | Types + Look | `FigureOut`, `Format` | all, plus CLI and notebook | +| 3 | Host action "open" | Behaviour | `Open()`, `LinkOut` | all, plus CLI and notebook | + +Order of work: 1 (largest, unlocks apps), then 2, then 3 (small, builds on 2 for HTML/PDF). Each item ships as a Tools PR (schema, validation, docs, example, `check` line, tests), then one PR per host, then the apps. A feature is "done" only when it meets the checklist at the end of MANIFEST.md. + +--- + +## 1. Time-lapse and 3D images + +### 1.1 Axis conventions +* Axis letters: `T` (time), `Z` (depth), `C` (channel), `Y`, `X`. Nothing else in Wave 2 (`S` samples/RGB stays "channel last, host-handled as today"; `Q` and unknown OME axes are refused, see 1.9). +* **The declaration decides the order the tool receives.** `Axes("TZYX")` means the function gets a 4D array whose dimension 0 is T, 1 is Z, 2 is Y, 3 is X. The worker transposes; the tool never sees the file's order. `Y` and `X` are always last, `C`, if declared, comes before `Y` (hence `CYX`, `TCYX`, `CZYX`, `TCZYX`); this keeps arrays C-contiguous in the common case and gives one canonical order to document. +* Wire order inside files is whatever the host writes; the **axes come from the file** (1.3), never guessed from the shape when metadata exists. +* Existing behaviour is unchanged: no `Axes` = host passes the image as it is; `Axes("YX")` = exactly 2D; `DEFAULT_AXES` for outputs without a declaration stays `{2: YX, 3: ZYX, 4: CZYX}` (a tool that returns a time series must declare `Axes("TYX")` on the output). + +### 1.2 How a tool declares what it needs +| Declaration | Meaning | The tool receives | +|---|---|---| +| `Axes("YX")` | 2D only (today) | 2D array | +| `Axes("ZYX")`, `Axes("TYX")`, `Axes("TZYX")`, `Axes("CYX")` ... | exactly these axes, in this order | array with those dimensions, in that order | +| `Axes("TYX", "YX")` (several strings) | accepts any of them; the worker picks the first that fits and the tool reads `ndim` | array of the matching shape | +| `Axes("TYX"), PickChannel()` | as `TYX`, for one channel chosen by the person | the stack of that channel (no `C`) | +| `Axes("YX"), PerPlane("TZ")` | the tool is written for one plane; the worker calls it once per plane (see 1.6) | 2D array, once per plane | + +Rules (declaration errors at registration, exact text): +* `parameter 'image': Axes('TYZX') is not valid: Y and X must be last (use 'TZYX')` +* `parameter 'image': Axes('TTYX') repeats the axis T` +* `parameter 'image': Axes('TYX') uses letters from T Z C Y X only; got 'Q'` (message lists the valid letters) +* `PickChannel` today requires `YX`; relaxed to "any axes without `C`" (error: `PickChannel gives the tool one channel, so Axes must not contain C`). +* An axis is only *needed* when it is in the string. Length-1 axes in the file that are not declared are squeezed away; an undeclared axis longer than 1 is refused (`wrong_dimensions`, 1.9). + +### 1.3 What the tool receives, and how the file says what the axes are +* **Still a TIFF path on the wire** (`image` / `labels` inputs). Hosts write **OME-TIFF or ImageJ-hyperstack TIFF with axes metadata** (tifffile `metadata={"axes": "TZCYX"}`); the worker reads `series[0].axes` with tifffile and reorders to the declared axes. +* Optional reserved input `_axes` (like `_job_dir`): `{"": "TZCYX"}`, the axes of the file the host just wrote when it cannot write metadata; it overrides the file. `_axes` with a wrong length is `bad_request`. +* A TIFF without axes metadata: 2D = `YX`; 3D = `ZYX` (as `DEFAULT_AXES` today); 4D = `CZYX`. Hosts that know better must send `_axes`. A tool can never be given a guessed `T`. +* Shared-memory ndarray objects (Appose) carry `axes` next to `shape` and `dtype`; without it the same defaults apply. +* Calibration: see 1.4. It is *not* embedded for the tool; the tool gets it as ordinary float parameters. +* dtype rules are those of Wave 1 (`portable_dtype`); nothing changes. + +### 1.4 `TimeIntervalOf` and `ZStepOf` (Links) +```python +@tool("Track speed") +def speed( + movie: Annotated[Image, Axes("TYX")], + pixel_size: Annotated[float, PixelSizeOf("movie"), Unit("um/px")] = 1.0, + dt: Annotated[float, TimeIntervalOf("movie"), Unit("s")] = 1.0, + dz: Annotated[Optional[float], ZStepOf("volume"), Unit("um")] = None, +): ... +``` +| | `PixelSizeOf` (today) | `TimeIntervalOf` | `ZStepOf` | +|---|---|---|---| +| Schema key | `pixel_size_of` | `time_interval_of` | `z_step_of` | +| Parameter type | `float` | `float` | `float` | +| Unit | um/px | **seconds** | **micrometres** | +| Prefill from | layer scale / calibration | Fiji `frameInterval`, OME `TimeIncrement`, QuPath server `getTimeUnit`, Napari layer `scale[T]` | Fiji `pixelDepth`, OME `PhysicalSizeZ`, QuPath `getZSpacingMicrons`, Napari `scale[Z]` | +* Declaration rules: applies to a `float` (or `Optional[float]`) parameter and names an `Image` / `Labels` parameter of the same tool whose `Axes` contain `T` (resp. `Z`) or that has no `Axes`. Error: `parameter 'dt': TimeIntervalOf('movie') needs 'movie' to have a T axis (its Axes are 'YX')`. +* Hosts convert to the fixed units above (Fiji `frameInterval` in `timeUnit` -> s; QuPath minutes/ms -> s). Unknown or zero calibration: the parameter keeps its default and the host shows "not calibrated" in the tooltip; it never sends 0. +* `Unit("s")` / `Unit("um")` stay free text for display; the units in the table are the contract. + +### 1.5 What a tool returns for stacks +| Output | Rule | +|---|---| +| `ImageOut` / `LabelsOut` | declare `Axes("TYX")` etc. on the output; the result `axes` carries it; hosts show a stack (Napari layer with dims sliders, Fiji hyperstack, QuPath preview + "Open in QuPath" for the plane set) | +| `PointsOut` | columns `y`, `x` stay first; **optional columns `z` and `t`** (integer, zero-based plane indices of the image named by `ApplyTo`, else the first image input) | +| `ShapesOut` | each feature has optional properties `t` and `z` (zero-based integers); coordinates stay `[x, y]` per plane | +| `TableOut` | an ordinary table; a `t` / `z` column by convention (hosts do not interpret it) | +| `Scalars`, `MessageOut`, `Affine` | unchanged (a per-frame Affine is refused with `PerPlane`, 1.6) | +* A point or shape without `t` / `z` when its `apply_to` image has that axis with length > 1 is placed on plane 0 and the host says so once (status line). `labconstrictor-tools test` reports it as a contract problem: `points 'peaks': image has a T axis but no 't' column (placed on frame 0)`. +* A `t` / `z` outside the image is dropped by the host and counted: `3 points were outside the image planes and were not shown` (never an error). +* Pixel centres at integers, as before; `t` and `z` are plane indices, never physical times. + +### 1.6 Per-plane or whole-stack execution: `PerPlane` (Behaviour) +* **Whole stack** (tool declares `TYX` / `ZYX` / ...): one task, the tool loops and calls `progress(fraction, "frame 12 of 300")` and `check_cancel()` itself, as today. +* **Per plane**: `Annotated[Image, Axes("YX"), PerPlane("TZ")]` on an image or labels input. The tool stays 2D; given a larger image the worker iterates over the listed axes that are present, in C order, calls the tool once per plane and merges the results: + * `ImageOut` / `LabelsOut`: stacked back along the iterated axes (shape check: every plane must return the same shape and dtype or `per_plane_mismatch`); + * `PointsOut` / `ShapesOut`: the worker adds `t` / `z` (the plane indices) to each row / feature; a tool that already returns the column keeps its own; + * `TableOut`: rows concatenated with `t` / `z` columns added when the tool did not return them; + * `Scalars`, `MessageOut`, `Affine`, `FileOut`, `FigureOut`: refused **at registration** (`PerPlane cannot be used with the output 'x' (a Scalars result); return a TableOut with one row per plane instead`). + * Several iterated inputs must have the same plane grid; other inputs are passed whole to every call. +* **Progress and cancel come for free:** the worker reports `UPDATE {message: "frame 12 of 300", current: 12, maximum: 300}` per plane (scaled to `PROGRESS_MAXIMUM`) and checks cancel between planes (cancel latency is one plane; a tool that never returns is killed like today). A failing plane ends the task: `per_plane_failed` with the plane named (below); partial results are discarded. +* **Memory:** the worker reads one plane at a time (tifffile page/memmap), so the stack never has to fit in RAM; this is why `PerPlane` is the recommended way for 2D tools. Outputs are written to disk plane by plane (tifffile `TiffWriter`). +* Without `PerPlane`, a 2D tool given a stack is refused (`wrong_dimensions`), exactly as today. Hosts offer a "Run on: this plane / all planes" chooser only for inputs with `PerPlane`; "this plane" sends just the viewer's plane (Napari current dims point, Fiji current slice/frame, QuPath current z/t). + +### 1.7 Host behaviour +| | Napari | Fiji | QuPath | CLI | Notebook | +|---|---|---|---|---|---| +| Show a stack input | any layer with ndim > 2; axes from layer metadata (OME reader) else a small "Axes of this layer" chooser, default from the tool's declaration | hyperstack (C, Z, T from the ImagePlus dimensions); no chooser | the open image's z-planes and timepoints | file path; axes from the file | file path or ndarray + `axes=` | +| Export | write OME-TIFF of the layer data (selected channel only with `PickChannel`) | export hyperstack (or one plane / one channel) as ImageJ TIFF with axes | export plane set as OME-TIFF; **Image area** gets a new line **Planes: current plane / all z / all t / all** (default current plane) | n/a | n/a | +| `TimeIntervalOf`, `ZStepOf` prefill | layer scale | calibration | server metadata | from file (OME / ImageJ tags) when parameter not given | same | +| Stack output | layer, dims sliders move through t/z | hyperstack window | plane-aware preview; annotations placed on their plane | file | `ndarray` or file path + preview of the middle plane | +| Points/shapes with `t`, `z` | N-D points/shapes layer (coordinates `[t, z, y, x]`) | point/polygon ROIs with `setPosition(c, z, t)` | annotations with `ImagePlane.getPlane(z, t)` | rows in the table | table | +| "Run on" chooser (`PerPlane`) | yes | yes | yes (with the Image area) | `all` unless `plane=t:3,z:2` given | `plane=` argument | +| Progress / cancel per plane | yes | yes (Esc) | yes | Ctrl-C | interrupt | +| Copy as command | image has no name: placeholder `movie=movie.tif` + comment; `plane=` is not added (the file is sent whole) | same, plus a note when only the current plane was sent | note line like the existing Image area one: `# movie: QuPath sent planes 'current plane' (the command line sends the whole file)` | n/a | n/a | + +### 1.8 Size and memory limits +| Constant | Value | Where | Meaning / how to change | +|---|---|---|---| +| `MAX_STACK_BYTES` | 4 GiB | worker (`convert`) | a whole-stack input larger than this (after `PickChannel` and axis selection) is refused; env `LC_MAX_STACK_BYTES`; not applied to `PerPlane` inputs | +| `MAX_PLANE_PIXELS` | 100 000 000 | QuPath guard, existing | per plane, unchanged (`lc.qupath.max_export_pixels`) | +| `lc.qupath.max_export_planes` | 500 | QuPath | planes of one export when "all" is chosen; above it the run is refused with the two ways out | +| `lc.qupath.max_export_bytes` | 4 GiB | QuPath | planes x plane bytes; same property style as the pixel budget; invalid value ignored and logged once | +| `MAX_POINTS_HINT` / `MAX_LABELS_HINT` | 1 000 000 points / 50 000 outlines **per run** | hosts | the Wave 1 outline limit now counts all planes together; hosts say how many were left out | +| Napari / Fiji | no limit of their own | | a stack that is already open is not copied twice: Napari writes in chunks (dask-aware), Fiji streams planes | +Nothing is ever downsampled or cropped silently. The QuPath refusal message pattern stays: `'' would send 1200 planes (limit 500). Choose Planes: current plane, or a smaller selection, or use a file.` + +### 1.9 Failure modes and error codes +| Code | When | Message (exact) | +|---|---|---| +| `wrong_dimensions` (existing, extended) | dims do not fit and no `PerPlane` | `'