[guijacquemet]

Summary

Bricks that measure objects (MorphoObjects, MorphoRelate, CellPatcher, MorphoFeature workflows) need each image paired with its masks, and need to know how the mask types relate: two nuclei in one cell, nucleoli inside nuclei inside cells, mitochondria labelled with their cell's number. Today each brick asks the user again. This PR lets the preprocessing brick find the masks, check them and suggest the relationships, for any number of mask types (MorphoFeature decision D78). The user then only has to name the masks and confirm.

By module:

• masks.py (new). find_masks pairs every image with its masks:
  • Layout. A masks folder outside the experiment, one subfolder per mask type, each mirroring the experiment's folders. A mask is paired with its image by leaf folder and position_id, ignoring case and extension. OME-Zarr label images also count as masks.
  • Header-only checks, no pixels read: the mask is 2D (length-1 axes aside), has the image's Y and X sizes, and has integer or boolean pixels. The folder must exist and be outside the experiment. Mask names must differ by more than case, and each image has one mask per type.
  • Warnings: mask_missing, mask_without_image, mask_outside_type_folder.
  • MaskInventory (for_image, to_manifest) and read_mask.
• mask_scan.py (new). The optional mask step. scan_masks reads the masks one image at a time and returns a MaskReport:
  • Kind of each mask type: instances, grouped, one_value or empty.
  • Relation for each ordered pair: same_objects, grouped_by, inside or none.
  • Evidence: the counts behind each relation, including parents with 0/1/2/3+ children, children outside every parent and children split between parents.
  • Hierarchy: the direct parents only. Nucleoli → nuclei, not cells. A type may have several direct parents.
  • Thresholds: DEFAULT_THRESHOLDS can be changed with thresholds=. mask_thresholds checks the values before any mask is read, and the values used are written in the report.
  • write_mask_report writes mask_relationships.json atomically. piece_counts and pair_counts are public.
• run.py: preflight(..., masks_folder=, ome_zarr_labels=) and run_experiment(...) with the same arguments. A mask error stops the run.
• manifest.py: an optional masks key. It is absent without masks, and the manifest stays version 1. plane_histograms.csv never depends on the masks.
• cli.py:
  • --masks FOLDER on check and run;
  • a new masks command: it prints the kinds, the relationships, the hierarchy tree and the thresholds used, then writes the report;
  • one option per threshold on masks, e.g. --inside-object-share 0.8, --same-objects-iou 0.7.
• report.py: plain-language summaries (mask_summary, mask_report_summary).
• pyproject.toml: scipy>=1.10 becomes a core dependency.
• Docs:
  • contract §13 (new), plus §8 and §9 rows;
  • plan §12 question 29;
  • user guide §2, §5, §6, §8 and §9;
  • AGENTS repository map, decisions log, commands and status;
  • README.

Decisions

Recorded in plan §12 Q29 and the AGENTS decisions log. Taken with the MorphoBricks owner (@guijacquemet):

• A separate masks folder, which must be outside the experiment, with one subfolder per mask type mirroring the experiment's folders. OME-Zarr label images are masks too.
• Confirmed: masks are 2D with the image's Y and X sizes. 3D and time-lapse masks are a later follow-up.
• Confirmed: mask pixels are integers or booleans.
• Thresholds are defaults the user may change. In Python with thresholds=, on the command line with one option each. They are checked, then written in the report.
• check and run read headers only. Reading the pixels is an optional step (masks command, scan_masks).
• Outputs: a masks key in the manifest, plus mask_relationships.json.
• Every mask provided is used. Users name the masks (by default the folder name; names must be unique) and confirm or correct the relationships. The report only suggests.
• SciPy is a core dependency. The owner agreed: "can use scipy if good reasons". Counting connected pieces per label on one 2048×2048 HPMEC cell mask took about 5 s with a NumPy union-find, and 0.3–0.9 s per mask with scipy.ndimage.

Notes for review

• Tag proposal (for you to decide, @IvanHCenalmor): could you tag v0.2.0 after this merges? The other bricks would then pin it to reuse the mask pairing (MorphoFeature decision D79: every brick depends on one preprocessing tag, which brings normalization). Follow-ups P4 and P5 would move to v0.3.0.
• Next, in separate PRs after this one:
  • a notebook step for the masks: name each mask, say what it contains, and confirm or correct the relationships;
  • a file saving those answers for the other bricks, as experiment_design.csv does for repeats.
• With a masks folder, the label images of OME-Zarr images are always used too. ome_zarr_labels=True uses them even without a folder; the masks command always does.
• Real-data check: on HPMEC (cells, nuclei and mitochondria masks, 2048×2048), masks printed the hierarchy cells > mito, nuclei: nuclei inside cells, and mitochondria grouped by cell with a median of 41 pieces per label.
• Synthetic cases in the tests:
  • Organoid > Cells > Nuclei > Nucleoli;
  • a second nuclei segmentation recognised as the same objects;
  • per-cell vesicles;
  • a cell with two nuclei and one with none;
  • a type with two unrelated direct parents;
  • changed thresholds turning a relation on or off.
• Not included: the notebook step and any change to the normalization brick.

Tests

• ruff check . and ruff format --check .: clean.
• pytest (Python 3.11, zarr 3): 969 passed, 1 skipped (main: 936 passed, 1 skipped). 33 new tests, in tests/test_masks.py and tests/test_mask_scan.py. Pieces are checked against an independent breadth-first-search reference on 50 random masks.
• The mask, scan and zarr-input tests also passed with zarr 2.18 before the thresholds change: 116 passed, 16 skipped.
• The mask-name case test is skipped on case-insensitive file systems, like the existing case_conflict test.

🤖 Generated with Claude Code

https://claude.ai/code/session_01U6vJfhGWQ7pTpsJhw48Zf7

[guijacquemet]

@IvanHCenalmor, one more commit on this PR, at the owner's request: 07a63bc, masks may also be named after their image.

Why. MorphoFeature ran the whole pipeline end to end on MorphoRelate's JUMP-Target-2 Cell Painting conversion: 4 fields; preprocessing, then the 4 bricks, then MorphoCoverage. That conversion writes masks as <image>_<type>.tif in the image's folders (DMSO/2021_04_26/E08/BR00121438_E08_f01_cells.tif), as many segmentation tools do, while this PR only read one folder per mask type.

What.

• The masks folder may now also mirror the experiment with the type after the image name, <condition>/.../<image>_<type>.tif.
• The two layouts may be mixed.
• A file that could name two images (an image name that starts another's: a_b_cells.tif beside images a and a_b), or that fits both layouts, is the new error ambiguous_mask. It is never guessed.
• Non-TIFF files directly in the masks folder keep the warning mask_outside_type_folder.
• Docs: contract §8 and §13, user guide §2 and §8, plan Q29, AGENTS.

Checked.

• On the four JUMP smoke fields, the masks command reading the converter's own files gives the same pairs and hierarchy as the per-type layout: cells → mito (grouped), nuclei → nucleoli, with nuclei holding 0/1/2/3+ nucleoli 0/55/65/217 times.
• 5 new tests: suffix pairing, mixed layouts, images in the experiment folder itself, an ambiguous name, and the same report from both layouts.
• pytest: 974 passed, 1 skipped; ruff clean.

---

Generated by Claude Code