diff --git a/AGENTS.md b/AGENTS.md index f0694846..e70cf30f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -62,10 +62,11 @@ src/morphobricks_preprocessing/ discovery.py folder walk, leaf folders, labels (condition / biological_repeat / technical_repeat), position_id per image and the duplicate check metadata.py TIFF header inspection (tifffile, no pixel reads), dtype/dimension/series checks, - OME channel names + significant bits, physical pixel sizes; header consistency - warnings (width/height, plane counts, pixel sizes) - pixel_sizes.py PhysicalSizes from OME/ImageJ TIFF metadata and the OME-Zarr level-0 scale, - sizes in micrometres for comparing (read_physical_sizes re-exported by zarr_export) + OME channel names + significant bits, physical sizes (TiffMetadata.physical_sizes); + header consistency warnings (width/height, plane counts, pixel sizes) + physical_sizes.py PhysicalSizes (pixel sizes, z-step, time step, a unit per axis, micrometers() + for comparing), read_physical_sizes (OME / ImageJ TIFF metadata; also importable + from zarr_export) and ngff_physical_sizes (OME-NGFF scale and axis units) dataset.py dataset consistency from the folders: biological repeats per condition, balance table, unbalanced_images, repeat_missing_conditions, CONSISTENCY_CODES (P2) design.py experiment design (P3): ExperimentDesign/DesignRow, default pairing, reconcile with @@ -85,14 +86,17 @@ src/morphobricks_preprocessing/ run.py preflight(), run_experiment() (incremental), sample_planes() for previews report.py plain-language summaries shared by CLI and notebook, group statistics tables cli.py, __main__ `morphobricks-preprocessing check|run` (also `python -m morphobricks_preprocessing`) - zarr_input.py OME-Zarr input: open_ngff (OME-NGFF 0.4/0.5 metadata -> level-0 array and axis - letters), ngff_label_names/open_ngff_labels (label images, for later bricks), + zarr_input.py OME-Zarr input: open_ngff (OME-NGFF 0.4/0.5 metadata -> level-0 array, axis + letters and physical sizes from the scale and axis units), + ngff_label_names/open_ngff_labels (label images, for later bricks), zarr 2/3 helpers, canonical_dtype; needs the `zarr` extra zarr_export.py TIFF -> OME-Zarr conversion (OmeZarrWriter, OME-NGFF 0.4, plane by plane or block by block), ConversionSettings (chunk size, compression, pyramid levels), estimate_conversion notebook_style.py notebook look and feel: cards, status cards, tiles, tables, ConfirmToggle, Browse dialogs - notebook_ui.py PreprocessingNotebook: the four notebook steps (needs the `notebook` extra) + notebook_ui.py PreprocessingNotebook: the four notebook steps (needs the `notebook` extra); + folder_step/axis_step take title, purpose, intro and show_next (and + show_output for step 1) so other bricks can embed them (plan §12 Q26) notebooks/MorphoBricks_Preprocessing.ipynb thin notebook: every code cell calls notebook_ui only tests/ pytest with simulated TIFFs (tests/tiff_tree.py, tests/simulation.py) experiments/binning_study/ the study behind the binning decisions (not part of the package) @@ -157,8 +161,10 @@ functions the tests use. - **Hierarchy:** `////.tif`. - **Leaf folders:** a leaf folder has images and no subfolders. Only TIFFs directly in leaf folders are analysed. -- **Missing levels:** a branch may stop early; missing labels are the reserved value - `single`. A folder *named* `single` (any case) is an error. +- **Missing levels:** a branch may stop early; missing labels are the value `single`. A + folder *named* `single` (lower case) that is the only folder in its place is that level's + value `single` (a level with one member, plan §12 question 23); spelled in another case, + or beside other folders, it is the error `reserved_folder_name`. - **Uniqueness:** labels are unique only among siblings, compared case-insensitively; siblings differing only in case are an error. Groups are identified by the full label path. @@ -356,6 +362,7 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p | OME-Zarr (Q12) | Input may be TIFF and/or OME-Zarr (folders ending in `.zarr`), mixed. OME-Zarr axes come from their metadata and are matched by role; level 0 of a single image only, several images per store is `multiple_series`. TIFF choice once per experiment (`convert_tiffs`): read directly (default) or convert to `/ome_zarr` (OME-NGFF 0.4) and compute rows from the copies; switching recomputes TIFF rows and removes unused copies. Reads OME-NGFF 0.4 and 0.5 (0.5 needs zarr 3). In the notebook: its own step-3 card "(Optional) Image conversion" after Settings (Q13): radio buttons (read directly = recommended), a disk-space warning (amber, red when the estimate exceeds free space, never blocking), and conversion settings (chunk size, compression + level, pyramid levels) shown only when converting. CLI `--convert-tiff` with `--chunk-size`, `--compression`, `--compression-level`, `--pyramid-levels`. No overlap setting: chunks never overlap | | System files (Q5) | Ignored silently; `single` check and sibling uniqueness are case-insensitive | | Too-deep folders (Q7) | Error | +| Folders named `single` (Q23) | Accepted as the level's value `single` when lower case and the only folder in its place (MorphoBricks one-member level, MorphoRelate M3); another case or other folders beside it stay `reserved_folder_name` | | Channel names (Q1) | Manifest only (from OME), warning `channel_names_differ`; CSV stays at schema 1 | | Plane coordinates (Q17) | `z`, `c`, `t` always written (0 for absent axes), `plane_id` always `::z=…,c=…,t=…`; CSV schema 2, version-1 tables recomputed on the next run | | OME-NGFF labels (Q18) | Read-only helpers `ngff_label_names`/`open_ngff_labels` for later bricks (CellPatcher masks); label images are never analysed and the outputs do not change | @@ -486,6 +493,11 @@ plainer version "not interactive enough". Descriptive, guided and forgiving is t ### Page layout (`notebook_style`) +- **Embedding steps 1 and 2 (Q26):** `folder_step` and `axis_step` keep this notebook's + words by default; another brick (MorphoRelate) passes its own `title`, `purpose` and + `intro` (HTML; `""` leaves the box out), `show_output=False` to hide the output folder + card (shown again when the check reports `output_inside_experiment`), and `show_next=False` to hide the "What next?" box naming this notebook's + steps. The widgets and their behaviour are the same either way. - **Page and cards:** `ui.page(title, subtitle, children)` wraps a step in a grey page (`mb-root`) with a title and a one-sentence purpose. `ui.card(header, children)` gives a white card with an upper-case header. `ui.intro(html)` adds a blue "what happens here" @@ -642,6 +654,11 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb and contract §3–§7 moved to the normalization contract (PRs #16 and #17, question 19), and `position_id` filled per cross-brick decision D02 (question 20), and follow-ups P2 (dataset consistency warnings, question 21) and P3 (experiment design, question 22). +- **Unreleased (for MorphoRelate milestone M3):** folders named `single` read as a + one-member level (question 23), a unit per axis in `PhysicalSizes` and physical sizes in + `inspect_image` (question 24), OME-Zarr physical sizes from the OME-NGFF scale (question + 25), and notebook steps 1 and 2 embeddable with their own words (question 26). No + version bump or tag. - **Next (not started):** follow-ups P4–P5 of the normalization plan (image type, intensity checks and plots). - **Not yet verified:** diff --git a/README.md b/README.md index a7885907..2a157025 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ Pre-alpha. Implemented and tested: - TIFF header inspection without reading pixels (`metadata`) - Dataset consistency warnings in the folder check: images per condition and biological repeat, repeats missing from some conditions, differing plane counts and pixel sizes - (`dataset`, `pixel_sizes`) + (`dataset`, `physical_sizes`) - Axis mapping: prefill from metadata, validation, applying one mapping to every file (`axes`) - Reading TIFFs one 2D plane at a time, with a memory limit and a preview plane; diff --git a/docs/contract.md b/docs/contract.md index d174de53..e76b05a9 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -23,7 +23,7 @@ whole experiment; any change to them goes through the normalization brick. | Leaf folder | A folder that contains images and no subfolders. Only images directly in leaf folders are analysed. | | Missing levels | A branch may stop early; the missing labels are `single`. If the root itself is the leaf, all three labels are `single`. | | Skipped levels | Not possible: a level only exists below its parent level. | -| Reserved name | A folder named `single` (in any upper/lower case) is an error. | +| Folders named `single` | `single` is the MorphoBricks value of a level with one member, the value missing levels get. A folder named `single` in lower case is accepted as that value when it is the only folder in its place (system folders aside): `/single/.tif` gives the biological repeat and the technical repeat `single`, the labels of `/.tif`. It stays an error (`reserved_folder_name`) when it is spelled in another case (`Single`, a different label) or has other folders beside it (its level would then have several members, one called `single`). | | Uniqueness | Folder names need only be unique among siblings, ignoring case: siblings whose names differ only in case are an error. | | Depth | At most three folder levels below the root. A folder below the technical-repeat level is an error. | | System files | Dot-files and dot-folders (`.DS_Store`, `._*`, `.ipynb_checkpoints`), `Thumbs.db`, `desktop.ini` and `__MACOSX` are ignored without a warning. | @@ -51,6 +51,7 @@ length 1 and no role, has coordinate 0. | OME-Zarr labels | The label images of an image's OME-NGFF `labels` group (0.4 and 0.5) are listed with `ngff_label_names` and opened, level 0, with `open_ngff_labels` (by name; the name may be left out when there is one). They are never analysed; later bricks such as patch extraction use them as masks. | | Experiment mapping | Defined on the TIFF arrangement when the experiment has TIFF files; in an experiment of OME-Zarr images only, on their arrangement with the most axes. | | Channel names, significant bits | Recorded in the run manifest when metadata provides them; not part of the CSV. Channel names that differ at the same channel index are a warning. | +| Physical sizes | Read with the header, not part of the CSV or the manifest, and compared between images in the folder check (§11, in micrometres with each axis's own unit; a size without a known unit cannot be compared): `PhysicalSizes` with the size of one step along X, Y, Z and T and each axis's unit (OME-NGFF unit names). TIFF: OME `PhysicalSize{X,Y,Z}` with their own units (micrometres when left out) and `TimeIncrement`, or ImageJ resolution, `spacing` and `finterval` with `unit`, `yunit`, `zunit`. OME-Zarr (0.4 and 0.5): the `scale` of level 0's `coordinateTransformations`, times the multiscales entry's own `scale` when given, with each axis's `unit` (common spellings such as `µm`, `um`, `micron` and `s` given as UDUNITS-2 names, others as written). Ångström (`Å`) and ImageJ's escaped micro sign (`\u00B5m`) are recognised. Nothing is guessed: what the metadata does not give is None; a scale in a separate file (`path`) or not one number per axis gives no sizes; an OME-Zarr scale without a unit is kept with the unit None. | | Plane counts | Images may have different numbers of channels, depth slices, time points or planes along an unassigned dimension; this is a warning (§11). | ## 3. Pixel types @@ -93,7 +94,7 @@ method is an offset and a scale, `(pixels − offset) / scale`. | Severity | Situations | Effect | |---|---|---| -| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); mixed dtypes; unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single`; sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; output folder inside the experiment; no usable planes | Processing is blocked | +| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); mixed dtypes; unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; output folder inside the experiment; no usable planes | Processing is blocked | | Warning | Files that are not images; empty folders; images outside leaf folders; images excluded by the user, or excluded paths that match no image; channel names differing between files; non-finite pixels; different image widths or heights; different numbers of channels, depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; planes over the memory limit that are read in blocks | Processing continues; the item is reported | Every message states what happened in plain language, lists the affected paths, and @@ -162,11 +163,12 @@ presentation does not change the conversion rules below. | Level 0 | The source values, unchanged, in the source pixel type; chunks of one plane along Z, C and T and at most the chunk size (16 to 8,192 px, default 1,024) in X and Y | | Further levels | 2 x 2 block means rounded to the pixel type; automatic (default: until a plane fits in 512 x 512, at most 8 levels in all), none, or 1 to 7 levels; for viewing only | | Large planes | A plane read in blocks is written block by block; its further levels are then built from the stored level above, in regions of even size, and equal those of a plane written whole | -| Scale | Physical pixel sizes, z-step and time step from OME or ImageJ metadata, with OME-NGFF units; 1 when unknown | +| Scale | Physical pixel sizes, z-step and time step from OME or ImageJ metadata, with OME-NGFF units, each axis with its own unit (OME `PhysicalSizeXUnit`, `PhysicalSizeYUnit`, `PhysicalSizeZUnit`, micrometres when left out; ImageJ `unit`, `yunit`, `zunit`); 1 when unknown | | Channels | `omero` block with channel names from metadata (or `Channel `) and each channel's observed minimum and maximum as display window | | Updates | Copies of unchanged images are kept, of new or changed images rewritten, and of images no longer analysed removed; a failed run leaves the earlier copies as they were (see §9 for publication) | | Switching | Turning conversion on or off recomputes every TIFF row; turning it off removes the copies the manifest recorded | | Missing copy | A converted TIFF whose copy was deleted counts as changed and is converted again | +| Outdated units | A copy whose axis units differ from those its TIFF's metadata gives now (copies written before each axis had its own unit, plan §12 question 24) counts as changed and is converted again; its rows do not change. Only the copy's `.zattrs` is read to tell | | Settings change | Copies written with other conversion settings (recorded in the manifest) are written again; their rows do not change | ## 11. Dataset consistency (folder check) diff --git a/docs/plan.md b/docs/plan.md index 408e5ea5..c237e00e 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -56,16 +56,19 @@ Images are TIFF files (`.tif`, `.tiff`) or OME-Zarr images: folders whose names TIFF and OME-Zarr images may be mixed in one experiment. Folder names are arbitrary labels and need only be unique among siblings. Each branch may -stop at any level; omitted levels receive the reserved value `single`. Levels cannot be -skipped. Labels are scoped to their parents: `bio_rep1` under two conditions are two -different samples, so groups are always identified by the full label path. +stop at any level; omitted levels receive the value `single` (a level with one member, +question 23). Levels cannot be skipped. Labels are scoped to their parents: `bio_rep1` +under two conditions are two different samples, so groups are always identified by the +full label path. A folder containing images and no subfolders is a leaf folder. Only images directly inside leaf folders are analysed. If the experiment root itself contains images and has no subfolders, it is the leaf folder and every label is `single`. During preflight, show a tree/summary of the inferred labels and ask the user to confirm -or correct it. The folder name `single` is reserved; report it as an error. Report +or correct it. A folder may be named `single` (lower case) when it is the only folder in +its place: it is then a level with one member, labelled `single` like a missing level +(question 23). Spelled in another case, or beside other folders, it is an error. Report warnings for files that are not images, empty folders and images outside leaf folders, and explain that they will not be analysed. @@ -213,7 +216,8 @@ Block processing for: mixed dtypes; unreadable TIFF files or OME-Zarr folders in folders (or OME-Zarr images without the `zarr` extra); OME-Zarr images whose declared axes do not fit the experiment's roles; unsupported sample formats or axes; missing/duplicate X or Y, duplicate roles or inconsistent mapping; a plane -that cannot be read within the memory limit, not even in blocks; a folder named `single`; sibling folders whose names differ only in +that cannot be read within the memory limit, not even in blocks; a folder named `single` +in another case or beside other folders; sibling folders whose names differ only in case; a folder below the technical-repeat level; an output folder inside the experiment; no usable planes. System files (dot-files, `Thumbs.db`, `desktop.ini`, `__MACOSX`) are ignored silently. @@ -312,7 +316,7 @@ API, CLI). The notebook and CLI only call the API. - **Dataset consistency (follow-up P2 of the normalization plan):** the folder check counts images per condition and biological repeat, warns about unbalanced counts and repeats missing from some conditions (`dataset`), and compares the headers' plane counts and - physical pixel sizes (`metadata`, `pixel_sizes`); the notebook shows them, with the + physical pixel sizes (`metadata`, `physical_sizes`); the notebook shows them, with the earlier consistency issues, in step 1's "Dataset consistency" panel. *Implemented, see question 21.* - **Experiment design (follow-up P3):** after the labels, step 1 pairs biological repeats @@ -320,6 +324,11 @@ API, CLI). The notebook and CLI only call the API. records control conditions, in `experiment_design.csv` (`design`); reruns with the same folders skip the questions, the command line uses the file or writes a template, and the manifest records its hash. *Implemented, see question 22.* +- **Input support for MorphoRelate milestone M3 (unreleased):** a folder named `single` + read as a one-member level, physical sizes with a unit per axis in the image header + (TIFF and OME-Zarr, from the OME-NGFF scale), and steps 1 and 2 of the notebook + embeddable by another brick. Outputs are unchanged. *Implemented, see questions 23 to + 26.* ## 11. Tests and acceptance criteria @@ -572,3 +581,65 @@ Raised during review; to be decided before the milestones that need them: `ExperimentDesign.repeat_groups()` give later bricks the `{(condition, biological_repeat): repeat_group}` mapping. `plane_histograms.csv` does not change.* +23. A folder named `single` was an error, but in the MorphoBricks convention (CellPatcher's + trees, MorphoRelate's data such as `/single/`) a level written + `single` means "this level exists with one member". Can such a folder be read? + *Decided (October 2026, for MorphoRelate milestone M3): yes. A folder named `single` in + lower case is accepted as its level's value `single` when it is the only folder in its + place (system folders such as `.ipynb_checkpoints` and `__MACOSX` aside), so + `/single/` gives `biological_repeat` and `technical_repeat` + `single`, the labels of `/`. The reservation had no recorded reason + beyond keeping `single` for missing levels; what it protects is that `single` keeps one + meaning, so two cases stay errors (`reserved_folder_name`): another spelling (`Single` + would be a label different from the `single` written for missing levels), and a + `single` folder with other folders beside it (the level would have several members, + one called `single`). The CSV, the manifest and their versions are unchanged.* +24. Physical pixel sizes: `read_physical_sizes` applied the X unit to Y and Z, and callers + (MorphoRelate milestone M3's `inspect`) had to open each file a second time to get + them. Should each axis keep its own unit, and should the header inspection carry them? + *Decided (October 2026): yes to both. `PhysicalSizes` gains `x_unit`, `y_unit` and + `z_unit`; `unit(role)` returns each axis's own unit, from OME `PhysicalSizeYUnit` and + `PhysicalSizeZUnit` (the OME default, micrometres, when left out, as for X) or ImageJ + `yunit` and `zunit` (the X unit when left out, as ImageJ writes them). The per-axis + units are kept as given and resolved when read: an axis without its own unit has + `space_unit`, so sizes made with `space_unit` alone keep it for every space axis and + stay consistent under `dataclasses.replace` and pickles written before; the readers + give `space_unit` the unit of X, as before (None when a space axis with a size has no + unit). Equality and hashing compare the resolved units, so existing callers are + unchanged. `inspect_image` (`TiffMetadata`) gains the + field `physical_sizes`, read from the header already open, for information only; the + existing fields are unchanged. OME-Zarr copies of images whose Y or Z unit differs + from X now write that axis's own unit; on the next run, a copy written before with + other axis units (read from its `.zattrs`) counts as changed and is converted again + through the usual staged publication, so one output folder never mixes the two (its + rows do not change). `PhysicalSizes` and `read_physical_sizes` moved + to the module `physical_sizes` and stay importable from `zarr_export` and the package. + The CSV, the manifest and their versions are unchanged.* +25. OME-Zarr physical sizes: `open_ngff` read the axes but not the OME-NGFF scale, so the + physical size of an OME-Zarr image's pixels was not available (MorphoRelate milestone + M3 reads OME-Zarr input). Should it be? + *Decided (October 2026): yes, in the form of question 24. `NgffImage` gains + `physical_sizes`: the `scale` of level 0's `coordinateTransformations`, multiplied by + the multiscales entry's own `coordinateTransformations` scale when it has one (OME-NGFF + applies them in turn; translations are ignored), for X, Y, Z and T, with each axis's + `unit` (OME-NGFF 0.4 and 0.5), common spellings (`µm`, `um`, `micron`, `s`) given as + the UDUNITS-2 names TIFF units get, other units as written. `inspect_image` records them for OME-Zarr + images as for TIFF files, and label images opened with `open_ngff_labels` carry their + own. Nothing is guessed: without a level-0 scale, with a scale stored in a separate + file (`path`) or not one number per axis, every size is None; an axis without a unit + has the unit None, and a scale that is not a positive number is None. A scale given + without a unit is kept, since only the caller can tell a real calibration from a + placeholder (this brick's own copies write 1 without a unit when the size is not + known). Information only: the CSV, the manifest and their versions are unchanged.* +26. Can another brick's notebook reuse steps 1 and 2 (folder check and axis review)? + MorphoRelate milestone M3's "your data" notebook needs them, but their title, purpose + and intro speak of this brick's histograms, step 1 has the card for this brick's output + folder, and step 2's "What next?" box sends the user to compute the histograms. + *Decided (October 2026): yes. `PreprocessingNotebook.folder_step` and `axis_step` take + an optional `title` and `purpose` (the sentence under the title), a keyword `intro` + (the blue box; an empty string leaves it out) and `show_next` (False hides the "What + next?" box), and `folder_step` takes `show_output` (False hides the "Output folder + (optional)" card; the check then uses the folder in its box, the default when empty, + and the card is shown again when the check finds that folder inside the experiment, + so the box to fix it is never hidden). The defaults give the pages as before, and this notebook calls them + without arguments; the widgets, the checks and the gating are the same either way.* diff --git a/docs/user_guide.md b/docs/user_guide.md index a9d5dfb0..93e1b56b 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -74,8 +74,12 @@ experiment/ it where you would put a TIFF file. TIFF and OME-Zarr images can be mixed. - Only images in folders **without subfolders** are analysed. Images next to subfolders are reported and skipped. -- Do not name a folder `single`, and do not use two neighbouring folder names that differ - only in upper/lower case (`Control` and `control`). Both are errors. +- A folder may be named `single` (in lower case) when it is the only folder in its place: + like a missing level, it means the level has one member, so `treated/single/img_01.tif` + has the biological repeat `single`. A folder named `Single` or `SINGLE`, or a `single` + folder next to other folders, is an error. +- Do not use two neighbouring folder names that differ only in upper/lower case + (`Control` and `control`). This is an error. - Each image is one position (field of view), named after its file without the extension: `img_01.tif` and `img_01.ome.zarr` are both position `img_01`. The same name may be used in different folders (`tech_rep1/img_01.tif` and `tech_rep2/img_01.tif` are two @@ -316,8 +320,10 @@ The default output folder is next to the experiment folder: `ome_zarr/control/bio_rep1/tech_rep1/img_01.ome.zarr`. The full-resolution level holds exactly the source pixel values; smaller levels are averages for fast viewing (for example in napari with the `napari-ome-zarr` plugin). Pixel sizes and channel names are - copied from the metadata when present. Copies of unchanged images are kept on later - runs, copies of deleted or excluded images are removed. Switching back to reading the + copied from the metadata when present, each axis with its own unit. Copies of unchanged + images are kept on later runs, copies of deleted or excluded images are removed. Copies + written by an earlier version with the X unit on every axis are converted again once, + with their own units; the table does not change. Switching back to reading the TIFF images directly removes the copies and recomputes their rows. Details in [contract.md](contract.md) §10. @@ -354,7 +360,7 @@ continues. |---|---|---| | `experiment_not_found` | Error | The folder does not exist. Select an existing folder. | | `unreadable_folder` | Error | A folder could not be opened. Check permissions. | -| `reserved_folder_name` | Error | A folder is named `single`. Rename it. | +| `reserved_folder_name` | Error | A folder is named `single` in another case (`Single`), or a folder named `single` has other folders beside it. Rename it to `single` in lower case, or give it a name of its own. | | `case_conflict` | Error | Neighbouring folders differ only in case. Rename or merge them. | | `duplicate_position_id` | Error | Two images in one folder have the same name apart from the extension or upper/lower case (`img_01.tif` and `img_01.ome.zarr`), so their positions cannot be told apart. Rename one, or move it out of the experiment. | | `folder_too_deep` | Error | Folders below the technical repeat. Move the images up, or select a lower folder as the experiment. | diff --git a/src/morphobricks_preprocessing/__init__.py b/src/morphobricks_preprocessing/__init__.py index 99aa0801..50d5dda4 100644 --- a/src/morphobricks_preprocessing/__init__.py +++ b/src/morphobricks_preprocessing/__init__.py @@ -76,6 +76,7 @@ inspect_ome_zarr, inspect_tiff, ) +from .physical_sizes import PhysicalSizes, read_physical_sizes from .planes import ( DEFAULT_MEMORY_LIMIT, Block, @@ -97,10 +98,8 @@ ConversionEstimate, ConversionSettings, OmeZarrWriter, - PhysicalSizes, estimate_conversion, ome_zarr_relative_path, - read_physical_sizes, ) from .zarr_input import ( NgffError, diff --git a/src/morphobricks_preprocessing/discovery.py b/src/morphobricks_preprocessing/discovery.py index fcc7e375..38c425a2 100644 --- a/src/morphobricks_preprocessing/discovery.py +++ b/src/morphobricks_preprocessing/discovery.py @@ -3,7 +3,10 @@ The experiment root holds up to three levels of folders (condition, biological repeat, technical repeat). A folder with images and no subfolders is a leaf; its labels are the folder names on the way down, with ``single`` for the levels -the branch does not have. The rules are in ``docs/contract.md`` §1. +the branch does not have. A folder may itself be named ``single`` (in lower +case) when it is the only folder in its place: like a missing level, it is a +level with one member (the MorphoBricks convention). The rules are in +``docs/contract.md`` §1. Images are TIFF files (``.tif``, ``.tiff``) and OME-Zarr images, which are folders whose names end in ``.zarr`` (usually ``.ome.zarr``). An OME-Zarr folder @@ -116,7 +119,8 @@ class LeafFolder: Path relative to the experiment root with ``/`` separators; ``"."`` when the experiment root itself is the leaf. condition, biological_repeat, technical_repeat - Labels from the folder names; ``"single"`` for missing levels. + Labels from the folder names; ``"single"`` for missing levels and for + folders named ``single``. images Relative paths of the images (TIFF files and OME-Zarr folders) directly in this folder, sorted. @@ -247,6 +251,7 @@ def __init__(self, root: Path): self.leaves: list[LeafFolder] = [] self.issues: list[ValidationIssue] = [] self.reserved: list[str] = [] + self.single_with_siblings: list[str] = [] self.case_conflicts: list[str] = [] self.too_deep: list[str] = [] self.unreadable: list[str] = [] @@ -307,8 +312,17 @@ def _check_names(self, subfolders: list[Path]) -> None: by_key: dict[str, list[Path]] = {} for path in subfolders: by_key.setdefault(path.name.casefold(), []).append(path) - if path.name.casefold() == SINGLE: + if path.name.casefold() != SINGLE: + continue + # A folder named "single" gives its level the value a missing level gets: + # one member. Two cases would make "single" ambiguous, so they stay errors: + # another spelling ("Single" would be a different label from the "single" + # written for missing levels), and other folders beside it (the level + # would then have several members, one of them called "single"). + if path.name != SINGLE: self.reserved.append(self.relative(path)) + elif len(subfolders) > 1: + self.single_with_siblings.append(self.relative(path)) for paths in by_key.values(): if len(paths) > 1: self.case_conflicts.extend(self.relative(path) for path in paths) @@ -340,10 +354,24 @@ def finish(self) -> tuple[ValidationIssue, ...]: issues.append( error( "reserved_folder_name", - f'"{SINGLE}" is reserved for missing hierarchy levels and cannot be ' - "used as a folder name.", + f'A folder named "{SINGLE}" marks a level with one member, and its label ' + f'is always written "{SINGLE}" in lower case. Written in another case it ' + "would be a different label.", self.reserved, - f'Rename these folders to something other than "{SINGLE}".', + f'Rename these folders to "{SINGLE}" in lower case, or give them a name of ' + "their own.", + ) + ) + if self.single_with_siblings: + issues.append( + error( + "reserved_folder_name", + f'A folder named "{SINGLE}" marks a level with one member, so it must be ' + "the only folder in its place. These folders have other folders beside " + "them.", + self.single_with_siblings, + "Give these folders a name of their own (for example rep1), or move the " + "other folders out.", ) ) if self.case_conflicts: diff --git a/src/morphobricks_preprocessing/metadata.py b/src/morphobricks_preprocessing/metadata.py index 673d8d1d..8ae16c8a 100644 --- a/src/morphobricks_preprocessing/metadata.py +++ b/src/morphobricks_preprocessing/metadata.py @@ -18,7 +18,7 @@ import logging import os from collections.abc import Callable, Hashable, Iterable -from dataclasses import dataclass, field +from dataclasses import dataclass from pathlib import Path from xml.etree import ElementTree @@ -27,7 +27,7 @@ from morphobricks_normalization.dtypes import UnsupportedDtypeError, check_supported from .discovery import DiscoveryResult -from .pixel_sizes import PhysicalSizes, ngff_physical_sizes, tiff_physical_sizes +from .physical_sizes import PhysicalSizes, _tiff_sizes from .validation import ValidationIssue, error, has_errors, plural, warning from .zarr_input import NgffError, ZarrUnavailableError, is_zarr_name, open_ngff @@ -74,8 +74,10 @@ class TiffMetadata: kind ``"tiff"`` or ``"ome-zarr"``. physical_sizes - Pixel sizes and z-step from the metadata (OME or ImageJ for TIFF files, - the level-0 scale for OME-Zarr images); None where unknown. + Pixel sizes, z-step and time step with their units, from OME or ImageJ + metadata (TIFF) or the OME-NGFF scale (OME-Zarr), as + :func:`.physical_sizes.read_physical_sizes` gives them; all None when + not given or the image does not open. For information only. """ relative_path: str @@ -89,7 +91,7 @@ class TiffMetadata: channel_names: tuple[str | None, ...] = () significant_bits: int | None = None kind: str = TIFF - physical_sizes: PhysicalSizes = field(default_factory=PhysicalSizes) + physical_sizes: PhysicalSizes = PhysicalSizes() def length(self, axis: str) -> int: """Return the length of the dimension labelled ``axis``, 1 if there is none.""" @@ -180,8 +182,8 @@ def inspect_tiff(path: str | os.PathLike, relative_path: str | None = None) -> T axes=str(first.axes), series_count=len(series), page_count=len(tif.pages), - physical_sizes=tiff_physical_sizes(tif), **_ome_details(tif), + physical_sizes=_tiff_sizes(tif), ) except Exception as exc: # any failure to parse means the file cannot be used logger.debug("Cannot read the TIFF header of %s", path, exc_info=True) @@ -214,7 +216,7 @@ def inspect_ome_zarr(path: str | os.PathLike, relative_path: str | None = None) page_count=0, channel_names=image.channel_names, kind=OME_ZARR, - physical_sizes=ngff_physical_sizes(image.axes, image.scale, image.units), + physical_sizes=image.physical_sizes, ) except (NgffError, ZarrUnavailableError) as exc: logger.debug("Cannot read the OME-Zarr image %s", path, exc_info=True) diff --git a/src/morphobricks_preprocessing/notebook_ui.py b/src/morphobricks_preprocessing/notebook_ui.py index d2e3fa81..25d95744 100644 --- a/src/morphobricks_preprocessing/notebook_ui.py +++ b/src/morphobricks_preprocessing/notebook_ui.py @@ -181,6 +181,7 @@ def __init__(self, experiment_folder: str = "", output_folder: str = ""): self.log = _LogCapture() self._checked_folder = "" self._preview_shown = False + self._show_output = True self._layouts_cache: tuple[tuple, dict[str, PlaneLayout]] | None = None self._build_folder_step(str(experiment_folder), str(output_folder)) self._build_axis_step() @@ -214,18 +215,64 @@ def welcome(self) -> widgets.Widget: ], ) - def folder_step(self) -> widgets.Widget: - """Step 1: choose and check the experiment folder, confirm the labels.""" + def folder_step( + self, + title: str | None = None, + purpose: str | None = None, + *, + intro: str | None = None, + show_output: bool = True, + show_next: bool = True, + ) -> widgets.Widget: + """Step 1: choose and check the experiment folder, confirm the labels. + + The defaults give this notebook's step. Another brick embedding the step + (it checks the folder and the image headers, nothing more) can give its + own words and leave out the output folder of this brick's histograms. + + Parameters + ---------- + title + Page title; None gives "Step 1 · Check the experiment folder". + purpose + One-sentence purpose under the title (HTML); None gives this + notebook's. + intro + Text of the blue "what happens here" box (HTML); None gives this + notebook's, an empty string leaves the box out. + show_output + False hides the "Output folder (optional)" card, which sets where this + brick writes its histogram table; the check then uses the folder in its + box (the default when empty). The card is shown again when the check + finds that folder inside the experiment, so it can be fixed. + show_next + False hides the "What next?" box, which names this notebook's next + steps. + + Returns + ------- + ipywidgets.Widget + The page. + """ + ui.set_visible(self.folder_next, show_next) + self._show_output = show_output + ui.set_visible(self.output_card, show_output) + if intro is None: + intro = ( + "What happens here: the folders are scanned, the labels are taken " + "from the folder names (condition / biological repeat / technical repeat) " + "and every image header is checked. Problems are listed with what to do." + ) return ui.page( - "Step 1 · Check the experiment folder", - "Choose the folder with your images. It is checked without reading any pixels " - "and without changing anything.", + "Step 1 · Check the experiment folder" if title is None else title, + ( + "Choose the folder with your images. It is checked without reading any pixels " + "and without changing anything." + ) + if purpose is None + else purpose, [ - ui.intro( - "What happens here: the folders are scanned, the labels are taken " - "from the folder names (condition / biological repeat / technical repeat) " - "and every image header is checked. Problems are listed with what to do." - ), + *([ui.intro(intro)] if intro else []), self.folder_card, self.output_card, self.exclude_box, @@ -240,20 +287,53 @@ def folder_step(self) -> widgets.Widget: ], ) - def axis_step(self) -> widgets.Widget: - """Step 2: axis roles, preview and confirmation.""" + def axis_step( + self, + title: str | None = None, + purpose: str | None = None, + *, + intro: str | None = None, + show_next: bool = True, + ) -> widgets.Widget: + """Step 2: axis roles, preview and confirmation. + + The defaults give this notebook's step; another brick embedding it can give + its own words, since this notebook's explain the roles by the planes its + histograms are computed on. + + Parameters + ---------- + title + Page title; None gives "Step 2 · Review the axis roles". + purpose + One-sentence purpose under the title (HTML); None gives this + notebook's. + intro + Text of the blue "why this matters" box (HTML); None gives this + notebook's, an empty string leaves the box out. + show_next + False hides the "What next?" box, which sends the user to this + notebook's step 3 to compute the histograms. + + Returns + ------- + ipywidgets.Widget + The page. + """ + ui.set_visible(self.axis_next, show_next) + if intro is None: + intro = ( + "Why this matters: every image is cut into flat 2D planes, one per " + "depth slice, channel and time point. The roles below are suggested from " + "the image metadata, which is not always right, so check them with the " + "preview before confirming." + ) return ui.page( - "Step 2 · Review the axis roles", - "Tell the brick which image dimension is width, height, depth, channel and time.", - [ - ui.intro( - "Why this matters: every image is cut into flat 2D planes, one per " - "depth slice, channel and time point. The roles below are suggested from " - "the image metadata, which is not always right, so check them with the " - "preview before confirming." - ), - self.axis_box, - ], + "Step 2 · Review the axis roles" if title is None else title, + "Tell the brick which image dimension is width, height, depth, channel and time." + if purpose is None + else purpose, + [*([ui.intro(intro)] if intro else []), self.axis_box], ) def run_step(self) -> widgets.Widget: @@ -383,7 +463,8 @@ def _build_folder_step(self, folder: str, output: str) -> None: [ ui.help_text( "Each row is a folder of images and the labels its images get. " - "single means the folder level is missing. To change a label, " + "single means the level has one member: the folder level is " + "missing, or the folder is named single. To change a label, " "rename the folder and press Check folder again." ), self.labels_table, @@ -588,6 +669,11 @@ def _show_issue_summary(self) -> None: self.folder_issues.value = issues_html(others) self.issue_counts.value = issue_count_chips(others) ui.set_visible(self.issues_card, bool(others)) + # A hidden output card comes back when its folder is the problem to fix. + ui.set_visible( + self.output_card, + self._show_output or any(i.code == "output_inside_experiment" for i in issues), + ) blocking = any(i.code in EXCLUDABLE_CODES for i in issues) ui.set_visible(self.exclude_button, blocking) self._refresh_folder_next() diff --git a/src/morphobricks_preprocessing/physical_sizes.py b/src/morphobricks_preprocessing/physical_sizes.py new file mode 100644 index 00000000..71659080 --- /dev/null +++ b/src/morphobricks_preprocessing/physical_sizes.py @@ -0,0 +1,403 @@ +"""Physical pixel sizes from image metadata, without reading pixels. + +The size of one pixel along X and Y, the z-step and the time step come from OME +or ImageJ metadata in TIFF files. Units are given as OME-NGFF (UDUNITS-2) names +such as ``"micrometer"`` and ``"second"``, one per axis, so that an image whose +Y pixels are measured in another unit than its X pixels is described as it is. +Sizes are never guessed: what the metadata does not give is None. + +OME-Zarr images give the same sizes in their OME-NGFF metadata (0.4 and 0.5): +the ``scale`` of the level-0 dataset's ``coordinateTransformations``, times the +``scale`` of the image's own ``coordinateTransformations`` when it has them, with +each axis's ``unit`` (:func:`ngff_physical_sizes`, used by :mod:`.zarr_input`). + +They are recorded with an image's header (:func:`.metadata.inspect_image`), +written as the scale of OME-Zarr copies (``docs/contract.md`` §2 and §10), and +compared between the images of an experiment in the folder check (§11). +""" + +from __future__ import annotations + +import logging +import math +import os +import re +from collections.abc import Sequence +from dataclasses import dataclass +from xml.etree import ElementTree + +import tifffile + +logger = logging.getLogger(__name__) + +# Units in OME and ImageJ metadata, and their OME-NGFF (UDUNITS-2) names. +_SPACE_UNITS = { + "pm": "picometer", + "Å": "angstrom", # the OME symbol, LATIN CAPITAL LETTER A WITH RING ABOVE + "\u212b": "angstrom", # ANGSTROM SIGN + "å": "angstrom", # ImageJ units are compared in lower case + "angstrom": "angstrom", + "nm": "nanometer", + "µm": "micrometer", + "μm": "micrometer", + "um": "micrometer", + "micron": "micrometer", + "microns": "micrometer", + "micrometer": "micrometer", + "mm": "millimeter", + "cm": "centimeter", + "m": "meter", +} +_TIME_UNITS = { + "ns": "nanosecond", + "µs": "microsecond", + "μs": "microsecond", + "us": "microsecond", + "ms": "millisecond", + "s": "second", + "sec": "second", + "min": "minute", + "h": "hour", +} + +# Micrometres per OME-NGFF space unit, to compare sizes given in different units. +_MICROMETERS = { + "angstrom": 1e-4, + "picometer": 1e-6, + "nanometer": 1e-3, + "micrometer": 1.0, + "millimeter": 1e3, + "centimeter": 1e4, + "meter": 1e6, +} + +# The OME schema's unit when a PhysicalSize*Unit attribute is left out. +_OME_DEFAULT_SPACE_UNIT = "µm" +_OME_DEFAULT_TIME_UNIT = "s" + + +@dataclass(frozen=True, eq=False) +class PhysicalSizes: + """Pixel sizes from metadata; None where unknown. + + Attributes + ---------- + x, y, z + Pixel size and z-step, each in its own unit (see :meth:`unit`). + t + Time between frames in ``time_unit``. + space_unit + The unit of the space axes that have no unit of their own. Given alone, as + before per-axis units, it is the unit of X, Y and Z; the readers give it + the unit of X (None when a space axis with a size has no unit). + time_unit + The unit of T. + x_unit, y_unit, z_unit + An axis's own unit, as an OME-NGFF unit name such as ``"micrometer"``; + None means ``space_unit``. :meth:`unit` resolves them, so they stay + consistent under :func:`dataclasses.replace` and pickling. + + Two sizes are equal when their sizes, ``space_unit`` and resolved units are. + """ + + x: float | None = None + y: float | None = None + z: float | None = None + t: float | None = None + space_unit: str | None = None + time_unit: str | None = None + x_unit: str | None = None + y_unit: str | None = None + z_unit: str | None = None + + def _space_units(self) -> tuple[str | None, str | None, str | None]: + """Return the units of X, Y and Z, each falling back to ``space_unit``.""" + return tuple( + own if own is not None else self.space_unit + for own in (self.x_unit, self.y_unit, self.z_unit) + ) + + def _key(self) -> tuple: + return (self.x, self.y, self.z, self.t, self.space_unit, self.time_unit) + ( + self._space_units() + ) + + def __eq__(self, other: object) -> bool: + if other.__class__ is not self.__class__: + return NotImplemented + return self._key() == other._key() + + def __hash__(self) -> int: + return hash(self._key()) + + def scale(self, role: str) -> float: + """Return the size of one step along ``role`` (1 when unknown).""" + value = {"X": self.x, "Y": self.y, "Z": self.z, "T": self.t}.get(role) + return 1.0 if value is None or not value > 0 else float(value) + + def unit(self, role: str) -> str | None: + """Return the unit of ``role``, or None when unknown or for channels.""" + if role == "T": + return self.time_unit if self.t is not None else None + if role == "C": + return None + index = "XYZ".index(role) + size = (self.x, self.y, self.z)[index] + return self._space_units()[index] if size is not None else None + + def micrometers(self, role: str) -> float | None: + """Return the size along ``"X"``, ``"Y"`` or ``"Z"`` in micrometres. + + Uses the axis's own unit. None when the size or its unit is unknown, so + that sizes in different units can be compared and an unknown unit is + never taken for another. + """ + value = {"X": self.x, "Y": self.y, "Z": self.z}[role] + factor = _MICROMETERS.get(self.unit(role) or "") + if value is None or factor is None: + return None + return value * factor + + +def _sizes( + x: float | None, + y: float | None, + z: float | None, + t: float | None, + units: dict[str, str | None], + time_unit: str | None, +) -> PhysicalSizes: + """Build sizes from metadata with each space axis's own unit (``units`` by letter). + + ``space_unit`` is the unit of X, as before per-axis units, unless a space axis + with a size has no unit: it would then fall back to X's unit, so ``space_unit`` + is None. + """ + unitless = any( + size is not None and units[a] is None for a, size in zip("XYZ", (x, y, z), strict=True) + ) + return PhysicalSizes( + x=x, + y=y, + z=z, + t=t, + space_unit=None if unitless else units["X"], + time_unit=time_unit, + x_unit=units["X"], + y_unit=units["Y"], + z_unit=units["Z"], + ) + + +def read_physical_sizes(path: str | os.PathLike) -> PhysicalSizes: + """Read pixel sizes from OME or ImageJ metadata, without reading pixels. + + Each space axis has its own unit: an OME ``PhysicalSizeYUnit`` or + ``PhysicalSizeZUnit`` is used for Y or Z (the OME default, micrometres, when + left out), and ImageJ's ``yunit`` and ``zunit`` when given. + + Returns all-None sizes when the file has no such metadata or it cannot be + parsed; sizes are never guessed. + """ + try: + with tifffile.TiffFile(path) as tif: + return _tiff_sizes(tif) + except Exception: # metadata is optional; a damaged header is reported elsewhere + logger.debug("Cannot read physical sizes of %s", path, exc_info=True) + return PhysicalSizes() + + +def ngff_physical_sizes(multiscale: dict, axes: str) -> PhysicalSizes: + """Return the level-0 physical sizes of an OME-NGFF image (0.4 or 0.5). + + The size of one step along each axis is the ``scale`` of the first dataset's + ``coordinateTransformations`` (level 0), multiplied by the ``scale`` of the + multiscales entry's own ``coordinateTransformations`` when it has them, as + OME-NGFF applies them in turn; translations do not change sizes. Units are + each axis's ``unit`` as UDUNITS-2 names, the names OME-NGFF uses and + :func:`read_physical_sizes` gives: common spellings such as ``"µm"``, ``"um"`` + or ``"micron"`` become ``"micrometer"``, other units are kept as written. + + Nothing is guessed: without a level-0 scale (OME-NGFF 0.1 to 0.3, or a scale + stored in a separate file through ``path``) or with a malformed one, every + size is None; an axis without a ``unit`` has the unit None, and a scale that + is not a positive finite number is None. A scale given without a unit is + kept: it may be a placeholder (some writers, this brick's OME-Zarr copies + included, write 1 when the size is not known). + + Parameters + ---------- + multiscale + One entry of the image's ``multiscales`` metadata. + axes + One letter per dimension, as :func:`.zarr_input.ngff_axes` gives them; + X, Y, Z and T are read, the others (C, Q) ignored. + + Returns + ------- + PhysicalSizes + Sizes and units of X, Y, Z and T. + """ + datasets = multiscale.get("datasets") or [] + first = datasets[0] if isinstance(datasets, list) and datasets else {} + try: + scale = _ngff_scale(first, len(axes)) + overall = _ngff_scale(multiscale, len(axes)) + except ValueError: + return PhysicalSizes() + if scale is None: + return PhysicalSizes() + if overall is not None: + scale = [ + a * b if a is not None and b is not None else None + for a, b in zip(scale, overall, strict=True) + ] + declared = multiscale.get("axes") + units: list[str | None] = [None] * len(axes) + if isinstance(declared, list) and len(declared) == len(axes): + units = [axis.get("unit") if isinstance(axis, dict) else None for axis in declared] + found: dict[str, tuple[float | None, str | None]] = {} + for letter, size, unit in zip(axes, scale, units, strict=True): + if letter in "XYZT": + table = _TIME_UNITS if letter == "T" else _SPACE_UNITS + found.setdefault(letter, (_number(size), _ngff_unit(unit, table))) + sizes = {letter: found.get(letter, (None, None)) for letter in "XYZT"} + return _sizes( + sizes["X"][0], + sizes["Y"][0], + sizes["Z"][0], + sizes["T"][0], + {letter: sizes[letter][1] for letter in "XYZ"}, + sizes["T"][1], + ) + + +def _ngff_scale(entry: dict, ndim: int) -> list[float | None] | None: + """Return the ``scale`` in an entry's ``coordinateTransformations``. + + None when the entry has no scale transformation; raises ValueError when its + transformations are not a list, or its scale cannot be used (stored in a + file, or not one number per axis). + """ + transforms = entry.get("coordinateTransformations") if isinstance(entry, dict) else None + if transforms is None: + return None + if not isinstance(transforms, list): + raise ValueError("the coordinate transformations are not a list") + for transform in transforms: + if not isinstance(transform, dict) or transform.get("type") != "scale": + continue + values = transform.get("scale") + if not isinstance(values, Sequence) or isinstance(values, str) or len(values) != ndim: + raise ValueError("the scale is not given as one number per axis") + return [_scale_value(v) for v in values] + return None + + +def _scale_value(value) -> float | None: + """Return one scale value as a float; None when it is not a usable number.""" + if isinstance(value, bool) or not isinstance(value, int | float): + return None + try: + return float(value) + except OverflowError: # an integer too large for a float is not a pixel size + return None + + +def _tiff_sizes(tif: tifffile.TiffFile) -> PhysicalSizes: + """Return the physical sizes of an open TIFF file (all None when not given).""" + try: + if tif.is_ome and tif.ome_metadata: + return _ome_sizes(tif.ome_metadata) + if tif.is_imagej: + return _imagej_sizes(tif) + except Exception: # metadata is optional; a damaged header is reported elsewhere + logger.debug("Cannot read physical sizes of %s", tif.filename, exc_info=True) + return PhysicalSizes() + + +def _ome_sizes(xml: str) -> PhysicalSizes: + root = ElementTree.fromstring(xml) + pixels = next((e for e in root.iter() if e.tag.rsplit("}", 1)[-1] == "Pixels"), None) + if pixels is None: + return PhysicalSizes() + units = { + axis: _SPACE_UNITS.get(pixels.get(f"PhysicalSize{axis}Unit", _OME_DEFAULT_SPACE_UNIT)) + for axis in "XYZ" + } + time_unit = _TIME_UNITS.get(pixels.get("TimeIncrementUnit", _OME_DEFAULT_TIME_UNIT)) + return _sizes( + _number(pixels.get("PhysicalSizeX")), + _number(pixels.get("PhysicalSizeY")), + _number(pixels.get("PhysicalSizeZ")), + _number(pixels.get("TimeIncrement")) if time_unit else None, + units, + time_unit, + ) + + +def _imagej_sizes(tif: tifffile.TiffFile) -> PhysicalSizes: + meta = tif.imagej_metadata or {} + unit = _imagej_unit(meta.get("unit")) + # ImageJ writes yunit and zunit only when they differ from the X unit. + y_unit = _imagej_unit(meta["yunit"]) if "yunit" in meta else unit + z_unit = _imagej_unit(meta["zunit"]) if "zunit" in meta else unit + page = tif.pages.first + time_unit = _TIME_UNITS.get(_imagej_text(meta.get("tunit", "sec"))) + # Without a unit the resolution tags are a default (often 1 pixel per inch). + return _sizes( + _resolution(page.tags.get("XResolution")) if unit else None, + _resolution(page.tags.get("YResolution")) if y_unit else None, + _number(meta.get("spacing")) if z_unit else None, + _number(meta.get("finterval")) if time_unit else None, + {"X": unit, "Y": y_unit, "Z": z_unit}, + time_unit, + ) + + +def _ngff_unit(value, table: dict[str, str]) -> str | None: + """Return an OME-NGFF axis unit with common spellings as UDUNITS-2 names. + + ``"µm"``, ``"um"`` and ``"micron"`` become ``"micrometer"``, ``"s"`` + becomes ``"second"``; other units are kept as written, and spellings of two + letters or fewer are matched exactly (``"Mm"`` is not ``"mm"``). + """ + if not isinstance(value, str) or not value.strip(): + return None + text = value.strip() + known = table.get(text) or (table.get(text.lower()) if len(text) > 2 else None) + return known or text + + +def _imagej_unit(value) -> str | None: + return _SPACE_UNITS.get(_imagej_text(value)) + + +def _imagej_text(value) -> str: + r"""Return an ImageJ unit in lower case, with its ``\uXXXX`` escapes decoded. + + ImageJ writes non-ASCII characters as ``\u00B5`` in the TIFF description, and + tifffile returns them as written, so ``\u00B5m`` is a micrometre. + """ + text = re.sub(r"\\u([0-9a-fA-F]{4})", lambda m: chr(int(m.group(1), 16)), str(value or "")) + return text.strip().lower() + + +def _resolution(tag) -> float | None: + if tag is None: + return None + try: + numerator, denominator = tag.value + return denominator / numerator if numerator else None + except (TypeError, ValueError): + return None + + +def _number(value) -> float | None: + if isinstance(value, bool): # True would otherwise read as a size of 1 + return None + try: + number = float(value) + except (TypeError, ValueError): + return None + return number if math.isfinite(number) and number > 0 else None diff --git a/src/morphobricks_preprocessing/pixel_sizes.py b/src/morphobricks_preprocessing/pixel_sizes.py deleted file mode 100644 index 8b774227..00000000 --- a/src/morphobricks_preprocessing/pixel_sizes.py +++ /dev/null @@ -1,227 +0,0 @@ -"""Physical pixel sizes from image metadata, without reading pixels. - -TIFF files give them in OME or ImageJ metadata, OME-Zarr images in the scale of -their full-resolution level. Sizes are never guessed: a size without a known -unit is left unknown. They are written into OME-Zarr copies -(:mod:`.zarr_export`) and compared between the images of an experiment in the -folder check (``docs/contract.md`` §11). -""" - -from __future__ import annotations - -import logging -import math -import os -from collections.abc import Sequence -from dataclasses import dataclass -from xml.etree import ElementTree - -import tifffile - -logger = logging.getLogger(__name__) - -# Units in OME and ImageJ metadata, and their OME-NGFF (UDUNITS-2) names. -_SPACE_UNITS = { - "pm": "picometer", - "nm": "nanometer", - "µm": "micrometer", - "μm": "micrometer", - "um": "micrometer", - "micron": "micrometer", - "microns": "micrometer", - "micrometer": "micrometer", - "mm": "millimeter", - "cm": "centimeter", - "m": "meter", -} -_TIME_UNITS = { - "ns": "nanosecond", - "µs": "microsecond", - "μs": "microsecond", - "us": "microsecond", - "ms": "millisecond", - "s": "second", - "sec": "second", - "min": "minute", - "h": "hour", -} - -# Micrometres per OME-NGFF space unit, to compare sizes given in different units. -_MICROMETERS = { - "angstrom": 1e-4, - "picometer": 1e-6, - "nanometer": 1e-3, - "micrometer": 1.0, - "millimeter": 1e3, - "centimeter": 1e4, - "meter": 1e6, -} - - -@dataclass(frozen=True) -class PhysicalSizes: - """Pixel sizes from metadata; None where unknown. - - Attributes - ---------- - x, y, z - Pixel size and z-step in ``space_unit``. - t - Time between frames in ``time_unit``. - space_unit, time_unit - OME-NGFF unit names, e.g. ``"micrometer"`` and ``"second"``. - """ - - x: float | None = None - y: float | None = None - z: float | None = None - t: float | None = None - space_unit: str | None = None - time_unit: str | None = None - - def scale(self, role: str) -> float: - """Return the size of one step along ``role`` (1 when unknown).""" - value = {"X": self.x, "Y": self.y, "Z": self.z, "T": self.t}.get(role) - return 1.0 if value is None or not value > 0 else float(value) - - def unit(self, role: str) -> str | None: - """Return the unit of ``role``, or None when unknown or for channels.""" - if role == "T": - return self.time_unit if self.t is not None else None - if role == "C": - return None - sizes = {"X": self.x, "Y": self.y, "Z": self.z} - return self.space_unit if sizes[role] is not None else None - - def micrometers(self, role: str) -> float | None: - """Return the size along ``"X"``, ``"Y"`` or ``"Z"`` in micrometres. - - None when the size or its unit is unknown, so that sizes in different - units can be compared and an unknown unit is never taken for another. - """ - value = {"X": self.x, "Y": self.y, "Z": self.z}[role] - factor = _MICROMETERS.get(self.space_unit or "") - if value is None or factor is None: - return None - return value * factor - - -def read_physical_sizes(path: str | os.PathLike) -> PhysicalSizes: - """Read pixel sizes from OME or ImageJ metadata, without reading pixels. - - Returns all-None sizes when the file has no such metadata or it cannot be - parsed; sizes are never guessed. - """ - try: - with tifffile.TiffFile(path) as tif: - return tiff_physical_sizes(tif) - except Exception: # metadata is optional; a damaged header is reported elsewhere - logger.debug("Cannot read physical sizes of %s", path, exc_info=True) - return PhysicalSizes() - - -def tiff_physical_sizes(tif: tifffile.TiffFile) -> PhysicalSizes: - """Return the pixel sizes of an open TIFF file (see :func:`read_physical_sizes`).""" - try: - if tif.is_ome and tif.ome_metadata: - return _ome_sizes(tif.ome_metadata) - if tif.is_imagej: - return _imagej_sizes(tif) - except Exception: # metadata is optional; never let it stop reading the header - logger.debug("Cannot read physical sizes of %s", tif.filename, exc_info=True) - return PhysicalSizes() - - -def ngff_physical_sizes( - axes: str, scale: Sequence[float | None], units: Sequence[str | None] -) -> PhysicalSizes: - """Return the pixel sizes of an OME-Zarr image from its level-0 scale. - - Parameters - ---------- - axes - One letter per dimension (:func:`.zarr_input.ngff_axes`). - scale - The scale of each dimension at level 0; empty when not given. - units - The unit of each dimension from the axes metadata; empty when not given. - A space size without a unit is unknown: OME-NGFF writers put 1 there - when they do not know the size. - """ - - def size(letter: str) -> tuple[float | None, str | None]: - if letter not in axes or len(scale) != len(axes): - return None, None - index = axes.index(letter) - unit = units[index] if index < len(units) else None - if unit is None: - return None, None - return _number(scale[index]), unit - - (x, x_unit), (y, _), (z, z_unit) = size("X"), size("Y"), size("Z") - t, t_unit = size("T") - space_unit = x_unit or z_unit - space_unit = _SPACE_UNITS.get(space_unit, space_unit) - return PhysicalSizes( - x=x, - y=y, - z=z, - t=t, - space_unit=space_unit, - time_unit=_TIME_UNITS.get(t_unit, t_unit) if t is not None else None, - ) - - -def _ome_sizes(xml: str) -> PhysicalSizes: - root = ElementTree.fromstring(xml) - pixels = next((e for e in root.iter() if e.tag.rsplit("}", 1)[-1] == "Pixels"), None) - if pixels is None: - return PhysicalSizes() - unit = _SPACE_UNITS.get(pixels.get("PhysicalSizeXUnit", "µm")) - time_unit = _TIME_UNITS.get(pixels.get("TimeIncrementUnit", "s")) - return PhysicalSizes( - x=_number(pixels.get("PhysicalSizeX")), - y=_number(pixels.get("PhysicalSizeY")), - z=_number(pixels.get("PhysicalSizeZ")), - t=_number(pixels.get("TimeIncrement")) if time_unit else None, - space_unit=unit, - time_unit=time_unit, - ) - - -def _imagej_sizes(tif: tifffile.TiffFile) -> PhysicalSizes: - meta = tif.imagej_metadata or {} - unit = _SPACE_UNITS.get(str(meta.get("unit", "")).strip().lower()) - page = tif.pages.first - x = _resolution(page.tags.get("XResolution")) - y = _resolution(page.tags.get("YResolution")) - time_unit = _TIME_UNITS.get(str(meta.get("tunit", "sec")).strip().lower()) - if unit is None: - # Without a unit the resolution tags are a default (often 1 pixel per inch). - x = y = None - return PhysicalSizes( - x=x, - y=y, - z=_number(meta.get("spacing")) if unit else None, - t=_number(meta.get("finterval")) if time_unit else None, - space_unit=unit, - time_unit=time_unit, - ) - - -def _resolution(tag) -> float | None: - if tag is None: - return None - try: - numerator, denominator = tag.value - return denominator / numerator if numerator else None - except (TypeError, ValueError): - return None - - -def _number(value) -> float | None: - try: - number = float(value) - except (TypeError, ValueError): - return None - return number if math.isfinite(number) and number > 0 else None diff --git a/src/morphobricks_preprocessing/run.py b/src/morphobricks_preprocessing/run.py index c343b3e1..4df35ffb 100644 --- a/src/morphobricks_preprocessing/run.py +++ b/src/morphobricks_preprocessing/run.py @@ -73,6 +73,7 @@ from .zarr_export import ( ConversionSettings, OmeZarrWriter, + copy_units_current, is_copy_path, ome_zarr_relative_path, read_physical_sizes, @@ -431,8 +432,16 @@ def stop(**counts) -> RunResult: # During a rebuild there are no rows to check; otherwise the rows must be complete. and (reason is not None or len(previous_records.get(rel, ())) == before.plane_count) # Rows of a converted TIFF are only kept while its copy is still there, as - # written with the same settings. - and (copy is None or ((output / copy).is_dir() and same_conversion)) + # written with the same settings and with the axis units read now (copies + # written before each axis had its own unit are written again). + and ( + copy is None + or ( + (output / copy).is_dir() + and same_conversion + and copy_units_current(output / copy, header.physical_sizes) + ) + ) ): unchanged.append(rel) else: diff --git a/src/morphobricks_preprocessing/zarr_export.py b/src/morphobricks_preprocessing/zarr_export.py index b58fc379..933022b6 100644 --- a/src/morphobricks_preprocessing/zarr_export.py +++ b/src/morphobricks_preprocessing/zarr_export.py @@ -21,6 +21,7 @@ from __future__ import annotations +import json import logging import math import os @@ -33,7 +34,7 @@ from morphobricks_normalization.dtypes import is_integer from .axes import AxisMapping -from .pixel_sizes import ( # noqa: F401 - re-exported for callers of this module +from .physical_sizes import ( # noqa: F401 - moved there, still importable from here PhysicalSizes, read_physical_sizes, ) @@ -498,6 +499,30 @@ def is_copy_path(output_folder: str | os.PathLike, relative_path: str) -> bool: return base in target.parents and target.name.endswith(OME_ZARR_SUFFIX) +def copy_units_current(copy: str | os.PathLike, sizes: PhysicalSizes) -> bool: + """Return True if a copy's axis units are those its source's sizes give now. + + Copies written before each axis had its own unit (plan §12 question 24) may + carry X's unit on Y or Z; such copies are written again on the next run. Only + the copy's metadata file is read. A copy whose metadata cannot be read counts + as out of date, so it is written again rather than trusted. + + Parameters + ---------- + copy + The copy's folder (``/ome_zarr/....ome.zarr``). + sizes + The source image's physical sizes, as :func:`read_physical_sizes` gives them. + """ + try: + attrs = json.loads((Path(copy) / ".zattrs").read_text(encoding="utf-8")) + axes = attrs["multiscales"][0]["axes"] + return all(axis.get("unit") == sizes.unit(str(axis["name"]).upper()) for axis in axes) + except (OSError, ValueError, KeyError, IndexError, TypeError, AttributeError): + logger.debug("Cannot read the axis units of the copy %s", copy, exc_info=True) + return False + + # zarr 2 / 3 --------------------------------------------------------------------------- diff --git a/src/morphobricks_preprocessing/zarr_input.py b/src/morphobricks_preprocessing/zarr_input.py index eb4133f1..0abec6c6 100644 --- a/src/morphobricks_preprocessing/zarr_input.py +++ b/src/morphobricks_preprocessing/zarr_input.py @@ -15,6 +15,9 @@ several fields) is described by its first image and reports how many images it holds, so that the preflight can block it like a multi-series TIFF. +The physical size of a pixel along each axis, with its unit, comes from the +OME-NGFF scale of level 0 (:attr:`NgffImage.physical_sizes`). + The label images of an image (segmentations in its OME-NGFF ``labels`` group, such as ``cells`` or ``nuclei``) can be listed with :func:`ngff_label_names` and opened with :func:`open_ngff_labels`, for later bricks that use them as @@ -26,6 +29,7 @@ from __future__ import annotations +import logging import os from dataclasses import dataclass from pathlib import Path @@ -33,6 +37,10 @@ import numpy as np +from .physical_sizes import PhysicalSizes, ngff_physical_sizes + +logger = logging.getLogger(__name__) + ZARR_SUFFIX = ".zarr" @@ -92,12 +100,12 @@ class NgffImage: Channel labels from the ``omero`` metadata; empty when not given. dataset_path Path of the level-0 array inside the store, e.g. ``"0"`` or ``"0/0"``. - scale - Size of one step along each dimension at level 0, from the - ``coordinateTransformations``; empty when not given. - units - Unit of each dimension from the axes metadata (None where not given); - empty when the axes are not described. + physical_sizes + Level-0 size of one step along X, Y, Z and T with each axis's unit, from + the OME-NGFF ``coordinateTransformations`` scale and axis units + (:func:`.physical_sizes.ngff_physical_sizes`), in the form + :func:`.physical_sizes.read_physical_sizes` gives for TIFF files; None + where the metadata does not give them. """ array: Any @@ -106,8 +114,7 @@ class NgffImage: image_count: int channel_names: tuple[str | None, ...] = () dataset_path: str = "" - scale: tuple[float | None, ...] = () - units: tuple[str | None, ...] = () + physical_sizes: PhysicalSizes = PhysicalSizes() @property def shape(self) -> tuple[int, ...]: @@ -278,50 +285,21 @@ def _level0(group, meta: dict, prefix: str, image_count: int) -> NgffImage: image_count=image_count, channel_names=labels if any(labels) else (), dataset_path=f"{prefix}{dataset_path}", - scale=_level0_scale(first, datasets[0], len(axes)), - units=tuple( - (str(axis["unit"]) if axis.get("unit") else None) if isinstance(axis, dict) else None - for axis in first.get("axes") or [] - ), + physical_sizes=_physical_sizes(first, axes, dataset_path), ) -def _level0_scale(multiscale: dict, dataset: dict, ndim: int) -> tuple[float | None, ...]: - """Return the step size along each dimension at level 0, or ``()`` when not given. +def _physical_sizes(multiscale: dict, axes: str, where: str) -> PhysicalSizes: + """Return the image's physical sizes; empty when its scale metadata cannot be used. - The dataset's scale may be followed by a scale for the whole image - (OME-NGFF 0.4 and 0.5); both apply. + The sizes are for information only, so metadata that cannot be understood + never makes the image unreadable. """ - scale: list[float | None] | None = None - for transforms in ( - dataset.get("coordinateTransformations"), - multiscale.get("coordinateTransformations"), - ): - for transform in transforms if isinstance(transforms, list) else []: - if not isinstance(transform, dict) or transform.get("type") != "scale": - continue - values = transform.get("scale") - if not isinstance(values, list): - continue - if len(values) != ndim: - return () - numbers = [_float(v) for v in values] - scale = ( - numbers - if scale is None - else [ - None if a is None or b is None else a * b - for a, b in zip(scale, numbers, strict=True) - ] - ) - return () if scale is None else tuple(scale) - - -def _float(value: Any) -> float | None: try: - return float(value) - except (TypeError, ValueError): - return None + return ngff_physical_sizes(multiscale, axes) + except Exception: # sizes are optional; never block an image over them + logger.debug("Cannot read the physical sizes of %s", where, exc_info=True) + return PhysicalSizes() def _labels_group(group): diff --git a/tests/test_discovery.py b/tests/test_discovery.py index c99d8735..ecc80018 100644 --- a/tests/test_discovery.py +++ b/tests/test_discovery.py @@ -220,18 +220,92 @@ def test_is_tiff_name(name, expected): assert is_tiff_name(name) is expected +# Folders named single (the MorphoBricks one-member level) ----------------------------- + + +def test_folder_named_single_is_a_level_with_one_member(tmp_path): + # Shaped like an experiment whose conditions have one biological repeat, written + # as a folder named single: /single/. + make_tree( + tmp_path, + [ + "control/single/img_01.tif", + "control/single/img_02.tif", + "treated/single/img_01.tif", + "treated/single/img_02.tif", + ], + ) + result = discover_experiment(tmp_path) + assert result.ok and result.issues == () + assert leaf_labels(result) == { + "control/single": ("control", SINGLE, SINGLE), + "treated/single": ("treated", SINGLE, SINGLE), + } + assert result.labels_for("treated/single/img_02.tif") == ("treated", SINGLE, SINGLE) + + +def test_folder_named_single_gives_the_labels_of_a_missing_level(tmp_path): + # The same labels as images placed directly in the condition folder. + named = discover_experiment(make_tree(tmp_path / "named", ["control/single/a.tif"])) + missing = discover_experiment(make_tree(tmp_path / "missing", ["control/a.tif"])) + assert named.ok and missing.ok + assert named.leaves[0].labels == missing.leaves[0].labels == ("control", SINGLE, SINGLE) + + +@pytest.mark.parametrize( + ("tree", "labels"), + [ + (["single/rep1/a.tif"], ("single", "rep1", SINGLE)), + (["control/single/tech_1/a.tif"], ("control", SINGLE, "tech_1")), + (["control/rep1/single/a.tif"], ("control", "rep1", SINGLE)), + (["single/single/single/a.tif"], (SINGLE, SINGLE, SINGLE)), + ], +) +def test_folder_named_single_at_any_level(tmp_path, tree, labels): + result = discover_experiment(make_tree(tmp_path, tree)) + assert result.ok and result.issues == () + assert [leaf.labels for leaf in result.leaves] == [labels] + + +def test_folder_named_single_ignores_system_folders_beside_it(tmp_path): + make_tree(tmp_path, ["control/single/a.tif", "control/.ipynb_checkpoints/x.txt"]) + (tmp_path / "control" / "__MACOSX").mkdir() + result = discover_experiment(tmp_path) + assert result.ok and result.issues == () + assert leaf_labels(result) == {"control/single": ("control", SINGLE, SINGLE)} + + # Errors ------------------------------------------------------------------------------- -@pytest.mark.parametrize("name", ["single", "Single", "SINGLE"]) -def test_folder_named_single_is_an_error(tmp_path, name): +@pytest.mark.parametrize("name", ["Single", "SINGLE"]) +def test_folder_named_single_in_another_case_is_an_error(tmp_path, name): + # "Single" would be a different label from the "single" written for one-member + # levels, so only the lower-case spelling stands for that value. make_tree(tmp_path, [f"control/{name}/a.tif", "treated/a.tif"]) result = discover_experiment(tmp_path) assert not result.ok found = issue(result, "reserved_folder_name") assert found.severity is Severity.ERROR assert found.paths == (f"control/{name}",) - assert found.action + assert "lower case" in found.action + + +def test_folder_named_single_beside_other_folders_is_an_error(tmp_path): + # "single" says the level has one member; beside rep2 it would have two. + make_tree(tmp_path, ["control/single/a.tif", "control/rep2/a.tif", "treated/a.tif"]) + result = discover_experiment(tmp_path) + assert not result.ok + found = issue(result, "reserved_folder_name") + assert found.severity is Severity.ERROR + assert found.paths == ("control/single",) + assert "only folder" in found.message + + +def test_folder_named_single_with_an_empty_sibling_is_an_error(tmp_path): + make_tree(tmp_path, ["control/single/a.tif", "control/rep2/"]) + result = discover_experiment(tmp_path) + assert issue(result, "reserved_folder_name").paths == ("control/single",) def test_folders_differing_only_in_case_are_an_error(tmp_path): diff --git a/tests/test_metadata.py b/tests/test_metadata.py index d2e2b9aa..019adcf2 100644 --- a/tests/test_metadata.py +++ b/tests/test_metadata.py @@ -5,13 +5,16 @@ import tifffile from morphobricks_preprocessing import ( + PhysicalSizes, Severity, TiffMetadata, check_metadata, discover_experiment, inspect_experiment, + inspect_image, inspect_images, inspect_tiff, + read_physical_sizes, ) from tiff_tree import make_tree, write_file, write_tiff @@ -124,6 +127,44 @@ def test_ome_channel_names_and_significant_bits(tmp_path, no_pixel_reads): assert info.significant_bits == 12 +def test_header_carries_the_physical_sizes(tmp_path, no_pixel_reads): + ome = tmp_path / "a.ome.tif" + tifffile.imwrite( + ome, + np.zeros((3, 8, 8), np.uint16), + ome=True, + photometric="minisblack", + metadata={ + "axes": "ZYX", + "PhysicalSizeX": 0.2, + "PhysicalSizeXUnit": "µm", + "PhysicalSizeY": 0.3, + "PhysicalSizeYUnit": "nm", + "PhysicalSizeZ": 1.5, + }, + ) + info = inspect_image(ome) + assert info.physical_sizes == read_physical_sizes(ome) + assert (info.physical_sizes.x, info.physical_sizes.y, info.physical_sizes.z) == (0.2, 0.3, 1.5) + assert info.physical_sizes.unit("Y") == "nanometer" + imagej = tmp_path / "b.tif" + tifffile.imwrite( + imagej, + np.zeros((8, 8), np.uint16), + imagej=True, + resolution=(1 / 0.25, 1 / 0.25), + metadata={"unit": "um"}, + ) + sizes = inspect_image(imagej).physical_sizes + assert (sizes.x, sizes.y, sizes.unit("X")) == (0.25, 0.25, "micrometer") + + +def test_header_without_sizes_has_empty_physical_sizes(tmp_path): + assert inspect_tiff(write_tiff(tmp_path / "a.tif")).physical_sizes == PhysicalSizes() + bad = inspect_tiff(write_file(tmp_path / "bad.tif", b"not a tiff")) + assert not bad.readable and bad.physical_sizes == PhysicalSizes() + + def test_no_channel_names_without_ome_metadata(tmp_path): info = inspect_tiff(write_tiff(tmp_path / "a.tif", (2, 6, 7), axes="CYX", imagej=True)) assert info.channel_names == () and info.significant_bits is None diff --git a/tests/test_notebook_ui.py b/tests/test_notebook_ui.py index 7628cfe9..f9502d98 100644 --- a/tests/test_notebook_ui.py +++ b/tests/test_notebook_ui.py @@ -612,6 +612,110 @@ def test_conflicting_axes_explain_why_no_roles_are_offered(tmp_path): assert "check the experiment folder and fix any errors" in nb.readiness.value +# Steps 1 and 2 embedded in another brick --------------------------------------------------- + + +def page_parts(page): + """Return the title, the purpose and every widget below them in a step's page.""" + root = page.children[1] + found = [] + + def walk(widget): + found.append(widget) + for child in getattr(widget, "children", ()): + walk(child) + + for child in root.children[2:]: + walk(child) + return root.children[0].value, root.children[1].value, found + + +def intros(widgets_found): + return [w.value for w in widgets_found if "mb-intro" in str(getattr(w, "value", ""))] + + +def test_steps_keep_their_words_by_default(): + nb = PreprocessingNotebook() + title, purpose, found = page_parts(nb.folder_step()) + assert "Step 1 · Check the experiment folder" in title + assert "Choose the folder with your images." in purpose + assert len(intros(found)) == 1 and "What happens here" in intros(found)[0] + assert nb.output_card in found and nb.folder_next in found + assert ui.is_visible(nb.folder_next) + title, purpose, found = page_parts(nb.axis_step()) + assert "Step 2 · Review the axis roles" in title + assert "Tell the brick which image dimension" in purpose + assert len(intros(found)) == 1 and "flat 2D planes" in intros(found)[0] + assert nb.axis_box in found and ui.is_visible(nb.axis_next) + + +def test_folder_step_with_its_own_words_and_no_output_card(experiment): + nb = PreprocessingNotebook(str(experiment)) + title, purpose, found = page_parts( + nb.folder_step( + "Step 1 · Your data", + "Choose the folder with your images.", + intro="Here your folders are read.", + show_output=False, + show_next=False, + ) + ) + assert "Step 1 · Your data" in title and "Step 1 · Check" not in title + assert purpose == "
Choose the folder with your images.
" + assert intros(found) == ["
Here your folders are read.
"] + assert not ui.is_visible(nb.output_card) + assert nb.folder_card in found and nb.check_button in found and nb.labels_card in found + # The check works without the card, with the default output folder. + nb.check_button.click() + assert nb.preflight is not None and nb.preflight.ok + nb.labels_confirmed.button.click() + confirm_repeats(nb) # embedded steps keep the experiment-design gating (P3) + assert "Go to Step 2" in nb.folder_next.value and not ui.is_visible(nb.folder_next) + + +def test_hidden_output_card_comes_back_when_its_folder_is_inside_the_experiment(experiment): + nb = PreprocessingNotebook(str(experiment), output_folder=str(experiment / "out")) + page = nb.folder_step(show_output=False) + assert nb.output_card in page_parts(page)[2] and not ui.is_visible(nb.output_card) + pre = nb.check_folder() + assert "output_inside_experiment" in [i.code for i in pre.issues] + assert ui.is_visible(nb.output_card) # the box to fix it is shown + nb.output.value = "" + assert nb.check_folder().ok + assert not ui.is_visible(nb.output_card) + nb.folder_step() # the default page shows it again + assert ui.is_visible(nb.output_card) + + +def test_steps_without_an_intro_box(): + nb = PreprocessingNotebook() + assert intros(page_parts(nb.folder_step(intro=""))[2]) == [] + assert intros(page_parts(nb.axis_step(intro=""))[2]) == [] + + +def test_axis_step_with_its_own_words(experiment): + nb = PreprocessingNotebook(str(experiment)) + nb.folder_step() + title, purpose, found = page_parts( + nb.axis_step("Step 2 · Image dimensions", "Say which dimension is which.", show_next=False) + ) + assert "Step 2 · Image dimensions" in title and purpose.endswith( + "Say which dimension is which." + ) + assert len(intros(found)) == 1 # the default intro is kept + nb.check_button.click() + nb.labels_confirmed.button.click() + confirm_repeats(nb) # embedded steps keep the experiment-design gating (P3) + nb.preview_button.click() + nb.preview_checked.button.click() + nb.confirm_button.click() + assert nb.confirmed_mapping is not None + assert "compute the histograms" in nb.axis_next.value and not ui.is_visible(nb.axis_next) + # Building the default page again shows the box again. + nb.axis_step() + assert ui.is_visible(nb.axis_next) + + # Step 3 ----------------------------------------------------------------------------------- diff --git a/tests/test_physical_sizes.py b/tests/test_physical_sizes.py new file mode 100644 index 00000000..1d0c2110 --- /dev/null +++ b/tests/test_physical_sizes.py @@ -0,0 +1,241 @@ +import pickle +from dataclasses import replace + +import numpy as np +import pytest +import tifffile + +from morphobricks_preprocessing import PhysicalSizes, read_physical_sizes, zarr_export +from morphobricks_preprocessing import physical_sizes as physical_sizes_module + +# PhysicalSizes(x=0.1, y=0.2, z=0.5, space_unit="micrometer") pickled before per-axis units. +OLD_PICKLE = bytes.fromhex( + "80049597000000000000008c266d6f7270686f627269636b735f70726570726f63657373696e672e7a617272" + "5f6578706f7274948c0d506879736963616c53697a65739493942981947d94288c017894473fb99999999999" + "9a8c017994473fc999999999999a8c017a94473fe00000000000008c0174944e8c0a73706163655f756e6974" + "948c0a6d6963726f6d65746572948c0974696d655f756e6974944e75622e" +) + + +def write_ome(path, **sizes): + tifffile.imwrite( + path, + np.zeros((3, 8, 8), np.uint16), + ome=True, + photometric="minisblack", + metadata={"axes": "ZYX", **sizes}, + ) + return path + + +# Units per axis ------------------------------------------------------------------------ + + +def test_ome_units_are_read_per_axis(tmp_path): + path = write_ome( + tmp_path / "a.ome.tif", + PhysicalSizeX=120.0, + PhysicalSizeXUnit="nm", + PhysicalSizeY=0.2, + PhysicalSizeYUnit="µm", + PhysicalSizeZ=2.0, + PhysicalSizeZUnit="mm", + ) + sizes = read_physical_sizes(path) + assert (sizes.x, sizes.y, sizes.z) == (120.0, 0.2, 2.0) + assert [sizes.unit(role) for role in "XYZ"] == ["nanometer", "micrometer", "millimeter"] + assert (sizes.x_unit, sizes.y_unit, sizes.z_unit) == ("nanometer", "micrometer", "millimeter") + assert sizes.space_unit == "nanometer" # the unit of X, as before + + +def test_ome_units_left_out_are_micrometres(tmp_path): + # The OME schema's default unit: Y and Z do not take X's unit. + path = write_ome( + tmp_path / "a.ome.tif", + PhysicalSizeX=120.0, + PhysicalSizeXUnit="nm", + PhysicalSizeY=0.5, + PhysicalSizeZ=2.0, + ) + sizes = read_physical_sizes(path) + assert [sizes.unit(role) for role in "XYZ"] == ["nanometer", "micrometer", "micrometer"] + + +def test_imagej_y_and_z_units(tmp_path): + path = tmp_path / "a.tif" + tifffile.imwrite( + path, + np.zeros((3, 8, 8), np.uint16), + imagej=True, + resolution=(1 / 0.25, 1 / 0.5), + metadata={"axes": "ZYX", "unit": "um", "yunit": "nm", "zunit": "mm", "spacing": 1.5}, + ) + sizes = read_physical_sizes(path) + assert (sizes.x, sizes.y, sizes.z) == (0.25, 0.5, 1.5) + assert [sizes.unit(role) for role in "XYZ"] == ["micrometer", "nanometer", "millimeter"] + + +def test_imagej_without_y_and_z_units_uses_the_unit_for_all(tmp_path): + path = tmp_path / "a.tif" + tifffile.imwrite( + path, + np.zeros((3, 8, 8), np.uint16), + imagej=True, + resolution=(1 / 0.25, 1 / 0.25), + metadata={"axes": "ZYX", "unit": "um", "spacing": 1.5}, + ) + sizes = read_physical_sizes(path) + assert [sizes.unit(role) for role in "XYZ"] == ["micrometer"] * 3 + + +# The dataclass, for callers written before per-axis units ------------------------------- + + +def test_space_unit_alone_is_the_unit_of_every_space_axis(): + sizes = PhysicalSizes(0.5, 0.5, 2.0, None, "micrometer") + assert [sizes.unit(role) for role in "XYZ"] == ["micrometer"] * 3 + assert sizes == PhysicalSizes( + x=0.5, + y=0.5, + z=2.0, + space_unit="micrometer", + x_unit="micrometer", + y_unit="micrometer", + z_unit="micrometer", + ) + + +def test_per_axis_units_fall_back_to_space_unit(): + sizes = PhysicalSizes(x=1.0, y=2.0, x_unit="nanometer", y_unit="micrometer") + assert (sizes.unit("X"), sizes.unit("Y")) == ("nanometer", "micrometer") + # An axis given no unit has space_unit, here none, even when X has one. + assert PhysicalSizes(x=1.0, y=2.0, x_unit="nanometer", z_unit="meter").unit("Y") is None + partial = PhysicalSizes(y=1.0, z=1.0, space_unit="micrometer", y_unit="nanometer") + assert (partial.unit("Y"), partial.unit("Z")) == ("nanometer", "micrometer") + # An axis's own unit wins over space_unit. + assert PhysicalSizes(x=1.0, space_unit="micrometer", x_unit="nanometer").unit("X") == ( + "nanometer" + ) + + +def test_replace_keeps_units_consistent(): + sizes = PhysicalSizes(x=0.1, y=0.1, space_unit="micrometer") + assert replace(sizes, space_unit="nanometer").unit("Y") == "nanometer" + assert replace(sizes, x_unit="nanometer").unit("X") == "nanometer" + assert replace(sizes, x_unit="nanometer").unit("Y") == "micrometer" + assert replace(sizes, space_unit=None).unit("X") is None + + +def test_sizes_pickled_before_per_axis_units_keep_their_unit(): + # PhysicalSizes(x=0.1, y=0.2, z=0.5, space_unit="micrometer") pickled by the version + # before per-axis units (as morphobricks_preprocessing.zarr_export.PhysicalSizes). + sizes = pickle.loads(OLD_PICKLE) + assert [sizes.unit(role) for role in "XYZ"] == ["micrometer"] * 3 + assert sizes == PhysicalSizes(x=0.1, y=0.2, z=0.5, space_unit="micrometer") + assert pickle.loads(pickle.dumps(sizes)) == sizes + + +def test_equal_sizes_hash_alike(): + given = PhysicalSizes(x=0.1, space_unit="micrometer") + spelled_out = PhysicalSizes( + x=0.1, space_unit="micrometer", x_unit="micrometer", y_unit="micrometer" + ) + assert given == spelled_out and len({given, spelled_out}) == 1 + assert given != PhysicalSizes(x=0.1, space_unit="micrometer", y_unit="nanometer") + assert given != "not sizes" + + +def test_units_of_unknown_sizes_and_channels_are_none(): + sizes = PhysicalSizes(x=1.0, space_unit="micrometer", time_unit="second") + assert sizes.unit("Y") is None and sizes.unit("T") is None and sizes.unit("C") is None + assert sizes.scale("Y") == 1.0 + + +def test_still_importable_from_zarr_export(): + assert zarr_export.PhysicalSizes is physical_sizes_module.PhysicalSizes is PhysicalSizes + assert zarr_export.read_physical_sizes is read_physical_sizes + + +# Unit spellings ------------------------------------------------------------------------ + + +@pytest.mark.parametrize("symbol", ["Å", "Å"]) +def test_ome_angstroms(tmp_path, symbol): + path = write_ome( + tmp_path / "a.ome.tif", + PhysicalSizeX=5.0, + PhysicalSizeXUnit=symbol, + PhysicalSizeY=5.0, + PhysicalSizeYUnit=symbol, + ) + sizes = read_physical_sizes(path) + assert (sizes.unit("X"), sizes.unit("Y")) == ("angstrom", "angstrom") + + +def test_imagej_escaped_micro_sign(tmp_path): + # ImageJ writes the micro sign as µ in the description; tifffile keeps the text. + path = tmp_path / "a.tif" + tifffile.imwrite( + path, + np.zeros((3, 8, 8), np.uint16), + imagej=True, + resolution=(1 / 0.25, 1 / 0.5), + metadata={ + "axes": "ZYX", + "unit": "\\u00B5m", + "yunit": "\\u00B5m", + "zunit": "\\u00B5m", + "spacing": 1.5, + }, + ) + assert tifffile.TiffFile(path).imagej_metadata["unit"] == "\\u00B5m" + sizes = read_physical_sizes(path) + assert (sizes.x, sizes.y, sizes.z) == (0.25, 0.5, 1.5) + assert [sizes.unit(role) for role in "XYZ"] == ["micrometer"] * 3 + + +def test_ngff_unit_spellings_become_udunits_names(): + multiscale = { + "axes": [ + {"name": "t", "type": "time", "unit": "s"}, + {"name": "z", "type": "space", "unit": "µm"}, + {"name": "y", "type": "space", "unit": "Micron"}, + {"name": "x", "type": "space", "unit": "um"}, + ], + "datasets": [ + {"path": "0", "coordinateTransformations": [{"type": "scale", "scale": [1, 2, 3, 4]}]} + ], + } + sizes = physical_sizes_module.ngff_physical_sizes(multiscale, "TZYX") + assert [sizes.unit(role) for role in "TZYX"] == ["second"] + ["micrometer"] * 3 + # Other units are kept as written; short spellings are matched exactly. + multiscale["axes"][1]["unit"] = "kilometer" + multiscale["axes"][2]["unit"] = "Mm" + sizes = physical_sizes_module.ngff_physical_sizes(multiscale, "TZYX") + assert (sizes.unit("Z"), sizes.unit("Y")) == ("kilometer", "Mm") + + +def test_imagej_boolean_values_are_not_sizes(tmp_path): + # tifffile reads "spacing=true" as True, which float() would turn into 1. + path = tmp_path / "a.tif" + tifffile.imwrite( + path, + np.zeros((3, 8, 8), np.uint16), + imagej=True, + resolution=(1 / 0.25, 1 / 0.25), + metadata={"axes": "ZYX", "unit": "um", "spacing": True, "finterval": True}, + ) + assert tifffile.TiffFile(path).imagej_metadata["spacing"] is True + sizes = read_physical_sizes(path) + assert (sizes.x, sizes.z, sizes.t) == (0.25, None, None) + + +def test_ngff_boolean_scales_are_not_sizes(): + multiscale = { + "axes": [{"name": "y", "type": "space"}, {"name": "x", "type": "space"}], + "datasets": [ + {"path": "0", "coordinateTransformations": [{"type": "scale", "scale": [True, 0.5]}]} + ], + } + sizes = physical_sizes_module.ngff_physical_sizes(multiscale, "YX") + assert (sizes.y, sizes.x) == (None, 0.5) diff --git a/tests/test_run.py b/tests/test_run.py index b06564cc..e18f52f5 100644 --- a/tests/test_run.py +++ b/tests/test_run.py @@ -236,6 +236,27 @@ def test_every_row_has_its_position_id(experiment): assert all("position_id" in f for f in data["files"]) +def test_folder_named_single_runs_as_a_one_member_level(tmp_path): + # /single/: the repeat level exists with one member. + root = tmp_path / "exp" + for seed, rel in enumerate(["ctrl/single/a.tif", "ctrl/single/b.tif", "drug/single/a.tif"]): + write_data(root / rel, stack(seed), "ZYX") + assert preflight(root).ok + records = read_plane_histograms(run_experiment(root, ZYX).csv_path) + assert { + (r.condition, r.biological_repeat, r.technical_repeat, r.position_id) for r in records + } == { + ("ctrl", SINGLE, SINGLE, "a"), + ("ctrl", SINGLE, SINGLE, "b"), + ("drug", SINGLE, SINGLE, "a"), + } + assert {r.source_relative_path for r in records} == { + "ctrl/single/a.tif", + "ctrl/single/b.tif", + "drug/single/a.tif", + } + + def test_position_ids_that_are_text_stay_text(tmp_path): root = tmp_path / "exp" for seed, rel in enumerate(["ctrl/001.tif", "ctrl/NA.tif", "drug/TRUE.tif"]): diff --git a/tests/test_zarr_export.py b/tests/test_zarr_export.py index 705f8f3f..52e8930c 100644 --- a/tests/test_zarr_export.py +++ b/tests/test_zarr_export.py @@ -196,8 +196,17 @@ def test_physical_sizes_from_ome(tmp_path): "TimeIncrementUnit": "s", }, ) + # PhysicalSizeZUnit is left out, so Z is in the OME default, micrometres. assert read_physical_sizes(path) == PhysicalSizes( - x=120.0, y=120.0, z=500.0, t=30.0, space_unit="nanometer", time_unit="second" + x=120.0, + y=120.0, + z=500.0, + t=30.0, + space_unit="nanometer", + time_unit="second", + x_unit="nanometer", + y_unit="nanometer", + z_unit="micrometer", ) @@ -307,6 +316,31 @@ def test_every_dtype_is_copied_unchanged(tmp_path, dtype): np.testing.assert_array_equal(open_copy(result.output_folder, "a.tif")["0"][:], data) +@needs_zarr +def test_copies_give_each_axis_its_own_unit(tmp_path): + path = tmp_path / "exp/a.ome.tif" + path.parent.mkdir(parents=True) + tifffile.imwrite( + path, + random_data((2, 8, 8), "uint16"), + ome=True, + photometric="minisblack", + metadata={ + "axes": "ZYX", + "PhysicalSizeX": 120.0, + "PhysicalSizeXUnit": "nm", + "PhysicalSizeY": 0.25, + "PhysicalSizeYUnit": "µm", + "PhysicalSizeZ": 2.0, + "PhysicalSizeZUnit": "mm", + }, + ) + result = run_experiment(tmp_path / "exp", ZYX, convert_tiffs=True) + meta = multiscale(open_copy(result.output_folder, "a.ome.tif")) + assert [axis["unit"] for axis in meta["axes"]] == ["millimeter", "micrometer", "nanometer"] + assert meta["datasets"][0]["coordinateTransformations"][0]["scale"] == [2.0, 0.25, 120.0] + + @needs_zarr def test_pyramid_levels_and_scales(tmp_path): data = random_data((2, 1100, 700), "uint16") @@ -703,6 +737,44 @@ def test_copies_follow_the_conversion_settings(tmp_path, settings, compressor): assert read_manifest(result.manifest_path).settings.conversion == custom +@needs_zarr +def test_copies_with_outdated_axis_units_are_written_again(tmp_path): + root = tmp_path / "exp" + for name in ("a.ome.tif", "b.ome.tif"): + (root / "ctrl").mkdir(parents=True, exist_ok=True) + tifffile.imwrite( + root / "ctrl" / name, + random_data((2, 8, 8), "uint8"), + ome=True, + photometric="minisblack", + metadata={ + "axes": "ZYX", + "PhysicalSizeX": 120.0, + "PhysicalSizeXUnit": "nm", + "PhysicalSizeY": 0.25, + "PhysicalSizeYUnit": "µm", + "PhysicalSizeZ": 2.0, + }, + ) + first = run_experiment(root, ZYX, convert_tiffs=True) + table = first.csv_path.read_bytes() + # Make a.ome.tif's copy look like one written before per-axis units: X's unit + # on every space axis. + attrs_path = first.output_folder / ome_zarr_relative_path("ctrl/a.ome.tif") / ".zattrs" + attrs = json.loads(attrs_path.read_text(encoding="utf-8")) + for axis in attrs["multiscales"][0]["axes"]: + axis["unit"] = "nanometer" + attrs_path.write_text(json.dumps(attrs), encoding="utf-8") + second = run_experiment(root, ZYX, convert_tiffs=True) + assert second.converted == ("ctrl/a.ome.tif",) + assert second.csv_path.read_bytes() == table # same pixels, same rows + units = [ + a["unit"] for a in multiscale(open_copy(second.output_folder, "ctrl/a.ome.tif"))["axes"] + ] + assert units == ["micrometer", "micrometer", "nanometer"] + assert run_experiment(root, ZYX, convert_tiffs=True).converted == () + + @needs_zarr def test_changed_conversion_settings_rewrite_the_copies(experiment): first = run_experiment(experiment, ZYX, convert_tiffs=True) diff --git a/tests/test_zarr_input.py b/tests/test_zarr_input.py index 2387b247..28e6a839 100644 --- a/tests/test_zarr_input.py +++ b/tests/test_zarr_input.py @@ -7,6 +7,7 @@ from morphobricks_preprocessing import ( AxisMapping, NgffError, + PhysicalSizes, Severity, TiffMetadata, declared_mapping, @@ -15,11 +16,13 @@ inspect_image, inspect_images, iter_planes, + ome_zarr_relative_path, open_ngff, plane_layout, prefill_mapping, preflight, read_manifest, + read_physical_sizes, read_plane, read_plane_histograms, read_sample_plane, @@ -31,6 +34,7 @@ from morphobricks_preprocessing import run as run_module from morphobricks_preprocessing.cli import main from morphobricks_preprocessing.histogram import compute_plane_histogram +from morphobricks_preprocessing.zarr_export import ConversionSettings from morphobricks_preprocessing.zarr_input import ngff_axes from tiff_tree import write_data, write_file from zarr_tree import write_bioformats2raw, write_ngff, write_plate, zarr_major @@ -108,6 +112,167 @@ def test_open_ngff(tmp_path, version): assert image.chunks == (1, 1, 8, 9) +# Physical sizes (OME-NGFF coordinateTransformations and axis units) -------------------- + +MICRONS = {"z": "micrometer", "y": "micrometer", "x": "micrometer"} + + +@pytest.mark.parametrize("version", ["0.4", "0.5"]) +def test_physical_sizes_with_units(tmp_path, version): + path = write_ngff( + tmp_path / "a.ome.zarr", + np.zeros((2, 3, 8, 8), np.uint8), + "czyx", + version=version, + scale=[1.0, 2.0, 0.5, 0.25], + units={**MICRONS, "y": "nanometer"}, + ) + sizes = open_ngff(path).physical_sizes + assert sizes == PhysicalSizes( + x=0.25, + y=0.5, + z=2.0, + space_unit="micrometer", + x_unit="micrometer", + y_unit="nanometer", + z_unit="micrometer", + ) + assert [sizes.unit(role) for role in "XYZCT"] == [ + "micrometer", + "nanometer", + "micrometer", + None, + None, + ] + + +@pytest.mark.parametrize("version", ["0.4", "0.5"]) +def test_physical_sizes_without_units(tmp_path, version): + path = write_ngff( + tmp_path / "a.ome.zarr", + np.zeros((3, 8, 8), np.uint8), + "zyx", + version=version, + scale=[2.0, 0.5, 0.25], + ) + sizes = open_ngff(path).physical_sizes + assert (sizes.x, sizes.y, sizes.z, sizes.t) == (0.25, 0.5, 2.0, None) + assert [sizes.unit(role) for role in "XYZ"] == [None, None, None] + assert sizes.space_unit is None + + +@pytest.mark.parametrize("version", ["0.4", "0.5"]) +def test_physical_sizes_with_a_multiscale_level_scale(tmp_path, version): + # OME-NGFF applies the multiscales entry's transformations after the dataset's. + path = write_ngff( + tmp_path / "a.ome.zarr", + np.zeros((4, 3, 8, 8), np.uint8), + "tzyx", + version=version, + scale=[1.0, 1.5, 0.2, 0.25], + units={**MICRONS, "t": "second"}, + multiscale_transforms=[{"type": "scale", "scale": [30.0, 2.0, 3.0, 4.0]}], + ) + sizes = open_ngff(path).physical_sizes + assert (sizes.t, sizes.z, sizes.y, sizes.x) == pytest.approx((30.0, 3.0, 0.6, 1.0)) + assert (sizes.unit("T"), sizes.unit("X")) == ("second", "micrometer") + + +def test_multiscale_level_translation_does_not_change_sizes(tmp_path): + path = write_ngff( + tmp_path / "a.ome.zarr", + np.zeros((8, 8), np.uint8), + "yx", + scale=[0.5, 0.25], + multiscale_transforms=[{"type": "translation", "translation": [10.0, 20.0]}], + ) + sizes = open_ngff(path).physical_sizes + assert (sizes.x, sizes.y) == (0.25, 0.5) + + +@pytest.mark.parametrize( + "transforms", + [ + [], # no coordinateTransformations at all (OME-NGFF before 0.4) + [{"type": "scale", "path": "scale.bin"}], # stored in a file: not read + [{"type": "scale", "scale": [0.5, 0.25]}], # not one number per axis + [{"type": "translation", "translation": [0.0, 0.0, 0.0]}], + ], +) +def test_physical_sizes_absent_when_not_given(tmp_path, transforms): + path = write_ngff( + tmp_path / "a.ome.zarr", + np.zeros((3, 8, 8), np.uint8), + "zyx", + dataset_transforms=transforms, + units=MICRONS, + ) + assert open_ngff(path).physical_sizes == PhysicalSizes() + + +def test_unusable_multiscale_level_scale_makes_sizes_unknown(tmp_path): + path = write_ngff( + tmp_path / "a.ome.zarr", + np.zeros((8, 8), np.uint8), + "yx", + scale=[0.5, 0.25], + multiscale_transforms=[{"type": "scale", "path": "scale.bin"}], + ) + assert open_ngff(path).physical_sizes == PhysicalSizes() + + +def test_scales_that_are_not_positive_numbers_are_unknown(tmp_path): + path = write_ngff( + tmp_path / "a.ome.zarr", + np.zeros((3, 8, 8), np.uint8), + "zyx", + scale=[0.0, -1.0, 0.25], + units=MICRONS, + ) + sizes = open_ngff(path).physical_sizes + assert (sizes.x, sizes.y, sizes.z) == (0.25, None, None) + assert (sizes.unit("X"), sizes.unit("Y")) == ("micrometer", None) + + +@pytest.mark.parametrize("version", ["0.4", "0.5"]) +def test_header_of_a_store_carries_its_physical_sizes(tmp_path, version): + path = write_ngff( + tmp_path / "a.ome.zarr", + np.zeros((3, 8, 8), np.uint8), + "zyx", + version=version, + scale=[2.0, 0.5, 0.25], + units=MICRONS, + ) + info = inspect_image(path) + assert info.is_ome_zarr and info.physical_sizes == open_ngff(path).physical_sizes + assert (info.physical_sizes.x, info.physical_sizes.unit("Z")) == (0.25, "micrometer") + + +def test_converted_copies_give_the_sizes_of_their_tiff(tmp_path): + source = tmp_path / "exp/a.ome.tif" + source.parent.mkdir(parents=True) + tifffile.imwrite( + source, + np.zeros((2, 3, 8, 8), np.uint16), + ome=True, + photometric="minisblack", + metadata={ + "axes": "TZYX", + "PhysicalSizeX": 120.0, + "PhysicalSizeXUnit": "nm", + "PhysicalSizeY": 0.25, + "PhysicalSizeYUnit": "µm", + "PhysicalSizeZ": 2.0, + "TimeIncrement": 30.0, + "TimeIncrementUnit": "s", + }, + ) + result = run_experiment(tmp_path / "exp", prefill_mapping("TZYX"), convert_tiffs=True) + copy = result.output_folder / ome_zarr_relative_path("a.ome.tif") + assert open_ngff(copy).physical_sizes == read_physical_sizes(source) + + def test_bioformats2raw_layout(tmp_path): one = write_bioformats2raw(tmp_path / "one.ome.zarr", [random_data((2, 8, 8))]) image = open_ngff(one) @@ -597,6 +762,73 @@ def test_tiff_reading_is_unchanged_next_to_stores(tmp_path): assert len(list(iter_planes(path, prefill_mapping("ZYX")))) == 2 +# Physical sizes never block an image ----------------------------------------------------- + + +def test_scale_too_large_for_a_float_is_unknown_and_does_not_block(tmp_path): + root = tmp_path / "exp" + path = write_ngff( + root / "ctrl/a.ome.zarr", + np.zeros((2, 8, 8), np.uint8), + "zyx", + scale=[10**400, 0.5, 0.25], + units={"z": "micrometer", "y": "micrometer", "x": "micrometer"}, + ) + sizes = open_ngff(path).physical_sizes + assert (sizes.z, sizes.y, sizes.x) == (None, 0.5, 0.25) + assert preflight(root, tmp_path / "out").ok + + +def test_unexpected_scale_metadata_does_not_block_the_image(tmp_path, monkeypatch): + def broken(*args, **kwargs): + raise RuntimeError("not understood") + + monkeypatch.setattr(zarr_input, "ngff_physical_sizes", broken) + root = tmp_path / "exp" + path = write_ngff(root / "ctrl/a.ome.zarr", np.zeros((2, 8, 8), np.uint8), "zyx") + assert open_ngff(path).physical_sizes == PhysicalSizes() + assert preflight(root, tmp_path / "out").ok + + +@pytest.mark.parametrize("where", ["dataset", "multiscale"]) +def test_transformations_that_are_not_a_list_give_no_sizes(tmp_path, where): + malformed = {"type": "scale", "scale": [2.0, 0.5, 0.25]} + kwargs = ( + {"dataset_transforms": malformed} + if where == "dataset" + else {"scale": [2.0, 0.5, 0.25], "multiscale_transforms": malformed} + ) + path = write_ngff(tmp_path / "a.ome.zarr", np.zeros((3, 8, 8), np.uint8), "zyx", **kwargs) + assert open_ngff(path).physical_sizes == PhysicalSizes() + + +def test_sizes_come_from_level_0_of_a_pyramid(tmp_path): + # The brick's own copy has three levels with scales 0.5, 1 and 2 along X and Y. + source = tmp_path / "exp/a.ome.tif" + source.parent.mkdir(parents=True) + tifffile.imwrite( + source, + np.zeros((2, 64, 64), np.uint16), + ome=True, + photometric="minisblack", + metadata={"axes": "ZYX", "PhysicalSizeX": 0.5, "PhysicalSizeY": 0.5, "PhysicalSizeZ": 2}, + ) + result = run_experiment( + tmp_path / "exp", + prefill_mapping("ZYX"), + convert_tiffs=True, + conversion=ConversionSettings(chunk_size=16, pyramid_levels=2), + ) + copy = result.output_folder / ome_zarr_relative_path("a.ome.tif") + image = open_ngff(copy) + assert image.dataset_path == "0" and image.shape == (2, 64, 64) + sizes = image.physical_sizes + assert (sizes.x, sizes.y, sizes.z) == (0.5, 0.5, 2.0) + + +# Pixel sizes compared in the folder check ------------------------------------------------- + + def with_scale(path, scale, units, image_scale=None): """Give the OME-Zarr image at ``path`` a level-0 scale and axis units.""" import zarr @@ -615,9 +847,7 @@ def with_scale(path, scale, units, image_scale=None): def test_ome_zarr_pixel_sizes_come_from_the_scale(tmp_path): path = write_ngff(tmp_path / "a.ome.zarr", np.zeros((3, 8, 8), np.uint8), "zyx") with_scale(path, [2.0, 0.25, 0.25], ["micrometer", "micrometer", "micrometer"]) - image = open_ngff(path) - assert image.scale == (2.0, 0.25, 0.25) - assert image.units == ("micrometer",) * 3 + assert open_ngff(path).physical_sizes == inspect_image(path, "a.ome.zarr").physical_sizes sizes = inspect_image(path, "a.ome.zarr").physical_sizes assert (sizes.x, sizes.y, sizes.z, sizes.space_unit) == (0.25, 0.25, 2.0, "micrometer") # A scale for the whole image multiplies the level's scale. @@ -625,10 +855,12 @@ def test_ome_zarr_pixel_sizes_come_from_the_scale(tmp_path): assert inspect_image(path, "a.ome.zarr").physical_sizes.micrometers("X") == 0.5 -def test_ome_zarr_scale_without_units_is_unknown(tmp_path): +def test_ome_zarr_scale_without_units_cannot_be_compared(tmp_path): + # A unitless scale is kept (PR #20) but has no unit, so it is never compared. path = write_ngff(tmp_path / "a.ome.zarr", np.zeros((8, 8), np.uint8), "yx") - assert open_ngff(path).scale == (1.0, 1.0) - assert inspect_image(path, "a.ome.zarr").physical_sizes.x is None + sizes = inspect_image(path, "a.ome.zarr").physical_sizes + assert sizes.unit("X") is None + assert sizes.micrometers("X") is None def test_pixel_sizes_are_compared_across_tiff_and_ome_zarr(tmp_path): diff --git a/tests/zarr_tree.py b/tests/zarr_tree.py index dbb8b79c..0cf0d7c0 100644 --- a/tests/zarr_tree.py +++ b/tests/zarr_tree.py @@ -25,11 +25,19 @@ def write_ngff( channel_names=None, axis_types=None, extra_multiscales=0, + scale=None, + units=None, + dataset_transforms=None, + multiscale_transforms=None, ): """Write ``data`` as a one-level OME-Zarr image (OME-NGFF 0.4 or 0.5) at ``path``. ``axes`` gives one OME-NGFF axis name per dimension; ``axis_types`` overrides - their types. Returns the path. + their types. ``scale`` is the level-0 scale (1 per axis by default), + ``dataset_transforms`` replaces the level-0 ``coordinateTransformations`` + whole (``[]`` leaves them out), ``units`` maps axis names to their ``unit``, + and ``multiscale_transforms`` adds ``coordinateTransformations`` to the + multiscales entry. Returns the path. """ zarr = pytest.importorskip("zarr") if version == "0.5" and zarr_major() < 3: @@ -41,15 +49,21 @@ def write_ngff( chunks = chunks or (*(1,) * (data.ndim - 2), *data.shape[-2:]) _array(group, "0", data, chunks) types = axis_types or [AXIS_TYPES.get(a, "space") for a in axes] + units = units or {} + if dataset_transforms is None: + dataset_transforms = [{"type": "scale", "scale": scale or [1.0] * data.ndim}] + dataset = {"path": "0"} + if dataset_transforms: + dataset["coordinateTransformations"] = dataset_transforms multiscale = { - "axes": [{"name": a, "type": t} for a, t in zip(axes, types, strict=True)], - "datasets": [ - { - "path": "0", - "coordinateTransformations": [{"type": "scale", "scale": [1.0] * data.ndim}], - } + "axes": [ + {"name": a, "type": t, **({"unit": units[a]} if a in units else {})} + for a, t in zip(axes, types, strict=True) ], + "datasets": [dataset], } + if multiscale_transforms is not None: + multiscale["coordinateTransformations"] = multiscale_transforms attrs: dict = {"multiscales": [multiscale] * (1 + extra_multiscales)} if channel_names: attrs["omero"] = {"channels": [{"label": name} for name in channel_names]}