diff --git a/AGENTS.md b/AGENTS.md index 6f1a41e..b1889d2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -335,6 +335,9 @@ src/morphobricks_preprocessing/ notebook_viewers.py which roles each step reads an image with notebook_scan_axes.py the quick scan's "Axes of this image": roles proposed from the header, changed right there, the proposal the Axes step starts from + notebook_zarr_copy.py the first screen's "Also save a copy as OME-Zarr" and the Axes step's + card that writes it once the roles are confirmed; ome_zarr_copy.py + the writers (OME-NGFF 0.4, written directly) and the copy's name notebook_projection.py the Axes step's card for Z-stacks: choose the projection, write the projected copies into the output folder (convert_experiment), open them as the experiment; the greyed 3-D option @@ -1098,7 +1101,8 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb [docs/archive/agents-status-history-v0.1-v0.3.md](docs/archive/agents-status-history-v0.1-v0.3.md). - **Unreleased:** none yet; later changes go in a new "Unreleased" section of the changelog. - **Scope:** 2-D per plane (a Z-stack is refused as `z_stack_not_supported`); TIFF and OME-Zarr - input read directly (TIFF to OME-Zarr conversion and the `_moved.py` re-exports were removed); + input read directly (the v0.3 TIFF to OME-Zarr export and the `_moved.py` re-exports were + removed; since v0.5.2 `convert_experiment(store="ome-zarr")` writes recorded OME-Zarr copies); optional masks, the mask step and the notebook's **Describe the masks**; named-axis copies (`convert`, notebook card); calibration; resumable, crash-consistent runs. - **Command line:** `check`, `run`, `masks`, `calibrate`, `describe`, `convert` (`check`, diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a59ff4..40163cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -119,6 +119,17 @@ experiment-v1 §4.5, §4.5a-§4.5c, §7, §8; ADR 0002 resolver contract version shown greyed out with "3-D analysis is not enabled yet"; `Capabilities.volume_support` and `A_VOLUME_NOT_SUPPORTED` are unchanged. Copies written earlier are offered when the stacks are opened again. +- **Also save a copy as OME-Zarr.** An option on the first screen (unticked; it writes + nothing). Once the axis roles are confirmed, the Axes step offers **Write the OME-Zarr copy + and continue with it**: `convert_experiment(store="ome-zarr")` writes every image that is + read as `.ome.zarr` (OME-NGFF 0.4, written directly: uncompressed, one chunk per + plane, the same bytes for the same source; each `omero` channel has the `color` and + `window` OME-NGFF 0.4 requires, white and the pixel type's whole range) into `/ome_zarr`, with the conversion + record, and the copies are opened as the experiment. With Z-stacks the projected copies are + written as OME-Zarr. The record's entry of such a copy has the copy's path as + `relative_path` and the new key `source_relative_path`; records without OME-Zarr copies are + byte for byte as before (record version 3 is kept). New refusal: `copy_names_collide`. + `plan_conversion` takes `projection`, `masks_folder` and `store` as keywords. - **Notebook file, cell 3:** on Google Colab it calls `google.colab.output.no_vertical_scroll()`, so the app is not shown in an output frame that scrolls on its own. Nothing changes elsewhere. diff --git a/docs/contract.md b/docs/contract.md index 68273b7..b1ee51d 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -791,6 +791,7 @@ exclude=, missing_as_singleton=, memory_limit=, progress=)` (`conversion`), the | Repeat | A copy already in the tree is reused when its source (fingerprint and axes), roles, copy axes, pixel sizes, unknown sizes (roles and reasons), channel names, significant bits, the copy's fingerprint and the copy's SHA-256 (read whole) are as recorded; a new image is added; when every copy is reused the record is kept as it was (same SHA-256). Anything else (a source changed, other roles, a copy changed, an image no longer in the experiment, another experiment's copy tree, a file at a new image's place that the record does not list) is `copy_differs`: nothing is overwritten. A conversion record that cannot be read, or of another version than 3, is `conversion_record_unreadable`. | | Conversion record | `.morphobricks-conversion.json` in the copy tree's root (a dot-file, so discovery skips it), record version 3: `source_root`, `mapping`, `projection` (null, or `{method, z_range}`: the method and slice range of a tree of Z-projected copies, see "Projected copies" below), `missing_as_singleton`, `written_at`, `versions`, and per image `relative_path`, `action` (`converted` or `copied`), `reason` (the axis code, `excluded`, or null), `source` and `copy` fingerprints (`size`, `mtime_ns`), `copy_sha256` (the copy's SHA-256, computed in one pass after it is written; an OME-Zarr copy's is the SHA-256 of its files' relative paths and SHA-256s in path order, its `labels` group left out as for fingerprints: `content_hash`), `source_axes`, `roles`, `copy_axes`, `pixel_sizes` (by role, as §9), `unknown_sizes` (role: why), `channel_names` (as the copy holds them; `[]` for none), `significant_bits` (or null) and `projection` (null, or the projected copy's provenance, below). Sources keep the fingerprint rule (design v0.4 §1); the copies' SHA-256 is the owner's decision of 2 November 2026 (plan §12 question 59 (16)). Version 2 (PP-11c) added `copy_sha256`, `channel_names` and `significant_bits`; version 3 (issue #176) added the record-level `projection` and the entry's `projection`, and the action `projected`. Only version 3 is read (v0.5.1, SWEEP-PP: no backward compatibility, Spec ADR 0002 AR10): a record of any other version, 1 and 2 included, is refused everywhere it is read (`convert`, the run's `review_conversion`, `RunInputs`), never converted, with a message to convert the originals again into a new folder. `read_conversion_record(folder)` reads it with its SHA-256. | | Projected copies (issue #176, record version 3) | A tree made with a Z projection holds, for every image, a 2-D copy of its Z-stack made with the record-level `projection` `{method, z_range}` (`max`, `mean` or `sum`; there is no default method; `z_range` is `[start, stop]`, zero-based, stop excluded, the same for every image, or null for every slice). An entry has the action `projected` and a `projection` object with exactly: `source_relative_path`, `source_content_sha256` (the source's content as read while writing), `source_z_axis` (`{letter, index}` of the header's Z axis, or null for an image without one, which is one slice), `source_z_size`, `z_range` (the slices actually used), `method`, `source_dtype`, `output_dtype`, `dtype_rule`, `rounding` (`none`, `float64, one division` or `float64 addition, z ascending`), `nan_policy` (`propagate`), `dropped_sizes` (`{"Z": {value, unit, source}}`, or `{"Z": null}` when the source gave no Z size: the Z size is never written to the copy) and `significant_bits_dropped` (a reason, or null: kept for `max`, dropped for `sum` and `mean`). Excluded images stay plain copies (`copied`, reason `excluded`). The output type depends only on the input type, the method and the slice count (never on the data): `max` keeps the type; `sum` of `uint8`/`uint16` is `uint32`, of `int8`/`int16` is `int32`, of `uint32`/`int32` is `float64` holding the exact integer, of floats is `float64`; `mean` is `float64`; 64-bit integers are refused (`cannot_convert_pixel_type`). A sum or mean whose worst case (the type's extreme times the slices) could not be held exactly is refused (`projection_cannot_be_exact`), never wrapped or clipped. A NaN makes its pixel NaN, infinities are kept. The reader refuses an entry whose output type or rounding is not the table's for its method, a projected entry in a record without a projection (and the reverse), and a method or range that is not the record's. The reuse key holds the projection (everything of it except the content SHA-256, which the write computes), so another method or range into the same tree is `copy_differs`. Masks are never projected: `convert_experiment(..., projection=Projection(...), masks_folder=)` scans the masks folder from the headers only and refuses, before writing anything, masks of the images to project that have depth slices (`projection_masks_3d`) or more than Y and X with axes that are not named (`projection_masks_axes_unknown`); 2-D masks are left alone. The other refusals, all before anything is written: `nothing_to_project` (no image has a Z role), `projection_z_range_outside`, `projection_cannot_be_exact`, `cannot_convert_pixel_type`, `projection_over_memory_limit` (one input plane, the running plane and, when it is another type, the output plane must fit `memory_limit`) and `projection_zarr_not_supported` (OME-Zarr images are not projected yet). The copy's OME-XML holds an `Image/Description` with the method, slices, source, source SHA-256 and types, and a UUID derived from it, so one projection of one source is the same bytes; the source's content SHA-256 is taken before and after the projection and the source fingerprint before and after, and a source that changed is refused. A sum that overflows to infinity (`projection_overflow`) or adds an infinity to its opposite (`projection_undefined`) stops the conversion and is never written. Excluded images are copied byte for byte. | +| OME-Zarr copies (v0.5.2, record version 3) | `convert_experiment(..., store="ome-zarr")` (the notebook's **Also save a copy as OME-Zarr**; default `"ome-tiff"`) writes every image that is read as an OME-Zarr image instead of an OME-TIFF: also the images whose headers name their axes as confirmed, which are otherwise copied byte for byte. Excluded images and OME-Zarr images are copied byte for byte as before. With `projection` the projected copies are OME-Zarr. A copy is named after its image with `.ome.zarr` in place of the TIFF extension, in the same folder, so its position ID is the same; two images that would get one copy name are refused (`copy_names_collide`). In the record the entry's `relative_path` is the copy's path and `source_relative_path` (a key written only for such an entry, so every other record is byte for byte as before) is the source's path relative to `source_root`; `significant_bits` is null (OME-NGFF 0.4 has no field for it). The store is OME-NGFF 0.4 (Zarr version 2), written directly: one array `0`, uncompressed, little-endian, one chunk per plane (the whole Y and X), metadata as sorted JSON, so one copy of one source is the same bytes. Each plane is the plane read under the confirmed roles, bit for bit; the axes are the roles (`t`, `c`, `z`, `y`, `x`) with the units of the copy's sizes, the level-0 `scale` holds the sizes and 1 for an unknown one, which is named under the multiscale's `metadata.unknown_scale` and so read as unknown; channel names are under `omero.channels[].label`, each channel with the `color` (`FFFFFF`) and `window` (the pixel type's whole range as `min`, `max`, `start`, `end`) that OME-NGFF 0.4 requires, display only and never read from the pixels; a projected copy's description is under `morphobricks.description`. Reading the copies needs the `zarr` extra, as for any OME-Zarr image. Plan, lock, `.partial` writes, reuse, the copy's SHA-256 and the checks of the run on the copies are those of the other copies. | | The run on the copies | The user runs preprocessing on the copy tree with the same output folder (`--output`): the recorded experiment root becomes the copy tree and the report says so (`experiment_root_changed`, and `conversion` with the record, its SHA-256 and `source_root`; the text says "named-axis copies of ..."); every row is recomputed (the row key includes the read path, §9); the design, image type and calibration are kept (same relative paths). The manifest records `conversion_record_sha256`, and for projected copies the `projection` summary (§9); the report says "max projections of slices 1:4 (2 images) of " (and `conversion.projection` in `discovery_report.json`). Masks are paired as usual (the image has no Z now, so 2-D masks fit; a mask with depth slices stays `mask_planes_mismatch` and stops the run), and the run warns `masks_on_projected_images`, naming the method, for masks paired with projections, in the report and the manifest's `projection.warnings`. The run and `check` warn when a source changed or went missing after the conversion (`conversion_source_changed`) and refuse a copy that changed (its fingerprint or its SHA-256: each copy is read whole, so `check` and `run` on a copy tree cost one more read of the copies), went missing or was added since (`copy_changed_after_conversion`: an error since PP-12, owner decision of 2 November 2026; the run stops after the header checks, before any pixel is read, writing only its report, since `RunInputs` would refuse the run; the message says to convert the originals again into a new folder); an unreadable record stops the run (`conversion_record_unreadable`). `check` and `run` also read the OME `Image/Description` header (first page's description tag only, no pixels) of every analysed TIFF the record does not list as `projected`: a description that begins "MorphoBricks Z projection:" in a tree with no record, or whose record does not list the image as projected, is refused as `projection_record_missing` (owner decision of 8 November 2026; plain images are unaffected). `RunInputs` reads the copies like any images, and refuses a run whose conversion record changed or went missing since (`conversion_record_sha256`), is of version 1, or whose recorded images' copies no longer have the recorded fingerprint and SHA-256 (§9.1). | | Notebook (PP-11b) | Step 2's **Make named-axis copies (optional)** card, under the axis check, shown only when it reports images with `axes_yx_swapped`, `axes_contradict_header`, `axes_x_before_y` or `axes_not_named` for the roles being edited (design v0.4 §5: offered only for files with errors), and not with `folder_step(show_output=False)`. The roles are the user's: those confirmed in step 2, or the roles shown once the preview is checked and the card's own question is answered; any change of a role or of the missing-axes answer takes it back. A required folder for the copies (Browse…, or pasted) checked with the command's rule (`copy_folder_issues`); `preview_conversion` (`conversion_plan`, headers only, writes nothing) gives what will be converted and copied byte for byte (the command's `plan_conversion`), the space needed and the free space; then `convert_experiment` with a progress bar, under the copy tree's lock. Refusals use the existing cards (the lock card with **Remove the lock of a stopped run**, the rule of `--remove-stale-lock`). After the copies: the record's path, and **Check the copies in step 1**, which sets the experiment folder to the copy tree, keeps the output folder and checks again; the card says that the missing-axes answer must be given again and lists the copies whose relabelled sizes are unknown (`ConversionResult.unknown_sizes`), to state under **Pixel sizes**. | diff --git a/docs/user_guide.md b/docs/user_guide.md index 3018988..901753a 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -382,6 +382,17 @@ lists each one (folder, path, fingerprint, SHA-256, slices used). **Keep the 3-D When you open the stacks again with the same output folder, the copies written earlier are offered (**Continue with the copies written earlier**). +**A copy as OME-Zarr.** The Folders card of step 1 has **Also save a copy as OME-Zarr (in the +output folder)**. Ticking it writes nothing. Once the axis roles are confirmed, the Axes step +shows **OME-Zarr copy** with **Write the OME-Zarr copy and continue with it**: every image is +written plane for plane, with the confirmed roles, to `/ome_zarr` (one +`.ome.zarr` per image, OME-NGFF 0.4, uncompressed), with a conversion record, and +the copies are opened as the experiment (scanned and checked again; you confirm their axes and +go on). Your images are only read. With Z-stacks the projected copies are written as OME-Zarr +instead (`/projected_max_zarr`). Pixel sizes you stated for the whole experiment or a +folder keep applying; a size stated for a single image names the image file, so state it again +for the copy if needed (the check says so: `calibration_target_not_found`). + Folders may have different numbers of channels (for example three stains everywhere and a fourth only in `Biological_repeat_2`): on **Channels** you give each label once and tag each folder's channels; the same label means the same stain in every folder. A stain a folder does @@ -983,6 +994,7 @@ continues. | `copy_folder_overlaps_experiment` | Error | `convert`: the folder for the copies is the experiment folder, inside it, or holds it. Choose a new folder outside the experiment. | | `copy_folder_overlaps_masks` | Error | `convert`: the folder for the copies is the masks folder, inside it, or holds it; copies there would be read as masks. Choose a folder outside the masks folder. | | `copy_folder_not_a_folder` | Error | `convert`: the path given with `--to` is a file. Choose a folder. | +| `copy_names_collide` | Error | An OME-Zarr copy: two images of one folder differ only by their extension, so their copies would have the same name. Nothing is written. Rename or exclude one of them. | | `copy_folder_not_empty` | Error | `convert`: the folder for the copies holds files and no conversion record. Nothing is overwritten; choose a new or empty folder. | | `copy_differs` | Error | `convert`: images in the folder of copies differ from the copies this conversion would write (an original changed, other roles, a copy changed, an image no longer in the experiment), or the folder holds copies of another experiment. Nothing is overwritten; convert into a new folder, or remove those copies first. | | `cannot_convert_pixel_type` | Error | `convert`: images of 64-bit integers cannot be saved as OME-TIFF, and pixel values are never changed to fit. Save them again with named axes in a 32-bit type if their values fit, or correct the roles. | diff --git a/pyproject.toml b/pyproject.toml index 4736ebe..54f936a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -148,6 +148,7 @@ baseline = "tools/code_rules_baseline.json" "morphobricks_preprocessing.notebook_quick_scan" = "ui" "morphobricks_preprocessing.notebook_viewer" = "ui" "morphobricks_preprocessing.notebook_projection" = "ui" +"morphobricks_preprocessing.notebook_zarr_copy" = "ui" "morphobricks_preprocessing.notebook_scan_axes" = "ui" "morphobricks_preprocessing.notebook_viewers" = "ui" "morphobricks_preprocessing.image_view" = "io" @@ -224,6 +225,7 @@ baseline = "tools/code_rules_baseline.json" "morphobricks_preprocessing.projection_plan" = "orchestration" "morphobricks_preprocessing.planned_copy" = "pure" "morphobricks_preprocessing.ome_tiff_copy" = "io" +"morphobricks_preprocessing.ome_zarr_copy" = "io" "morphobricks_preprocessing.discovery_report" = "pure" "morphobricks_preprocessing.dataset" = "pure" "morphobricks_preprocessing.design" = "io" diff --git a/src/morphobricks_preprocessing/conversion.py b/src/morphobricks_preprocessing/conversion.py index 4797374..9e03c59 100644 --- a/src/morphobricks_preprocessing/conversion.py +++ b/src/morphobricks_preprocessing/conversion.py @@ -60,6 +60,7 @@ from .conversion_reuse import compare from .conversion_write import ConversionFailed, write_all from .manifest import package_versions +from .ome_zarr_copy import OME_TIFF, STORES from .planes import DEFAULT_MEMORY_LIMIT from .planned_copy import PlannedCopy from .projection import Projection @@ -132,6 +133,7 @@ class ConversionOptions(TypedDict, total=False): progress: Callable[[int, int, str], None] | None masks_folder: str | os.PathLike | None projection: Projection | None + store: str def convert_experiment( @@ -180,6 +182,10 @@ def convert_experiment( decided by the dtype table (:func:`.projection.projection_dtype`), and the record says how every copy was made (record version 3). Masks are never projected. Excluded images are still copied byte for byte. None converts as before. + store + ``"ome-zarr"`` writes every image that is read as an OME-Zarr copy (OME-NGFF 0.4, + :mod:`.ome_zarr_copy`) named ``.ome.zarr``, with or without a projection; + the record keeps each source's own path. Default ``"ome-tiff"``: as before. Returns ------- @@ -188,8 +194,9 @@ def convert_experiment( Raises ------ ValueError - If ``missing_as_singleton`` is not True or False, or ``memory_limit`` is not a - whole number of bytes above 0 (nothing is read or written then). + If ``missing_as_singleton`` is not True or False, ``memory_limit`` is not a + whole number of bytes above 0, or ``store`` is unknown (nothing is read or written + then). """ unknown = sorted(set(options) - set(ConversionOptions.__annotations__)) if unknown: @@ -197,14 +204,24 @@ def convert_experiment( missing_as_singleton = options.get("missing_as_singleton", False) memory_limit = options.get("memory_limit", DEFAULT_MEMORY_LIMIT) masks_folder, projection = options.get("masks_folder"), options.get("projection") - progress = options.get("progress") + progress, store = options.get("progress"), options.get("store", OME_TIFF) _check_arguments(missing_as_singleton, memory_limit, projection) + if store not in STORES: + raise ValueError(f"store must be one of {STORES}, found {store!r}") root, dest = Path(experiment_root).resolve(), Path(destination).resolve() refused = copy_folder_issues(root, dest, masks_folder) if refused: return ConversionResult(dest, False, tuple(refused)) planned, issues = plan_conversion( - root, mapping, dest, exclude, missing_as_singleton, memory_limit, projection, masks_folder + root, + mapping, + dest, + exclude, + missing_as_singleton, + memory_limit, + projection=projection, + masks_folder=masks_folder, + store=store, ) if has_errors(issues): return ConversionResult(dest, False, _sorted(issues)) diff --git a/src/morphobricks_preprocessing/conversion_issues.py b/src/morphobricks_preprocessing/conversion_issues.py index 9f101c9..c6ac4f8 100644 --- a/src/morphobricks_preprocessing/conversion_issues.py +++ b/src/morphobricks_preprocessing/conversion_issues.py @@ -51,6 +51,17 @@ def not_a_folder_issue(dest: Path) -> ValidationIssue: ) +def names_collide_issue(paths: list[str]) -> ValidationIssue: + """Refuse OME-Zarr copies of images that differ only by their extension.""" + return error( + "copy_names_collide", + "Images of one folder differ only by their extension, so their OME-Zarr copies would " + "have the same name. Nothing was written.", + paths, + "Rename one of them (its position ID changes), or exclude it.", + ) + + def space_issue(dest: Path, needed: int, free: int) -> ValidationIssue: """Refuse a conversion that would not fit on the drive of the copy tree.""" return error( diff --git a/src/morphobricks_preprocessing/conversion_plan.py b/src/morphobricks_preprocessing/conversion_plan.py index 0173901..4da7674 100644 --- a/src/morphobricks_preprocessing/conversion_plan.py +++ b/src/morphobricks_preprocessing/conversion_plan.py @@ -20,6 +20,7 @@ from collections.abc import Iterable, Mapping from dataclasses import dataclass from pathlib import Path +from typing import TypedDict, Unpack from ._io import free_space from .axes import AxisMapping, AxisReview @@ -34,6 +35,7 @@ from .conversion_issues import ( header_changed_issue, masks_overlap_issue, + names_collide_issue, not_a_folder_issue, overlap_issue, pixel_type_issue, @@ -44,6 +46,7 @@ from .links import relative_inside from .metadata import TiffMetadata from .ome_tiff_copy import copy_nbytes +from .ome_zarr_copy import OME_TIFF, OME_ZARR, as_zarr_copy from .planes import DEFAULT_MEMORY_LIMIT, check_planes from .planned_copy import PlannedCopy from .projection import Projection @@ -64,8 +67,7 @@ def plan_conversion( exclude: Iterable[str] = (), missing_as_singleton: bool = False, memory_limit: int = DEFAULT_MEMORY_LIMIT, - projection: Projection | None = None, - masks_folder: str | os.PathLike | None = None, + **how: Unpack[PlanOptions], ) -> tuple[tuple[PlannedCopy, ...], list[ValidationIssue]]: """Check the experiment with the confirmed roles and plan each image's copy. @@ -97,6 +99,11 @@ def plan_conversion( (``masks_folder``, :func:`.projection_masks.projection_mask_issues`) do. masks_folder The masks folder, scanned for 3-D masks when ``projection`` is given. + store + ``"ome-zarr"`` plans every image that is read as an OME-Zarr copy + (:mod:`.ome_zarr_copy`): also the images whose headers name their axes as confirmed, + which are otherwise copied byte for byte. Two images that would get the same copy + name stop the plan (``copy_names_collide``). Default ``"ome-tiff"``. Returns ------- @@ -104,6 +111,8 @@ def plan_conversion( The planned copies, sorted by path (empty when an error stops the conversion), and the issues: the errors that stop it, and the warnings. """ + projection, masks_folder = how.get("projection"), how.get("masks_folder") + as_zarr = how.get("store", OME_TIFF) == OME_ZARR pre = preflight(root, destination, exclude=exclude, bind_headers=True) unsettled = sorted(rel for rel, bound in pre.header_fingerprints.items() if bound is None) if unsettled: @@ -120,9 +129,16 @@ def plan_conversion( to_convert = set(review.axis_error_files) | set(review.confirmed_files) headers = {meta.relative_path: meta for meta in pre.metadata.files} if projection is not None: - return _plan_projected( + planned, issues = _plan_projected( pre, review, mapping, projection, issues, (memory_limit, masks_folder) ) + return _stored(planned, issues, as_zarr) + if as_zarr: # every TIFF image that is read is written, not only those to correct + to_convert = { + rel + for rel, meta in headers.items() + if rel in review.file_mappings and not meta.is_ome_zarr + } issues += _cannot_convert(root, to_convert, headers, review.file_mappings, memory_limit) if has_errors(issues): return (), issues @@ -134,7 +150,31 @@ def plan_conversion( planned.append(_converted(headers[rel], review.file_mappings[rel], source)) else: planned.append(_copied(rel, headers.get(rel), source, rel in pre.excluded)) - return tuple(planned), issues + return _stored(tuple(planned), issues, as_zarr) + + +class PlanOptions(TypedDict, total=False): + """The keyword options of :func:`plan_conversion` besides the folders and the limit.""" + + projection: Projection | None + masks_folder: str | os.PathLike | None + store: str + + +def _stored( + planned: tuple[PlannedCopy, ...], issues: list[ValidationIssue], as_zarr: bool +) -> tuple[tuple[PlannedCopy, ...], list[ValidationIssue]]: + """Return the plan with its copies as OME-Zarr when asked; refuse two with one name.""" + if not as_zarr or not planned: + return planned, issues + renamed = tuple(as_zarr_copy(item) for item in planned) + names: dict[str, list[str]] = {} + for item in renamed: + names.setdefault(item.relative_path.casefold(), []).append(item.source_path) + twice = sorted(path for paths in names.values() if len(paths) > 1 for path in paths) + if twice: + return (), [*issues, names_collide_issue(twice)] + return tuple(sorted(renamed, key=lambda item: item.relative_path)), issues def _plan_projected( diff --git a/src/morphobricks_preprocessing/conversion_record.py b/src/morphobricks_preprocessing/conversion_record.py index e6fe683..f66bac6 100644 --- a/src/morphobricks_preprocessing/conversion_record.py +++ b/src/morphobricks_preprocessing/conversion_record.py @@ -104,6 +104,11 @@ class CopyEntry: The significant bits the copy holds, or None. projection How a projected image was made (:data:`PROJECTED` only), else None. + source_relative_path + The source's path relative to the experiment when the copy has another name (an + OME-Zarr copy of a TIFF image, :mod:`.ome_zarr_copy`); None when it is + ``relative_path``, as for every OME-TIFF and byte-for-byte copy. Written to the + record only when it is given, so records without OME-Zarr copies are unchanged. """ relative_path: str @@ -120,6 +125,12 @@ class CopyEntry: channel_names: tuple[str | None, ...] = () significant_bits: int | None = None projection: ProjectionEntry | None = None + source_relative_path: str | None = None + + @property + def source_path(self) -> str: + """The source image, relative to the experiment folder.""" + return self.source_relative_path or self.relative_path def __post_init__(self) -> None: """Refuse a projection on an entry that is not projected, and the reverse.""" @@ -153,11 +164,18 @@ def _plan(self) -> tuple: tuple(self.channel_names), self.significant_bits, None if self.projection is None else self.projection.plan_key(), + self.source_relative_path, ) def to_dict(self) -> dict: """Return the entry as JSON-compatible data.""" + named = ( + {} + if self.source_relative_path is None + else {"source_relative_path": self.source_relative_path} + ) return { + **named, "relative_path": self.relative_path, "action": self.action, "reason": self.reason, @@ -200,6 +218,11 @@ def from_dict(cls, data: Mapping) -> CopyEntry: if data["projection"] is None else ProjectionEntry.from_dict(data["projection"]) ), + source_relative_path=( + None + if data.get("source_relative_path") is None + else str(data["source_relative_path"]) + ), ) @@ -413,9 +436,9 @@ def review_conversion( record, sha256 = found sources = Path(record.source_root) changed = [ - e.relative_path + e.source_path for e in record.files - if current_fingerprint(sources / e.relative_path) != e.source + if current_fingerprint(sources / e.source_path) != e.source ] listed = {e.relative_path for e in record.files} copies = sorted(set(changed_copies(root, record)) | (set(images) - listed)) diff --git a/src/morphobricks_preprocessing/conversion_write.py b/src/morphobricks_preprocessing/conversion_write.py index 5a2bcca..c8fdd6e 100644 --- a/src/morphobricks_preprocessing/conversion_write.py +++ b/src/morphobricks_preprocessing/conversion_write.py @@ -18,6 +18,7 @@ from .conversion_record import CopyEntry from .fingerprint import Fingerprint, current_fingerprint from .ome_tiff_copy import CopyWriteError, write_named_copy, write_projected_copy +from .ome_zarr_copy import OME_ZARR, write_named_zarr, write_projected_zarr from .planes import PlaneReadError from .planned_copy import PlannedCopy from .projection import ProjectionRefused @@ -127,7 +128,7 @@ def _write_partial(root: Path, dest: Path, item: PlannedCopy, memory_limit: int) Returns the source's content SHA-256 for a projected copy (taken before and after the projection, which must agree), else None. """ - source, partial = root / item.relative_path, partial_path(dest, item) + source, partial = root / item.source_path, partial_path(dest, item) partial.parent.mkdir(parents=True, exist_ok=True) _remove(partial) # left by an interrupted conversion: it is never reused if current_fingerprint(source) != item.entry.source: @@ -136,13 +137,13 @@ def _write_partial(root: Path, dest: Path, item: PlannedCopy, memory_limit: int) if item.entry.projection is not None: content = source_content_sha256(source) details = item.entry.projection.with_content(content) - write_projected_copy( - source, partial, item.layout, item.mapping, details, memory_limit=memory_limit - ) + project = write_projected_zarr if item.store == OME_ZARR else write_projected_copy + project(source, partial, item.layout, item.mapping, details, memory_limit=memory_limit) if source_content_sha256(source) != content: raise _SourceChanged("its content changed while it was projected") elif item.layout is not None: - write_named_copy( + write = write_named_zarr if item.store == OME_ZARR else write_named_copy + write( source, partial, item.layout, item.mapping, dtype=item.dtype, memory_limit=memory_limit ) elif source.is_dir(): diff --git a/src/morphobricks_preprocessing/notebook_compact.py b/src/morphobricks_preprocessing/notebook_compact.py index 212d811..adb5343 100644 --- a/src/morphobricks_preprocessing/notebook_compact.py +++ b/src/morphobricks_preprocessing/notebook_compact.py @@ -62,7 +62,8 @@ def _images_page(self) -> list[widgets.Widget]: ] ) self.folders_card = ui.card( - "Folders", [rows, self.output_hint, self.masks_recorded, self.masks_drop] + "Folders", + [rows, self.output_hint, self.masks_recorded, self.masks_drop, self.zarr_option_row], ) self.found_fold = ui.accordion("Folders found", [self.labels_card]) self.check_report.flat() diff --git a/src/morphobricks_preprocessing/notebook_projection.py b/src/morphobricks_preprocessing/notebook_projection.py index 8580214..5f41ab0 100644 --- a/src/morphobricks_preprocessing/notebook_projection.py +++ b/src/morphobricks_preprocessing/notebook_projection.py @@ -40,6 +40,7 @@ from .notebook_progress import Wait from .notebook_render import blocked_line, issues_brief_html from .notebook_viewer import READ_ERRORS +from .ome_zarr_copy import OME_TIFF, OME_ZARR, ZARR_COPY_SUFFIX from .projection import METHODS, Projection from .size_checks import Z_STACK_CODE from .validation import plural @@ -70,12 +71,12 @@ ) -def copies_folder(output: Path, projection: Projection) -> Path: +def copies_folder(output: Path, projection: Projection, store: str = OME_TIFF) -> Path: """Return the folder of the projected copies inside the output folder.""" name = FOLDER_PREFIX + projection.method if projection.z_range is not None: name += f"_z{projection.z_range[0]}-{projection.z_range[1]}" - return output / name + return output / (name + ("_zarr" if store == OME_ZARR else "")) def projection_words(method: str, z_range: tuple[int, int] | None) -> str: @@ -257,10 +258,12 @@ def _projection_changed(self) -> None: if chosen is None: self.projection_where.value = "" return - dest = copies_folder(self._output_path(self.preflight.discovery.root), chosen) + store = self._copy_store() + dest = copies_folder(self._output_path(self.preflight.discovery.root), chosen, store) + kind = " as OME-Zarr (asked on Images)" if store == OME_ZARR else "" self.projection_where.value = ( f"
The {html.escape(projection_words(chosen.method, chosen.z_range))} of every " - f"image is written to {html.escape(str(dest))}.
" + f"image is written{kind} to {html.escape(str(dest))}." ) # Looking ------------------------------------------------------------------------------- @@ -327,13 +330,14 @@ def progressed(done: int, total: int, image: str) -> None: result = convert_experiment( root, self.mapping, - copies_folder(output, chosen), + copies_folder(output, chosen, self._copy_store()), exclude=self.preflight.excluded, missing_as_singleton=self.missing_as_singleton.value, memory_limit=self._memory_limit(), progress=progressed, masks_folder=self._runs_masks_folder(), projection=chosen, + store=self._copy_store(), ) except Exception as exc: # unexpected failures still get a readable message self.log.lines.append(traceback.format_exc()) @@ -361,6 +365,8 @@ def _list_earlier(self) -> None: buttons = [] for folder, projection in earlier_projections(root, output): words = projection_words(projection.method, projection.z_range) + if folder.name.endswith("_zarr"): + words += f", as OME-Zarr ({ZARR_COPY_SUFFIX})" button = ui.button( f" Continue with the copies written earlier: {words}", icon="folder-open", diff --git a/src/morphobricks_preprocessing/notebook_steps.py b/src/morphobricks_preprocessing/notebook_steps.py index 301f7f3..5bd4eae 100644 --- a/src/morphobricks_preprocessing/notebook_steps.py +++ b/src/morphobricks_preprocessing/notebook_steps.py @@ -42,6 +42,7 @@ from .notebook_step_states import _StepStates from .notebook_stepper import Step, Stepper, about from .notebook_viewers import VIEW_HELP, _Viewers +from .notebook_zarr_copy import _ZarrCopyCard from .run import Preflight AXES_INTRO = ( @@ -92,6 +93,7 @@ class _StepsFlow( _QuickAxes, _ScanAxes, _ProjectionCard, + _ZarrCopyCard, _Viewers, _StepStates, ): @@ -129,6 +131,7 @@ def steps(self) -> widgets.Widget: self._use_viewers() self._build_scan_axes() self._build_projection_card() + self._build_zarr_copy() self._build_quick_scan() self.consistency_slot = widgets.VBox([self.consistency_card]) self._build_design_step() @@ -149,7 +152,8 @@ def _step_list(self) -> list[Step]: self._images_page()), ("Axes", "What each image dimension is, the pixel size and the image type.", [about(AXES_INTRO), self.copy_note, self.axes_change_note, self.axis_box, - self.projection_card, self.quick_card, self.type_line.widget]), + self.projection_card, self.quick_card, self.zarr_card, + self.type_line.widget]), ("Channels", "Name what you stained, then tag each folder's channels.", [about(CHANNELS_INTRO), self.channel_labels.widget, self.channels_look, self.channel_grid.widget, self.moves_box, self.names_box]), @@ -249,6 +253,7 @@ def check_folder(self) -> Preflight | None: self._show_checked_viewer() self._show_scan_axes() self._show_copy_note() + self._refresh_zarr_card() self.scan_status.value = "" self._refresh_folds() self._go_first_open() @@ -285,10 +290,12 @@ def _mapping_changed(self) -> None: def _toggle_confirmation(self) -> None: super()._toggle_confirmation() self._refresh_quick_axes() + self._refresh_zarr_card() def _refresh_convert_card(self, review: object) -> None: super()._refresh_convert_card(review) self._refresh_projection_card(review) + self._refresh_zarr_card() def _confirm_blockers(self) -> list[str]: """Name the projection card when Z-stacks are all that stops the confirmation.""" diff --git a/src/morphobricks_preprocessing/notebook_zarr_copy.py b/src/morphobricks_preprocessing/notebook_zarr_copy.py new file mode 100644 index 0000000..8f83216 --- /dev/null +++ b/src/morphobricks_preprocessing/notebook_zarr_copy.py @@ -0,0 +1,182 @@ +"""*Also save a copy as OME-Zarr*: asked on the first screen, written once the axes are confirmed. + +Owner (11 November 2026): an option on the first screen, **Also save a copy as OME-Zarr (in +the output folder)**. Ticking it writes nothing. The copy can only be written faithfully once +the roles of the dimensions are known, so the Axes step offers it after the axis roles are +confirmed (and while no pixel-size statement has an error): one click writes every image that +is read as an OME-Zarr image into ``/ome_zarr`` with +:func:`.conversion.convert_experiment` (``store="ome-zarr"``: the plan, the lock, the +conversion record and every refusal of the named-axis copies), then opens the copies as the +experiment, scanned and checked again; nothing is confirmed for the user. The originals are +only read. + +With Z-stacks the projection card writes the copies (:mod:`.notebook_projection`); ticked, it +writes them as OME-Zarr, into ``/projected_..._zarr``. +""" + +from __future__ import annotations + +import html +import traceback +from importlib.util import find_spec +from pathlib import Path + +import ipywidgets as widgets + +from . import notebook_style as ui +from .conversion import ConversionResult, convert_experiment +from .notebook_calibration import CALIBRATION_ISSUE_CODES +from .notebook_progress import Wait +from .notebook_render import blocked_line, issues_brief_html +from .ome_zarr_copy import OME_TIFF, OME_ZARR +from .size_checks import Z_STACK_CODE + +OPTION_LABEL = "Also save a copy as OME-Zarr (in the output folder)" +OPTION_HELP = ( + "Nothing is written now. Once the axis roles are confirmed (Axes, step 2), one click there " + "writes the copy with its conversion record, and the later steps use it. Your images are " + "only read." +) +NO_ZARR = 'the zarr package is not installed: pip install "morphobricks-preprocessing[zarr]"' +WRITE_LABEL = " Write the OME-Zarr copy and continue with it" +FOLDER_NAME = "ome_zarr" + + +class _ZarrCopyCard: + """The first screen's OME-Zarr option and the Axes step's card that writes the copy.""" + + def _build_zarr_copy(self) -> None: + """Build the option (unticked) and the card (hidden until the option is ticked).""" + readable = find_spec("zarr") is not None + self.zarr_option = widgets.Checkbox( + value=False, + description=OPTION_LABEL, + indent=False, + disabled=not readable, + layout=widgets.Layout(width="auto"), + ) + self.zarr_option_row = widgets.VBox( + [ + self.zarr_option, + widgets.HTML(blocked_line([] if readable else [NO_ZARR])), + ui.help_text(OPTION_HELP), + ] + ) + self.zarr_where = widgets.HTML() + self.zarr_button = ui.button(WRITE_LABEL, style="success", icon="clone", main=True) + self.zarr_blocker = widgets.HTML() + self.zarr_status = widgets.HTML() + self.zarr_card = ui.card( + "OME-Zarr copy", + [self.zarr_where, self.zarr_button, self.zarr_blocker, self.zarr_status], + visible=False, + ) + self.zarr_option.observe(lambda _: self._refresh_zarr_card(), names="value") + self.zarr_option.observe(lambda _: self._projection_changed(), names="value") + self.zarr_button.on_click(lambda _: self._write_zarr_copy()) + + def _copy_store(self) -> str: + """Return the store the copies are written in: OME-Zarr when the option is ticked.""" + option = getattr(self, "zarr_option", None) + return OME_ZARR if option is not None and option.value else OME_TIFF + + def _zarr_reasons(self) -> list[str]: + """Return what stops the OME-Zarr copy now, the first thing to do first.""" + if self.confirmed_mapping is None: + return ["the axis roles are not confirmed: confirm them above first"] + shown = self._shown_calibration()[0] + wrong = [ + i for i in shown.metadata.issues if i.is_error and i.code in CALIBRATION_ISSUE_CODES + ] + if wrong: + return ["a pixel size statement has an error: see Pixel size above"] + return [] + + def _refresh_zarr_card(self) -> None: + """Offer the card for a checked folder of TIFF images while the option is ticked.""" + if getattr(self, "zarr_card", None) is None: + return + pre = self.preflight + files = [] if pre is None else [f for f in pre.metadata.files if f.readable] + stacks = any( + c == Z_STACK_CODE.replace("_", " ") for c in getattr(self, "_review_errors", ()) + ) + offered = self.zarr_option.value and bool(files) and not stacks + ui.set_visible(self.zarr_card, offered) + if not offered: + return + if all(f.is_ome_zarr for f in files): + self.zarr_where.value = ui.status( + "ok", "These images are OME-Zarr already: there is nothing to write." + ) + for widget in (self.zarr_button, self.zarr_blocker): + ui.set_visible(widget, False) + return + for widget in (self.zarr_button, self.zarr_blocker): + ui.set_visible(widget, True) + dest = self._output_path(pre.discovery.root) / FOLDER_NAME + self.zarr_where.value = ( + f"
You asked on Images for a copy as OME-Zarr. Every image is written, plane " + f"for plane with the confirmed roles, to {html.escape(str(dest))}, with a " + "conversion record; the later steps then use the copy. Your images are only " + "read.
" + ) + reasons = self._zarr_reasons() + self.zarr_button.disabled = bool(reasons) + self.zarr_blocker.value = blocked_line(reasons) + + def _write_zarr_copy(self) -> ConversionResult | None: + """Write the OME-Zarr copies with the confirmed roles, then open them as the experiment. + + Returns + ------- + ConversionResult or None + None when the click is not available or an unexpected problem stopped it. + """ + if self.preflight is None or not self.zarr_option.value or self._zarr_reasons(): + return None + root = Path(self.preflight.discovery.root) + output = self._output_path(root) + self._busy(self.zarr_button, "Writing the OME-Zarr copy…") + wait = Wait(self.zarr_status) + wait.start("Checking the images and the folder") + + def progressed(done: int, total: int, image: str) -> None: + if image: + wait.step(done, total, image) + + try: + with self.log.attached(): + result = convert_experiment( + root, + self.confirmed_mapping, + output / FOLDER_NAME, + exclude=self.preflight.excluded, + missing_as_singleton=self.missing_as_singleton.value, + memory_limit=self._memory_limit(), + progress=progressed, + masks_folder=self._runs_masks_folder(), + store=OME_ZARR, + ) + except Exception as exc: # unexpected failures still get a readable message + self.log.lines.append(traceback.format_exc()) + self.zarr_status.value = ui.status( + "error", + "The copy could not be written because of an unexpected problem.", + [f"{type(exc).__name__}: {exc}", "Open Technical details for the full report."], + ) + return None + finally: + self._idle(self.zarr_button, WRITE_LABEL, "clone") + self._show_log() + if not result.written: + self.zarr_status.value = ui.status( + "error", "No copy was written.", ["Nothing was written: see below."] + ) + issues_brief_html(result.issues) + return result + self.zarr_status.value = "" + self._use_copies(result.destination, output) + return result + + +__all__ = ["FOLDER_NAME", "NO_ZARR", "OPTION_HELP", "OPTION_LABEL", "_ZarrCopyCard"] diff --git a/src/morphobricks_preprocessing/ome_zarr_copy.py b/src/morphobricks_preprocessing/ome_zarr_copy.py new file mode 100644 index 0000000..e820bfc --- /dev/null +++ b/src/morphobricks_preprocessing/ome_zarr_copy.py @@ -0,0 +1,294 @@ +"""Write one image as an OME-Zarr copy (OME-NGFF 0.4), plane by plane (contract §17). + +Owner (11 November 2026): the notebook can also save the experiment as OME-Zarr, once the axes +are confirmed, as a recorded conversion. A copy is written by +:func:`.conversion.convert_experiment` with ``store="ome-zarr"``: the same plan, lock, +``.partial`` discipline and conversion record as the named-axis OME-TIFF copies, with these +writers in place of :mod:`.ome_tiff_copy`'s. + +The store is written directly (no ``zarr`` package is needed to write it; reading it needs the +``zarr`` extra, as for any OME-Zarr image): a Zarr version 2 group with one array ``0``, +uncompressed, one chunk per plane (the whole Y and X), little-endian, so every plane has a +fixed file and the same copy is the same bytes. Each plane is the plane read under the +confirmed roles, bit for bit; a projected copy's planes are computed as in +:func:`.ome_tiff_copy.write_projected_copy`. The metadata follow the copy's layout +(:class:`.copy_axes.CopyLayout`): the axes by role with their units, the level-0 ``scale`` from +the pixel sizes (an unknown size is the placeholder 1, named under ``metadata`` / +``unknown_scale`` so it is read as unknown, :mod:`.physical_sizes`), the channel names under +``omero`` (with the ``color`` and ``window`` OME-NGFF 0.4 requires: white and the pixel type's +whole range, never read from the pixels), and a projected copy's description under +``morphobricks``. Significant bits cannot be stated in OME-NGFF 0.4 and are not written (the +record says so). + +A copy's name is the image's with ``.ome.zarr`` in place of its TIFF extension +(:func:`zarr_copy_name`), so its position ID is unchanged; the record keeps the source's own +path (:attr:`.conversion_record.CopyEntry.source_relative_path`). +""" + +from __future__ import annotations + +import json +import os +from dataclasses import replace +from pathlib import Path + +import numpy as np + +from .axes import AxisMapping +from .copy_axes import CopyLayout +from .image_names import position_id +from .ome_tiff_copy import CopyWriteError +from .physical_sizes import UNKNOWN_SCALE_KEY +from .planes import DEFAULT_MEMORY_LIMIT, iter_planes, read_plane +from .planned_copy import PlannedCopy +from .projection import PlaneAccumulator +from .projection_record import ProjectionEntry, projection_description + +OME_TIFF = "ome-tiff" +"""The store of a named-axis copy written as OME-TIFF (the default).""" +OME_ZARR = "ome-zarr" +"""The store of a copy written as OME-Zarr.""" +STORES = (OME_TIFF, OME_ZARR) +ZARR_COPY_SUFFIX = ".ome.zarr" +NGFF_VERSION = "0.4" +_AXIS_TYPES = {"T": "time", "C": "channel", "Z": "space", "Y": "space", "X": "space"} +BITS_NOT_WRITTEN = "OME-NGFF 0.4 has no field for significant bits" +OMERO_COLOUR = "FFFFFF" +"""The ``color`` of every ``omero`` channel (required by OME-NGFF 0.4; display only).""" + + +def zarr_copy_name(relative_path: str) -> str: + """Return the OME-Zarr copy's path for an image: the same folder and position ID.""" + folder, _, _name = relative_path.rpartition("/") + name = position_id(relative_path) + ZARR_COPY_SUFFIX + return f"{folder}/{name}" if folder else name + + +def as_zarr_copy(planned: PlannedCopy) -> PlannedCopy: + """Return a planned OME-TIFF copy as the same copy written as OME-Zarr. + + A byte-for-byte copy (an excluded image, an OME-Zarr image) is left as it is. The entry + takes the copy's name and keeps the source's path; significant bits are not carried. + """ + if planned.layout is None: + return planned + source = planned.entry.relative_path + entry = replace( + planned.entry, + relative_path=zarr_copy_name(source), + source_relative_path=source, + significant_bits=None, + ) + layout = replace(planned.layout, significant_bits=None) + return replace(planned, entry=entry, layout=layout, store=OME_ZARR) + + +def _window(dtype: np.dtype) -> dict: + """Return an ``omero`` channel window: the pixel type's whole range (display only). + + OME-NGFF 0.4 requires a ``window`` and a ``color`` for every ``omero`` channel. Neither is + read from the pixels: the window is the type's own range, so nothing about the data (no + contrast, no normalization) is stated, and the same copy stays the same bytes. + """ + if dtype.kind in "iu": + low, high = int(np.iinfo(dtype).min), int(np.iinfo(dtype).max) + elif dtype.kind == "f": + low, high = float(np.finfo(dtype).min), float(np.finfo(dtype).max) + else: # a boolean image: off and on + low, high = 0, 1 + return {"min": low, "max": high, "start": low, "end": high} + + +def _metadata(layout: CopyLayout, dtype: np.dtype, description: str | None) -> dict: + """Return the group's attributes: OME-NGFF 0.4 multiscales, channel names, description.""" + sizes = {size.role: size for size in layout.sizes} + axes, scale, unknown = [], [], [] + for role in layout.axes: + axis = {"name": role.lower(), "type": _AXIS_TYPES[role]} + size = sizes.get(role) + if role != "C" and (size is None or size.value is None): + unknown.append(role.lower()) + if size is not None and size.value is not None: + axis["unit"] = size.unit + axes.append(axis) + scale.append(float(size.value) if size is not None and size.value is not None else 1.0) + multiscale: dict = { + "version": NGFF_VERSION, + "axes": axes, + "datasets": [ + {"path": "0", "coordinateTransformations": [{"type": "scale", "scale": scale}]} + ], + } + if unknown: + multiscale["metadata"] = {UNKNOWN_SCALE_KEY: unknown} + attrs: dict = {"multiscales": [multiscale]} + if layout.channel_names: + window = _window(dtype) + attrs["omero"] = { + "channels": [ + {"color": OMERO_COLOUR, "label": name, "window": dict(window)} + for name in layout.channel_names + ] + } + if description: + attrs["morphobricks"] = {"description": description} + return attrs + + +def _json(path: Path, data: dict) -> None: + """Write metadata as sorted JSON with LF line endings (the same data, the same bytes).""" + text = json.dumps(data, indent=1, sort_keys=True, ensure_ascii=False) + "\n" + path.write_bytes(text.encode("utf-8")) + + +def _lay_out(target: Path, layout: CopyLayout, dtype: np.dtype, description: str | None) -> Path: + """Write the store's metadata and return the folder of the array's chunks.""" + target.mkdir(parents=True) + array = target / "0" + array.mkdir() + _json(target / ".zgroup", {"zarr_format": 2}) + _json(target / ".zattrs", _metadata(layout, dtype, description)) + rows, cols = layout.plane_shape + stored = dtype.newbyteorder("<") if dtype.itemsize > 1 else dtype + _json( + array / ".zarray", + { + "zarr_format": 2, + "shape": list(layout.shape), + "chunks": [*([1] * (len(layout.shape) - 2)), rows, cols], + "dtype": stored.str, + "compressor": None, + "filters": None, + "fill_value": 0, + "order": "C", + "dimension_separator": "/", + }, + ) + return array + + +def _chunk_path(array: Path, position: tuple[int, ...]) -> Path: + """Return the file of the plane at ``position`` (its leading indices), making its folder.""" + path = array.joinpath(*(str(i) for i in position), "0", "0") + path.parent.mkdir(parents=True, exist_ok=True) + return path + + +def _little(pixels: np.ndarray, dtype: np.dtype) -> np.ndarray: + """Return pixels of exactly ``dtype`` as contiguous little-endian values; never a cast.""" + if pixels.dtype != dtype: + raise CopyWriteError(f"a plane is {pixels.dtype}, not {dtype} as its header said") + stored = dtype.newbyteorder("<") if dtype.itemsize > 1 else dtype + return np.ascontiguousarray(pixels).astype(stored, copy=False) + + +def write_named_zarr( + source: str | os.PathLike, + target: str | os.PathLike, + layout: CopyLayout, + mapping: AxisMapping, + *, + dtype: np.dtype, + memory_limit: int = DEFAULT_MEMORY_LIMIT, +) -> None: + """Write ``source``, read under ``mapping``, to ``target`` as an OME-Zarr image. + + Parameters + ---------- + source + The TIFF image. + target + The folder to write (the caller's ``.partial`` folder); it must not exist. + layout + The copy's axes, shape and metadata (:func:`.copy_axes.copy_layout`). + mapping + The image's own confirmed mapping. + dtype + The pixel type the inspected header gives; a plane of another type is refused. + memory_limit + Working-memory limit per plane, as the run's. + + Raises + ------ + PlaneReadError + If the source cannot be read under ``mapping`` within the memory limit. + CopyWriteError + If a plane is not of ``dtype``. + OSError + If the store cannot be written. + """ + dtype = np.dtype(dtype) + array = _lay_out(Path(target), layout, dtype, layout.description) + rows, cols = layout.plane_shape + roles = [role.lower() for role in layout.axes[:-2]] + row_nbytes = cols * dtype.itemsize + for plane in iter_planes(source, mapping, memory_limit=memory_limit): + position = tuple(int(plane.coordinates[role]) for role in roles) + with open(_chunk_path(array, position), "wb") as stream: + stream.truncate(rows * row_nbytes) + for block in plane.blocks(): + pixels = _little(block.pixels, dtype) + for r in range(pixels.shape[0]): + stream.seek((block.y + r) * row_nbytes + block.x * dtype.itemsize) + stream.write(memoryview(pixels[r]).cast("B")) + + +def write_projected_zarr( + source: str | os.PathLike, + target: str | os.PathLike, + layout: CopyLayout, + mapping: AxisMapping, + details: ProjectionEntry, + *, + memory_limit: int = DEFAULT_MEMORY_LIMIT, +) -> None: + """Write the Z projection of ``source`` to ``target`` as a 2-D OME-Zarr image. + + The planes are computed exactly as :func:`.ome_tiff_copy.write_projected_copy` computes + them (the slices ``details.z_range`` of each (t, c), ascending, into one running plane); + the store's ``morphobricks`` attribute says how the copy was made. + + Raises + ------ + PlaneReadError, CopyWriteError, ProjectionRefused, OSError + As :func:`.ome_tiff_copy.write_projected_copy`. + """ + start, stop = details.z_range + out = np.dtype(details.output_dtype) + array = _lay_out(Path(target), layout, out, projection_description(details)) + roles = [role.lower() for role in layout.axes[:-2]] + has_z = mapping.index_of("Z") is not None + for position in np.ndindex(*layout.shape[:-2]): + where = dict(zip(roles, position, strict=True)) + running = PlaneAccumulator( + layout.plane_shape, details.source_dtype, details.method, stop - start + ) + for z in range(start, stop): + plane = read_plane( + source, + mapping, + z=z if has_z else None, + c=where.get("c"), + t=where.get("t"), + memory_limit=memory_limit, + ) + if plane.pixels is None: + raise CopyWriteError(f"a plane of {source} does not fit the memory limit") + running.add(plane.pixels) + pixels = _little(running.result(), out) + _chunk_path(array, tuple(int(i) for i in position)).write_bytes(pixels.tobytes()) + + +__all__ = [ + "BITS_NOT_WRITTEN", + "NGFF_VERSION", + "OMERO_COLOUR", + "OME_TIFF", + "OME_ZARR", + "STORES", + "ZARR_COPY_SUFFIX", + "as_zarr_copy", + "write_named_zarr", + "write_projected_zarr", + "zarr_copy_name", +] diff --git a/src/morphobricks_preprocessing/planned_copy.py b/src/morphobricks_preprocessing/planned_copy.py index ac668ac..1d3d501 100644 --- a/src/morphobricks_preprocessing/planned_copy.py +++ b/src/morphobricks_preprocessing/planned_copy.py @@ -28,6 +28,9 @@ class PlannedCopy: dtype The pixel type a converted or projected image's header gives (native byte order, the source's: a projected copy's own type is in its entry), else None. + store + ``"ome-tiff"`` (a named-axis OME-TIFF) or ``"ome-zarr"`` (:mod:`.ome_zarr_copy`) for + a converted or projected image. """ entry: CopyEntry @@ -35,8 +38,14 @@ class PlannedCopy: mapping: AxisMapping | None = None nbytes: int = 0 dtype: np.dtype | None = None + store: str = "ome-tiff" + + @property + def source_path(self) -> str: + """The source image, relative to the experiment (the copy's path unless renamed).""" + return self.entry.source_path @property def relative_path(self) -> str: - """The image, relative to the experiment and to the copy tree.""" + """The copy, relative to the copy tree (the source's path too unless renamed).""" return self.entry.relative_path diff --git a/tests/test_ome_zarr_copy.py b/tests/test_ome_zarr_copy.py new file mode 100644 index 0000000..74bd2a0 --- /dev/null +++ b/tests/test_ome_zarr_copy.py @@ -0,0 +1,286 @@ +"""OME-Zarr copies: a recorded conversion, asked on the first screen (owner, 11 November 2026). + +``convert_experiment(store="ome-zarr")`` writes every image that is read as an OME-Zarr image +(OME-NGFF 0.4), plane for plane under the confirmed roles, with the conversion record; the +notebook offers it on the Axes step once the roles are confirmed, when the option of the first +screen is ticked. The originals are only read; records without OME-Zarr copies are unchanged. +""" + +import hashlib +import html +import json +import os +import re + +import numpy as np +import pytest +import tifffile + +pytest.importorskip("zarr") + +from morphobricks_preprocessing.axes import prefill_mapping # noqa: E402 +from morphobricks_preprocessing.axis_review import review_axes # noqa: E402 +from morphobricks_preprocessing.conversion import convert_experiment # noqa: E402 +from morphobricks_preprocessing.conversion_record import ( # noqa: E402 + RECORD_FILENAME, + CopyEntry, + read_record, +) +from morphobricks_preprocessing.ome_zarr_copy import zarr_copy_name # noqa: E402 +from morphobricks_preprocessing.planes import iter_planes # noqa: E402 +from morphobricks_preprocessing.projection import Projection # noqa: E402 +from morphobricks_preprocessing.run import preflight # noqa: E402 +from test_notebook_check_save import FOLDERS, answer_all # noqa: E402 +from test_notebook_projection import NAMES, digests, write_stacks # noqa: E402 + + +def _text(text): + return re.sub(r"\s+", " ", html.unescape(re.sub(r"<[^>]+>", " ", text))).strip() + + +def read_copy(dest, rel): + """Return the copy's array as the brick reads it, in the copy's axes order.""" + pre = preflight(dest, dest.parent / "check_out") + assert not [i for i in pre.issues if i.is_error], pre.issues + meta = next(f for f in pre.metadata.files if f.relative_path == rel) + mapping = review_axes(pre.metadata, pre.suggested_mapping).file_mappings[rel] + out = np.zeros(meta.shape, meta.dtype) + roles = [r.lower() for r in meta.axes[:-2]] + for plane in iter_planes(dest / rel, mapping): + out[tuple(plane.coordinates[r] for r in roles)] = plane.pixels + return meta, out + + +def test_a_copy_keeps_the_folder_and_the_position_id(): + assert zarr_copy_name("ctrl/rep1/fov_01.tif") == "ctrl/rep1/fov_01.ome.zarr" + assert zarr_copy_name("fov.ome.tiff") == "fov.ome.zarr" + + +def test_every_image_is_written_plane_for_plane_with_its_metadata_and_recorded(tmp_path): + root, dest = tmp_path / "exp", tmp_path / "copies" + data = write_stacks(root, "TCYX", (2, 3, 16, 20)) + before = digests(root) + result = convert_experiment(root, prefill_mapping("TCYX"), dest, store="ome-zarr") + assert result.written and not result.issues + assert list(result.converted) == [f"{f}/a.ome.zarr" for f in FOLDERS] # all of them + assert digests(root) == before + meta, copy = read_copy(dest, "ctrl/rep1/a.ome.zarr") + assert meta.axes == "TCYX" and np.array_equal(copy, data["ctrl/rep1/a.tif"]) + assert meta.channel_names == tuple(NAMES) + assert (meta.physical_sizes.x, meta.physical_sizes.x_unit) == (0.3, "micrometer") + attrs = json.loads((dest / "ctrl/rep1/a.ome.zarr/.zattrs").read_text(encoding="utf-8")) + assert attrs["multiscales"][0]["version"] == "0.4" + # The time step is not in the header: a placeholder scale, named as unknown, read as none. + assert attrs["multiscales"][0]["metadata"] == {"unknown_scale": ["t"]} + assert meta.physical_sizes.t is None + record, sha256 = read_record(dest) + entry = record.entry("ctrl/rep1/a.ome.zarr") + assert ( + entry.source_relative_path == "ctrl/rep1/a.tif" and entry.source_path == "ctrl/rep1/a.tif" + ) + assert entry.action == "converted" and entry.copy_axes == "TCYX" + assert entry.significant_bits is None and entry.copy_sha256 + assert result.record_sha256 == sha256 + # The same conversion again writes nothing and keeps the record. + again = convert_experiment(root, prefill_mapping("TCYX"), dest, store="ome-zarr") + assert again.written and len(again.reused) == len(FOLDERS) and again.record_sha256 == sha256 + + +def test_the_same_copy_is_the_same_bytes(tmp_path): + root = tmp_path / "exp" + write_stacks(root, "CYX", (3, 16, 16)) + + def tree(dest): + convert_experiment(root, prefill_mapping("CYX"), dest, store="ome-zarr") + store = dest / "ctrl/rep1/a.ome.zarr" + return { + p.relative_to(store).as_posix(): hashlib.sha256(p.read_bytes()).hexdigest() + for p in sorted(store.rglob("*")) + if p.is_file() + } + + first = tree(tmp_path / "one") + assert first == tree(tmp_path / "two") and "0/0/0/0" in first and ".zarray" not in first + assert "0/.zarray" in first + + +def test_a_projection_can_be_written_as_ome_zarr(tmp_path): + root, dest = tmp_path / "exp", tmp_path / "copies" + data = write_stacks(root, "ZCYX", (4, 3, 16, 16)) + result = convert_experiment( + root, prefill_mapping("ZCYX"), dest, projection=Projection("max"), store="ome-zarr" + ) + assert result.written and len(result.projected) == len(FOLDERS) + meta, copy = read_copy(dest, "drug/rep2/a.ome.zarr") + assert meta.axes == "CYX" and np.array_equal(copy, data["drug/rep2/a.tif"].max(axis=0)) + entry = read_record(dest)[0].entry("drug/rep2/a.ome.zarr") + assert entry.projection.source_relative_path == "drug/rep2/a.tif" + attrs = json.loads((dest / "drug/rep2/a.ome.zarr/.zattrs").read_text(encoding="utf-8")) + assert "max" in attrs["morphobricks"]["description"] + + +def test_a_changed_source_is_named_by_the_check_of_the_copies(tmp_path): + root, dest = tmp_path / "exp", tmp_path / "copies" + write_stacks(root, "CYX", (3, 16, 16)) + convert_experiment(root, prefill_mapping("CYX"), dest, store="ome-zarr") + clean = preflight(dest, tmp_path / "out") + assert clean.conversion is not None and not clean.conversion.issues + path = root / "ctrl/rep1/a.tif" + stat = path.stat() + os.utime(path, ns=(stat.st_atime_ns, stat.st_mtime_ns + 10**10)) + after = preflight(dest, tmp_path / "out") + assert [i.code for i in after.conversion.issues] == ["conversion_source_changed"] + assert "ctrl/rep1/a.tif" in str(after.conversion.issues[0].paths) # the source's own path + + +def test_an_unknown_store_is_refused_before_anything_is_read(tmp_path): + root, dest = tmp_path / "exp", tmp_path / "copies" + for folder, name in (("ctrl/rep1", "a.tif"), ("ctrl/rep1", "b.tif"), ("drug/rep1", "a.tif")): + path = root / folder / name + path.parent.mkdir(parents=True, exist_ok=True) + tifffile.imwrite(path, np.zeros((2, 8, 8), np.uint8), ome=True, metadata={"axes": "CYX"}) + with pytest.raises(ValueError, match="store must be one of"): + convert_experiment(root, prefill_mapping("CYX"), dest, store="zarr") + assert not dest.exists() + assert convert_experiment(root, prefill_mapping("CYX"), dest, store="ome-zarr").written + + +def test_two_images_that_would_share_a_copy_name_stop_the_plan(): + from morphobricks_preprocessing.conversion_plan import _stored + from morphobricks_preprocessing.copy_axes import CopyLayout + from morphobricks_preprocessing.fingerprint import Fingerprint + from morphobricks_preprocessing.planned_copy import PlannedCopy + + def planned(rel): + entry = CopyEntry(rel, "converted", None, Fingerprint(1, 1), copy_axes="YX") + return PlannedCopy(entry, CopyLayout("YX", (4, 4))) + + kept, issues = _stored((planned("c/r/a.tif"), planned("c/r/A.tiff")), [], True) + assert kept == () and [i.code for i in issues] == ["copy_names_collide"] + assert set(issues[0].paths) == {"c/r/a.tif", "c/r/A.tiff"} # the sources, by their own names + kept, issues = _stored((planned("c/r/a.tif"), planned("c/r/b.tif")), [], True) + assert [k.relative_path for k in kept] == ["c/r/a.ome.zarr", "c/r/b.ome.zarr"] and not issues + + +def test_records_without_ome_zarr_copies_are_written_as_before(tmp_path): + root, dest = tmp_path / "exp", tmp_path / "copies" + write_stacks(root, "ZCYX", (4, 3, 16, 16)) + convert_experiment(root, prefill_mapping("ZCYX"), dest, projection=Projection("max")) + text = (dest / RECORD_FILENAME).read_text(encoding="utf-8") + assert "source_relative_path" in text # the projection's own, as before + record = read_record(dest)[0] + assert all(e.source_relative_path is None for e in record.files) + assert all("source_relative_path" not in e.to_dict() for e in record.files) + entry = record.files[0] + assert CopyEntry.from_dict(entry.to_dict()) == entry + + +# --- the notebook -------------------------------------------------------------------------------- + +pytest.importorskip("ipywidgets") +pytest.importorskip("matplotlib") + +from morphobricks_preprocessing import notebook_style as ui # noqa: E402 +from morphobricks_preprocessing.notebook_ui import PreprocessingNotebook # noqa: E402 +from morphobricks_preprocessing.notebook_zarr_copy import OPTION_LABEL # noqa: E402 + + +def started(tmp_path, axes, shape, *, option=True): + root, out = tmp_path / "exp", tmp_path / "out" + data = write_stacks(root, axes, shape) + nb = PreprocessingNotebook(str(root), output_folder=str(out)) + nb.steps() + nb.zarr_option.value = option + nb.scan_button.click() + nb.check_button.click() + return nb, root, out, data + + +def test_the_option_is_on_the_first_screen_unticked_and_writes_nothing(tmp_path): + nb, root, out, _ = started(tmp_path, "CYX", (3, 16, 16), option=False) + assert nb.zarr_option.description == OPTION_LABEL and not nb.zarr_option.value + assert nb.zarr_option_row in nb.folders_card.children + assert not ui.is_visible(nb.zarr_card) + nb.quick_confirm.click() + assert not ui.is_visible(nb.zarr_card) and nb._write_zarr_copy() is None + nb.zarr_option.value = True # ticked later: offered at once, the roles being confirmed + assert ui.is_visible(nb.zarr_card) and not nb.zarr_button.disabled + assert not out.joinpath("ome_zarr").exists() + + +def test_the_copy_waits_for_the_confirmed_axes_and_says_so(tmp_path): + nb, root, out, _ = started(tmp_path, "CYX", (3, 16, 16)) + assert ui.is_visible(nb.zarr_card) and nb.zarr_button.disabled + assert "the axis roles are not confirmed" in _text(nb.zarr_blocker.value) + assert nb._write_zarr_copy() is None and not out.joinpath("ome_zarr").exists() + nb.quick_confirm.click() + assert not nb.zarr_button.disabled and nb.zarr_blocker.value == "" + assert str(out / "ome_zarr") in nb.zarr_where.value + + +def test_the_copy_is_written_recorded_and_used_by_the_later_steps(tmp_path): + nb, root, out, data = started(tmp_path, "TCYX", (2, 3, 16, 16)) + before = digests(root) + nb.quick_confirm.click() + result = nb._write_zarr_copy() + copies = out / "ome_zarr" + assert result.written and len(result.converted) == len(FOLDERS) + assert nb.folder.value == str(copies) and nb.output.value == str(out) + assert nb.preflight.discovery.root == copies and nb.confirmed_mapping is None + assert all(f.is_ome_zarr for f in nb.preflight.metadata.files) + assert "recorded copies of" in _text(nb.copy_note.value) + assert root.resolve().as_posix() in nb.copy_note.value # as the record spells it + assert "OME-Zarr already" in _text(nb.zarr_where.value) # nothing more to write + assert nb.scan_viewer.view is not None # the viewer reads the copies + nb.quick_confirm.click() + answer_all(nb) + assert nb.save_experiment() + run = nb.run() + assert run.written and run.plane_count == len(FOLDERS) * 2 + assert digests(root) == before + record = read_record(copies)[0] + assert record.source_root == root.resolve().as_posix() + assert {e.source_path for e in record.files} == set(data) + + +def test_ticked_with_z_stacks_the_projection_is_written_as_ome_zarr(tmp_path): + nb, root, out, data = started(tmp_path, "ZCYX", (4, 3, 16, 16)) + assert ui.is_visible(nb.projection_card) and not ui.is_visible(nb.zarr_card) + nb.projection_method.value = "max" + nb.preview_checked.button.click() + assert "as OME-Zarr" in _text(nb.projection_where.value) + result = nb._write_projection() + copies = out / "projected_max_zarr" + assert result.written and result.destination == copies + assert (copies / "ctrl/rep1/a.ome.zarr/.zattrs").is_file() + assert nb.folder.value == str(copies) and "Z" not in nb.mapping.roles + nb.quick_confirm.click() + answer_all(nb) + assert nb.save_experiment() and nb.run().written + + +def test_the_omero_channels_have_what_ome_ngff_0_4_requires(tmp_path): + # Review of #287: the OME-NGFF 0.4 image schema requires a ``window`` (start, end, min, + # max) and a ``color`` for every omero channel; a label alone is not valid 0.4. They are + # display only: white and the pixel type's whole range, never read from the pixels. + root, dest = tmp_path / "exp", tmp_path / "copies" + write_stacks(root, "TCYX", (2, 3, 16, 20)) + assert convert_experiment(root, prefill_mapping("TCYX"), dest, store="ome-zarr").written + attrs = json.loads((dest / "ctrl/rep1/a.ome.zarr/.zattrs").read_text(encoding="utf-8")) + channels = attrs["omero"]["channels"] + assert [c["label"] for c in channels] == NAMES + for channel in channels: + assert channel["color"] == "FFFFFF" + assert channel["window"] == {"min": 0, "max": 65535, "start": 0, "end": 65535} + meta, _copy = read_copy(dest, "ctrl/rep1/a.ome.zarr") + assert meta.channel_names == tuple(NAMES) # the brick still reads the labels + stacks, projected = tmp_path / "stacks", tmp_path / "projected" + write_stacks(stacks, "ZCYX", (4, 3, 16, 16)) + mean = Projection("mean") # written as 64-bit floats: the float64 range + assert convert_experiment( + stacks, prefill_mapping("ZCYX"), projected, projection=mean, store="ome-zarr" + ).written + attrs = json.loads((projected / "ctrl/rep1/a.ome.zarr/.zattrs").read_text(encoding="utf-8")) + top = float(np.finfo(np.float64).max) + for channel in attrs["omero"]["channels"]: + assert channel["window"] == {"min": -top, "max": top, "start": -top, "end": top}