diff --git a/AGENTS.md b/AGENTS.md index 312ed122..42da3ba7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,12 +35,12 @@ brick**. `morphobricks_normalization` (pinned to its release v0.3.2; first pinned at v0.2.0, plan §12 questions 27 and 28). This brick runs them over the experiment. Later bricks such as patch extraction use them to normalise any group of - planes without rereading pixels. The old names stay importable from here, deprecated, - for one version (follow-up P1, normalization plan N6). + planes without rereading pixels. Import them from `morphobricks_normalization`; the + deprecated paths through this package were removed in v0.4.0. - **Never:** modifies source images, writes inside the experiment folder, or applies a normalization itself. -- **Optional (`zarr` extra, plan §12 question 12):** OME-Zarr input, and converting TIFF - images to OME-Zarr in the output folder before processing them (contract §10). +- **Optional (`zarr` extra, plan §12 question 12):** OME-Zarr input. (Converting TIFF images + to OME-Zarr in the output folder, `convert_tiffs`, was removed in v0.4.0.) - **Large planes (plan §12 question 15):** planes over the memory limit are read in blocks, automatically and announced, with identical rows. - **Out of scope:** spatial corrections (flat-field, BaSiC, CLAHE). @@ -71,7 +71,7 @@ src/morphobricks_preprocessing/ physical_sizes.py PhysicalSizes (pixel sizes, z-step, time step, a unit per axis, micrometers() for comparing), space_unit/to_micrometers (the shared unit rule, contract §16), read_physical_sizes (OME / ImageJ TIFF metadata; also importable - from zarr_export) and ngff_physical_sizes (OME-NGFF scale and axis units) + ) and ngff_physical_sizes (OME-NGFF scale and axis units) dataset.py dataset consistency from the folders: biological repeats per condition, balance table, unbalanced_images, repeat_missing_conditions, CONSISTENCY_CODES (P2) design.py experiment design (P3): ExperimentDesign/DesignRow, default pairing, reconcile with @@ -85,9 +85,7 @@ src/morphobricks_preprocessing/ the notebook's plots, intensity_text for the CLI validation.py ValidationIssue (severity, code, message, paths, action), error()/warning() builders axes.py AxisMapping (role per dimension), prefill, check_mapping, review_axes (one mapping - applied to every file, missing axes as singletons only when confirmed), - check_declared_axes and declared_axes_transposed (deprecated; removed - before the v0.4.0 release) + applied to every file, missing axes as singletons only when confirmed) axis_check.py the axis check (contract §2): header_axis_issue, axis_disagreements, likely_axis_cause, check_header_axes; AXIS_ERROR_CODES, AXIS_CHECK_CODES planes.py plane-by-plane reader (iter_planes, read_plane, read_sample_plane), table rows as @@ -96,10 +94,7 @@ src/morphobricks_preprocessing/ check_planes (preflight memory / layout errors); planes over the limit read in blocks of TIFF strips/tiles or zarr chunks (Plane.blocks, BlockReader), reduced previews - _moved.py deprecated access to the code moved to morphobricks_normalization: the moved - names, warn-once DeprecationWarning, __getattr__ helpers - dtypes.py, histogram.py, records.py, csv_io.py, pooling.py, normalization.py - deprecated stand-ins (module __getattr__ only, no code) for one version + _io.py write_text_atomic (every file the brick writes) and free_space manifest.py RunManifest / RunSettings / FileEntry, JSON read/write, SHA-256 of the CSV, check_sources (images against a run, for other bricks, contract §16) run.py preflight(), run_experiment() (incremental), sample_planes() for previews @@ -120,9 +115,6 @@ src/morphobricks_preprocessing/ letters and physical sizes from the scale and axis units), ngff_label_names/open_ngff_labels (label images, for later bricks), zarr 2/3 helpers, canonical_dtype; needs the `zarr` extra - zarr_export.py TIFF -> OME-Zarr conversion (OmeZarrWriter, OME-NGFF 0.4, plane by plane or - block by block), - ConversionSettings (chunk size, compression, pyramid levels), estimate_conversion notebook_style.py notebook look and feel: cards, status cards, tiles, tables, ConfirmToggle, Browse dialogs notebook_axes.py the notebook's axis-check card and the marks under the role dropdowns notebook_ui.py PreprocessingNotebook: the four notebook steps (needs the `notebook` extra); @@ -150,8 +142,7 @@ Moved to [morphobricks-normalization](https://github.com/CellMigrationLab/morpho ``` Their tests (`test_histogram.py`, `test_csv_io.py`, `test_pooling.py`, -`test_normalization.py`) moved with them. `tests/test_moved.py` checks the deprecated -names here. +`test_normalization.py`) moved with them. The deprecated names here were removed in v0.4.0. The notebook and the CLI contain **no processing logic**. They call the same public functions the tests use. @@ -478,7 +469,7 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p |---|---| | Binning | Per-plane min–max, same bin count per experiment, pooled counts for groups (binning study) | | Core dependencies | `numpy`, `tifffile`, `scipy` (the mask step's piece counts, Q32) and `morphobricks-normalization` (git dependency pinned to its release tag v0.3.2, whose CSV reader checks each row's stored `plane_id`, `image_file` and `bit_depth`; `[tool.hatch.metadata] allow-direct-references`). `ipywidgets` and `matplotlib` in the `notebook` extra; `zarr` in the `zarr` extra; `pytest` and `ruff` in `dev` | -| OME-Zarr (Q12) | Input may be TIFF and/or OME-Zarr (folders ending in `.zarr`), mixed. OME-Zarr axes come from their metadata and are matched by role; level 0 of a single image only, several images per store is `multiple_series`. TIFF choice once per experiment (`convert_tiffs`): read directly (default) or convert to `/ome_zarr` (OME-NGFF 0.4) and compute rows from the copies; switching recomputes TIFF rows and removes unused copies. Reads OME-NGFF 0.4 and 0.5 (0.5 needs zarr 3). In the notebook: its own step-3 card "(Optional) Image conversion" after Settings (Q13): radio buttons (read directly = recommended), a disk-space warning (amber, red when the estimate exceeds free space, never blocking), and conversion settings (chunk size, compression + level, pyramid levels) shown only when converting. CLI `--convert-tiff` with `--chunk-size`, `--compression`, `--compression-level`, `--pyramid-levels`. No overlap setting: chunks never overlap | +| OME-Zarr (Q12) | Input may be TIFF and/or OME-Zarr (folders ending in `.zarr`), mixed. OME-Zarr axes come from their metadata and are matched by role; level 0 of a single image only, several images per store is `multiple_series`. *Removed in v0.4.0 (plan Q46): TIFF images are always read directly; the following describes v0.3.* TIFF choice once per experiment (`convert_tiffs`): read directly (default) or convert to `/ome_zarr` (OME-NGFF 0.4) and compute rows from the copies; switching recomputes TIFF rows and removes unused copies. Reads OME-NGFF 0.4 and 0.5 (0.5 needs zarr 3). In the notebook: its own step-3 card "(Optional) Image conversion" after Settings (Q13): radio buttons (read directly = recommended), a disk-space warning (amber, red when the estimate exceeds free space, never blocking), and conversion settings (chunk size, compression + level, pyramid levels) shown only when converting. CLI `--convert-tiff` with `--chunk-size`, `--compression`, `--compression-level`, `--pyramid-levels`. No overlap setting: chunks never overlap | | System files (Q5) | Ignored silently; `single` check and sibling uniqueness are case-insensitive | | Too-deep folders (Q7) | Error | | Folders named `single` (Q23) | Accepted as the level's value `single` when lower case and the only folder in its place (MorphoBricks one-member level, MorphoRelate M3); another case or other folders beside it stay `reserved_folder_name` | @@ -492,8 +483,8 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p | Corrupt files (Q6) | Explicit exclusion list, recorded in manifest, rows removed | | Notebook target | Local Jupyter (not Colab-first) | | CLI | `check ` (summary, label tree, issues, suggested `--axes`) and `run --axes ROLES` with `--output`, `--exclude`, `--bins`, `--histogram-range`, `--robust-quantiles`, `--memory-limit 2GiB`, `--missing-as-singleton`, `--force-rebuild`, `--log`, and the conversion options above. Axis roles are always stated explicitly, never taken silently from metadata. Exit 1 on errors | -| Large planes (Q15) | Planes over the memory limit are read in blocks automatically, announced by the warning `planes_read_in_blocks` and `read_in_blocks` in the manifest. Block size derived from the limit (whole strips/tiles/chunks, uncompressed pieces and chunks also by rows). Compressed single-strip/tile planes keep the `plane_over_memory_limit` error. Two-pass histograms identical to whole planes. OME-Zarr copies written block by block, smaller levels built from the stored level above. Axis preview reduced (every n-th pixel) with a caption saying so | -| Normalization brick (P1) | The modules `dtypes`, `histogram`, `records`, `csv_io`, `pooling` and `normalization` moved unchanged to `morphobricks_normalization` (normalization plan N1, N2, N6) and are imported from there. Their public names stay importable from `morphobricks_preprocessing` and from the old submodules for one version: module `__getattr__` stand-ins (`_moved.py`), one `DeprecationWarning` per old path naming `morphobricks_normalization..`, same objects. The package's own code never uses the old paths. Outputs unchanged (CSV schema 2); the manifest's `versions` adds `morphobricks-normalization`; Step 0 checks for it; contract §3–§7 text moved to the normalization contract (§1.2, now §6.2 for the CSV) | +| Large planes (Q15) | Planes over the memory limit are read in blocks automatically, announced by the warning `planes_read_in_blocks` and `read_in_blocks` in the manifest. Block size derived from the limit (whole strips/tiles/chunks, uncompressed pieces and chunks also by rows). Compressed single-strip/tile planes keep the `plane_over_memory_limit` error. Two-pass histograms identical to whole planes. (Up to v0.3, OME-Zarr copies were written block by block; conversion was removed in v0.4.0.) Axis preview reduced (every n-th pixel) with a caption saying so | +| Normalization brick (P1) | The modules `dtypes`, `histogram`, `records`, `csv_io`, `pooling` and `normalization` moved unchanged to `morphobricks_normalization` (normalization plan N1, N2, N6) and are imported from there. Up to v0.3 their public names stayed importable from `morphobricks_preprocessing` and the old submodules, with a `DeprecationWarning`; those stand-ins (`_moved.py`) were removed in v0.4.0 (plan Q46). The package's own code never uses the old paths. Outputs unchanged (CSV schema 2); the manifest's `versions` adds `morphobricks-normalization`; Step 0 checks for it; contract §3–§7 text moved to the normalization contract (§1.2, now §6.2 for the CSV) | | Dataset consistency (Q21, P2) | Step 1 "Dataset consistency" panel: balance table (images and technical-repeat folders per condition x biological repeat), `unbalanced_images` (< 0.5 or > 2 x median), `repeat_missing_conditions`, `different_pixel_sizes`, `missing_pixel_size` (OME-Zarr scale included), `different_plane_counts`; `different_sizes` narrowed to width/height; consistency issues shown only in that panel, not in Messages. Warnings only; outputs unchanged | | Experiment design (Q22, P3) | Step 1 after the labels: pairing dropdowns, one yellow "Yes, they are comparable" question per repeat group (single whole-experiment question without biological-repeat folders), control checkboxes; `experiment_design.csv` written after every answer; reruns load it and fold the questions behind a Change button; step 3 gated. CLI `run` reads it or `--design`, else writes a template and exits 1. `run_experiment(design=...)` optional; manifest `experiment_design_sha256` (optional key, version 1). Two folders of one condition in a group block that group | | Image type (Q29, P4) | Step 2 card after the axis roles: Fluorescence or RGB, suggestion preselected, confirmed with **Yes, this is the right image type**; RGB suggested for TIFF photometric RGB with three samples on the C axis; RGB needs three integer channels (`check_rgb`), else disabled with the reason and `rgb_not_possible` in a run. Manifest keys `image_type`, `channels_from_rgb_samples` (optional, version 1); type changes recompute nothing. CLI `--image-type`, default the printed suggestion | @@ -539,12 +530,9 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p - **No new core dependencies** without asking. tkinter (for folder dialogs) is optional and imported lazily with a fallback. - **Public API** is re-exported in `__init__.py`. `__all__` is sorted: constants, then - classes, then functions. It still lists the deprecated names moved to the normalization - brick; they are served by `__getattr__` (see `_moved.py`), never imported in - `__init__.py`. + classes, then functions. - **Code from the normalization brick:** import it as - `from morphobricks_normalization. import ...`, never through the deprecated - `morphobricks_preprocessing` paths (`tests/test_moved.py` checks this). + `from morphobricks_normalization. import ...`. ### Tests @@ -556,11 +544,10 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p - **Proving "no pixels are read":** monkeypatch `asarray` on `TiffFile`, `TiffPage`, `TiffFrame` and `TiffPageSeries` to raise. - **OME-Zarr test data:** `tests/zarr_tree.py` (`write_ngff`, `write_bioformats2raw`, - `write_plate`) writes stores with zarr directly, independent of `zarr_export`; 0.5 tests + `write_plate`) writes stores with zarr directly; 0.5 tests skip with zarr 2.18. - **Proving "one page at a time":** spy on `planes._read_page`. To prove "only changed - files are read", spy on `run.iter_xyc_planes` (`run.iter_planes` only converts TIFF - images to OME-Zarr). For planes in blocks, spy on + files are read", spy on `run.iter_xyc_planes`. For planes in blocks, spy on `planes._read_segment` and make `asarray` of every tifffile class fail. - **Planes in blocks with small test images:** the memory limit `plane_layout(path, mapping, memory_limit=1).block_memory_estimate` gives the smallest diff --git a/README.md b/README.md index 413c9b17..289156c1 100644 --- a/README.md +++ b/README.md @@ -54,12 +54,11 @@ Pre-alpha. Implemented and tested: From the normalization brick (`morphobricks_normalization`, a required dependency, pinned to its release v0.3.2): per-plane histograms with the robust range and the range check (`histogram`, `ranges`), the histogram table (`csv_io`, schema version 4), pooled group statistics -(`pooling`) and normalization parameters (`normalization`). Importing these -names from `morphobricks_preprocessing` still works for one version, with a -`DeprecationWarning`. +(`pooling`) and normalization parameters (`normalization`); import them from +`morphobricks_normalization` (the deprecated paths through `morphobricks_preprocessing` were +removed in v0.4.0). -Optional (the `zarr` extra): OME-Zarr images as input (`zarr_input`), and converting TIFF -images to OME-Zarr before processing them (`zarr_export`). **New users: start with the +Optional (the `zarr` extra): OME-Zarr images as input (`zarr_input`). **New users: start with the [user guide](docs/user_guide.md).** ## Installation diff --git a/docs/contract.md b/docs/contract.md index f62cf031..d15bece4 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -56,7 +56,7 @@ channel order; an image without a C axis has one channel. | OME-Zarr labels | The label images of an image's OME-NGFF `labels` group (0.4 and 0.5) are listed with `ngff_label_names` and opened, level 0, with `open_ngff_labels` (by name; the name may be left out when there is one). They are never analysed; later bricks such as patch extraction use them as masks. | | Experiment mapping | Defined on the TIFF arrangement when the experiment has TIFF files; in an experiment of OME-Zarr images only, on their arrangement with the most axes. | | Channel names, significant bits | Recorded in the run manifest when metadata provides them; not part of the CSV. Channel names that differ at the same channel index are the error `channel_names_differ` (issue #37): channels are matched by their number, so two stains at one number would be pooled as one channel. Only named channels are compared; a channel without a name in some images is not. Names are read without leading and trailing whitespace, for every image (TIFF, OME-Zarr and this brick's own copies), so `DAPI ` and `DAPI` are one name; the manifest records it (`channel_names_trimmed`, §9). Whitespace inside a name is compared exactly (`DA PI` and `DAPI` differ), with no override. An OME-Zarr channel label that is not text is malformed metadata: the image is unreadable (`unreadable_ome_zarr`), never read as an unnamed channel. | -| Physical sizes | Read with the header, not part of the CSV or the manifest, and compared between images in the folder check (§11, in micrometres with each axis's own unit; a size without a known unit cannot be compared): `PhysicalSizes` with the size of one step along X, Y, Z and T and each axis's unit (OME-NGFF unit names). TIFF: OME `PhysicalSize{X,Y,Z}` with their own units (micrometres when left out) and `TimeIncrement`, or ImageJ resolution, `spacing` and `finterval` with `unit`, `yunit`, `zunit`. OME-Zarr (0.4 and 0.5): the `scale` of level 0's `coordinateTransformations`, times the multiscales entry's own `scale` when given, with each axis's `unit` (common spellings such as `µm`, `um`, `micron` and `s` given as UDUNITS-2 names, others as written). Length units are the microscopy units of MorphoBricks decision D-a2 (§16): `pm`, `Å`/`angstrom`, `nm`, `µm` (`um`, `micron(s)`, `micrometer(s)`, `micrometre(s)`), `mm`, `cm`, `m`, and the names of each in US and British spelling (`nanometre`, ...); ImageJ's escaped micro sign (`\u00B5m`) is recognised. A size in any other unit (`inch`, `decimeter`, ...) is not converted to micrometres, so it cannot be compared: that axis is uncalibrated, and the unit as written is kept in `PhysicalSizes.unsupported_units` (axis letter and unit, information only, not part of equality) so that the folder check names it (`unsupported_pixel_size_unit`, §11). A plain TIFF's resolution tags (no OME or ImageJ metadata, for example 300 dpi with the unit inch) are never a pixel size (decision D-c): such an image has no calibration. Nothing is guessed: what the metadata does not give is None; a scale in a separate file (`path`) or not one number per axis gives no sizes; an OME-Zarr scale without a unit is kept with the unit None, except on the axes a multiscales entry's `metadata` lists as `unknown_scale` (this brick's copies, §10), whose sizes are None. | +| Physical sizes | Read with the header, not part of the CSV or the manifest, and compared between images in the folder check (§11, in micrometres with each axis's own unit; a size without a known unit cannot be compared): `PhysicalSizes` with the size of one step along X, Y, Z and T and each axis's unit (OME-NGFF unit names). TIFF: OME `PhysicalSize{X,Y,Z}` with their own units (micrometres when left out) and `TimeIncrement`, or ImageJ resolution, `spacing` and `finterval` with `unit`, `yunit`, `zunit`. OME-Zarr (0.4 and 0.5): the `scale` of level 0's `coordinateTransformations`, times the multiscales entry's own `scale` when given, with each axis's `unit` (common spellings such as `µm`, `um`, `micron` and `s` given as UDUNITS-2 names, others as written). Length units are the microscopy units of MorphoBricks decision D-a2 (§16): `pm`, `Å`/`angstrom`, `nm`, `µm` (`um`, `micron(s)`, `micrometer(s)`, `micrometre(s)`), `mm`, `cm`, `m`, and the names of each in US and British spelling (`nanometre`, ...); ImageJ's escaped micro sign (`\u00B5m`) is recognised. A size in any other unit (`inch`, `decimeter`, ...) is not converted to micrometres, so it cannot be compared: that axis is uncalibrated, and the unit as written is kept in `PhysicalSizes.unsupported_units` (axis letter and unit, information only, not part of equality) so that the folder check names it (`unsupported_pixel_size_unit`, §11). A plain TIFF's resolution tags (no OME or ImageJ metadata, for example 300 dpi with the unit inch) are never a pixel size (decision D-c): such an image has no calibration. Nothing is guessed: what the metadata does not give is None; a scale in a separate file (`path`) or not one number per axis gives no sizes; an OME-Zarr scale without a unit is kept with the unit None, except on the axes a multiscales entry's `metadata` lists as `unknown_scale` (copies written by this brick up to v0.3, §10), whose sizes are None. | | Plane counts | Images may have different numbers of depth slices, time points or planes along an unassigned dimension; this is a warning (§11). | | Channel counts | Every image of an experiment has the same number of channels (normalization plan N43): along the C axis, 1 without one (also when a missing C axis is confirmed as length 1). A different number is the error `different_channel_counts`, found before any image is read: in the folder check from the headers (the `C` dimension, or the RGB samples `S` when there is no `C`), and again with the confirmed axis roles, which decide (§11). | @@ -121,7 +121,7 @@ method is an offset and a scale, `(pixels − offset) / scale`. | Severity | Situations | Effect | |---|---|---| -| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2) or different channel names at the same channel number (`channel_names_differ`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; an image whose confirmed roles swap Y and X or contradict its header (`axes_yx_swapped`, `axes_contradict_header`, §2); plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); folders of one level differing only in case, anywhere in the experiment (`case_conflict`); folder or image names with outer whitespace or reading as a missing value (`invalid_name`, §1); images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; images in a folder that also holds subfolders (`tiff_outside_leaf`, §1); output folder inside the experiment; no usable planes; when masks are given, a masks folder missing or inside the experiment, mask names differing only in case, two masks of one type for an image, a mask whose name could belong to two images or types, or a mask that is unreadable, has axes other than Y, X and named Z/T (`mask_not_2d`), is of another size than its image (in `run`, its image's Y and X under the confirmed roles; not reported for an image with an axis error), is not of integers or booleans, has other depth slices or time points than the image's other masks or, in `run`, than the image (§15) | Processing is blocked | +| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2) or different channel names at the same channel number (`channel_names_differ`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; an image whose confirmed roles swap Y and X or contradict its header (`axes_yx_swapped`, `axes_contradict_header`, §2); plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); folders of one level differing only in case, anywhere in the experiment (`case_conflict`); folder or image names with outer whitespace or reading as a missing value (`invalid_name`, §1); images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; images in a folder that also holds subfolders (`tiff_outside_leaf`, §1); output folder inside the experiment; no usable planes; when masks are given, a masks folder missing or inside the experiment, mask names differing only in case, two masks of one type for an image, a mask whose name could belong to two images or types, or a mask that is unreadable, has axes other than Y, X and named Z/T (`mask_not_2d`), is of another size than its image (in `run`, its image's Y and X under the confirmed roles; not reported for an image with an axis error), is not of integers or booleans, has other depth slices or time points than the image's other masks or, in `run`, than the image (§15) | Processing is blocked | | Warning | Images whose metadata does not name their axes, read with the confirmed roles (`axes_not_named`, §2); images named as masks of other images, or folders named like a masks folder holding them (`image_named_as_mask`, `mask_folder_in_experiment`, §1); intensity checks of the written table (`image_intensity_outlier`, `repeat_intensity_shift`, `saturated_image`, `no_intensity_spread`, §14), reported apart from the run's issues; files that are not images; empty folders; images excluded by the user, or excluded paths that match no image; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; channels whose central intensity range covers too few histogram bins (`narrow_histogram_range`, §4); planes over the memory limit that are read in blocks; images without a mask of a type other images have, masks matching no analysed image, files outside the mask-type folders (§15) | Processing continues; the item is reported | Every message states what happened in plain language, lists the affected paths, and @@ -139,10 +139,7 @@ UTF-8 JSON next to `plane_histograms.csv`, written after the table and only when succeeded. Both files are prepared in a temporary folder before either earlier output is replaced. -OME-Zarr copies of TIFF images (§10), when requested, are written to the same temporary -folder and published first, so the manifest never names a copy that is not in place. Earlier files -are backed up before publication; earlier copies that are replaced or removed are moved -into the backup folder. A caught replacement failure restores the +Earlier files are backed up before publication. A caught replacement failure restores the previous state, including the absence of files on a first run. If restoration fails, the backup folder is retained and its location is reported for manual recovery. Failures to remove temporary files after a successful publication are logged. @@ -158,10 +155,10 @@ file's schema or statistical interpretation. | `written_at` | UTC time of writing, ISO 8601. | | `experiment_root` | Absolute path of the experiment folder, for information. | | `dtype` | Pixel type of the experiment. | -| `settings` | `bin_count`, `memory_limit` (bytes), `missing_as_singleton`, `exclude` (sorted relative paths), `convert_tiffs` (whether TIFF images are converted to OME-Zarr and processed from the copies; false when absent), `conversion` (`chunk_size`, `compression`, `compression_level`, `pyramid_levels` — null for automatic; defaults when absent), `histogram` (`HistogramSettings.to_dict()` of the normalization brick: `library_version`, `histogram_range`, `robust_quantiles`, `min_central_bins`, §4; the default `minmax` range when absent, in manifests written before follow-up P6). | +| `settings` | `bin_count`, `memory_limit` (bytes), `missing_as_singleton`, `exclude` (sorted relative paths), `histogram` (`HistogramSettings.to_dict()` of the normalization brick: `library_version`, `histogram_range`, `robust_quantiles`, `min_central_bins`, §4; the default `minmax` range when absent, in manifests written before follow-up P6). | | `axis_mapping` | `axes` (file axis labels) and `roles` (`X`, `Y`, `Z`, `C`, `T` or null per axis). | | `labels` | One entry per leaf folder: `folder` (relative path, `.` for the root) and its three labels. | -| `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count` (its rows in the table: XYC planes), `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `ome_zarr_path` (for a TIFF converted to OME-Zarr, the copy relative to the output folder; null otherwise or when the key is absent), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent), `position_id` (the image's position identifier, §1; null when absent, in manifests written before it), `channel_names_trimmed` (true when leading or trailing whitespace was removed from the image's channel names as they were read, §2; `channel_names` holds the trimmed names; false when absent), `roles` (the role of each of the file's `axes`, `X`, `Y`, `Z`, `C`, `T` or null, from the file's own mapping: an OME-Zarr image in another order than `axis_mapping` has its own roles, so later bricks read these rather than rebuild them from `axis_mapping`; null when absent, in manifests written before them; optional key, version 1). For an OME-Zarr image, `size` is the total size of its files and `mtime_ns` the time of its newest file. | +| `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count` (its rows in the table: XYC planes), `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent), `position_id` (the image's position identifier, §1; null when absent, in manifests written before it), `channel_names_trimmed` (true when leading or trailing whitespace was removed from the image's channel names as they were read, §2; `channel_names` holds the trimmed names; false when absent), `roles` (the role of each of the file's `axes`, `X`, `Y`, `Z`, `C`, `T` or null, from the file's own mapping: an OME-Zarr image in another order than `axis_mapping` has its own roles, so later bricks read these rather than rebuild them from `axis_mapping`; null when absent, in manifests written before them; optional key, version 1). For an OME-Zarr image, `size` is the total size of its files and `mtime_ns` the time of its newest file. | | `csv_sha256` | SHA-256 of the table written with this manifest. | | `experiment_design_sha256` | SHA-256 of the `experiment_design.csv` written with this run (§12), so later bricks can check that the design they read is the one used; null when the run had no design, or absent in manifests written before it. Optional key: the manifest stays version 1. | | `masks` | The masks paired with the images (§15), when the run was given a masks folder: `folder` (absolute path of the masks folder, null when only OME-Zarr label images were used), `types` (sorted mask-type names) and `files` (one entry per image and mask type, sorted by image then type: `image`, `type`, `path`, `source` `folder` or `ome-zarr`, `axes` and `shape`: the mask's axes and lengths without its length-1 axes other than Y and X, `YX` for a 2D mask (`[1, 512]` for a mask one pixel high), with `Z` and/or `T` for a mask with depth slices or time points, e.g. `TYX` and `[50, 512, 512]`; §15). Absent when the run had no masks. Optional key: the manifest stays version 1, and the table does not depend on the masks. `axes` and `shape` are absent in manifests written before them, whose masks are all 2D; a later brick must read `axes` before using a mask and never apply a 2D mask to several planes. | @@ -169,7 +166,8 @@ file's schema or statistical interpretation. | `channels_from_rgb_samples` | `true` when the C role is the RGB samples axis of images stored as RGB colour (§13), `false` otherwise; absent in manifests written before it. Optional key. | | `intensity_checks` | The limits of the intensity checks run on the table (§14): `IntensityCheckSettings.to_dict()` of the normalization brick (`library_version`, `outlier_limit`, `saturation_limit`, `min_images`); absent in manifests written before it. Optional key. Changing the limits recomputes nothing. | | `narrow_histogram_ranges` | The channel planes whose range check found the central range too narrow (§4), sorted by plane and channel: `relative_path`, `plane_id`, `channel` and the fields of the normalization brick's `RangeCheck` (`histogram_range`, `central_percentiles`, `central_bins`, `robust_quantiles`, `robust_limits`, `outside_robust_count`, `robust_central_bins`, `min_central_bins`). Empty when there are none; absent in manifests written before follow-up P6. Optional key: the manifest stays version 1. | -| Flags | `missing_as_singleton`, `convert_tiffs`, `read_in_blocks`, `channel_names_trimmed` and `channels_from_rgb_samples` are JSON booleans (`true`/`false`); any other value (`"false"`, `0`) makes the manifest malformed, never read as true or false, and `write_manifest` refuses to write one (`ValueError`), so a written manifest always reads back. | +| Removed keys | `settings.convert_tiffs`, `settings.conversion` and `files[].ome_zarr_path` were written by v0.3 runs that converted TIFF images to OME-Zarr in the output folder (removed in v0.4.0, §10); a manifest that holds them is still read, and they are ignored. The copies such a run left in `/ome_zarr` are not touched. | +| Flags | `missing_as_singleton`, `read_in_blocks`, `channel_names_trimmed` and `channels_from_rgb_samples` are JSON booleans (`true`/`false`); any other value (`"false"`, `0`) makes the manifest malformed, never read as true or false, and `write_manifest` refuses to write one (`ValueError`), so a written manifest always reads back. | | `versions` | Versions of `morphobricks-preprocessing`, `morphobricks-normalization` (which computes the histograms and writes the CSV; absent in manifests written before it), `numpy`, `tifffile` and Python. Information only: a different version never causes a rebuild. | Channel names and significant bits come from OME metadata. Significant bits are for @@ -177,35 +175,14 @@ information only; channel names must agree between images at each channel number later bricks that name channels should check their names against `channel_names` when it is not empty. -## 10. Converting TIFF images to OME-Zarr (optional) +## 10. Converting TIFF images to OME-Zarr (removed in v0.4.0) -When `convert_tiffs` is on, every TIFF image is converted to OME-Zarr in the output folder -and its rows are computed from the converted copy, like for any OME-Zarr image. The copy -holds the source values, so the rows equal those of reading the TIFF directly; the -`source_relative_path` of the rows stays the TIFF's path. OME-Zarr input images are never -converted. - -The notebook groups conversion controls in an expandable panel with storage block size, -lossless compression and preview-resolution cards. Choosing conversion opens the panel; -collapsing it preserves the settings and leaves the disk-space warning visible. This -presentation does not change the conversion rules below. - -| Rule | Detail | -|---|---| -| Location | `/ome_zarr/.ome.zarr` | -| Format | OME-NGFF 0.4, Zarr format 2, `/` chunk separator | -| Compression | Lossless: Blosc LZ4 (default), Blosc Zstandard or none; level 1–9 (default 5) | -| Axes | `t`, `c`, `z`, `y`, `x` in that order, only the roles of the confirmed mapping (missing axes resolved as singletons have length 1; axes without a role are dropped) | -| Level 0 | The source values, unchanged, in the source pixel type; chunks of one plane along Z, C and T and at most the chunk size (16 to 8,192 px, default 1,024) in X and Y | -| Further levels | 2 x 2 block means rounded to the pixel type; automatic (default: until a plane fits in 512 x 512, at most 8 levels in all), none, or 1 to 7 levels; for viewing only | -| Large planes | A plane read in blocks is written block by block; its further levels are then built from the stored level above, in regions of even size, and equal those of a plane written whole | -| Scale | Physical pixel sizes, z-step and time step from OME or ImageJ metadata, with OME-NGFF units, each axis with its own unit (OME `PhysicalSizeXUnit`, `PhysicalSizeYUnit`, `PhysicalSizeZUnit`, micrometres when left out; ImageJ `unit`, `yunit`, `zunit`). OME-NGFF 0.4 requires a scale for every axis, so an unknown size is written as 1 without a unit and its axis is listed in the multiscales entry's `metadata` as `"unknown_scale": ["z", "y", "x"]` (axis names; absent when every size is known). `ngff_physical_sizes` gives None for those axes, so the placeholder is never read as a size; other readers must check the list. A copy written before the list (its list differs from the one its TIFF gives now) counts as changed and is converted again; its rows do not change | -| Channels | `omero` block with each channel's observed minimum and maximum as display window, and its name from metadata as `label`, as it is read (§2); an unnamed channel has no `label` (copies written earlier with the placeholder `Channel ` count as changed and are converted again), so a copy reads back with its source's channel names (§2) | -| Updates | Copies of unchanged images are kept, of new or changed images rewritten, and of images no longer analysed removed; a failed run leaves the earlier copies as they were (see §9 for publication) | -| Switching | Turning conversion on or off recomputes every TIFF row; turning it off removes the copies the manifest recorded | -| Missing copy | A converted TIFF whose copy was deleted counts as changed and is converted again | -| Outdated units | A copy whose axis units differ from those its TIFF's metadata gives now (copies written before each axis had its own unit, plan §12 question 24) counts as changed and is converted again; its rows do not change. Only the copy's `.zattrs` is read to tell | -| Settings change | Copies written with other conversion settings (recorded in the manifest) are written again; their rows do not change | +Up to v0.3, a run could convert every TIFF image to OME-Zarr in `/ome_zarr` and +compute its rows from the copies (`convert_tiffs`, `zarr_export`, CLI `--convert-tiff`, the +notebook's "(Optional) Image conversion" card). The bricks never read those copies, and v0.4 +reads every image through the run's own readers, so the option was removed (spec revision 4, +§8 PP-2). Its rows were the TIFF's own, so a later run keeps them. Saving images with axis +errors as named-axis copies is a separate step, planned for v0.4.0 as OME-TIFF. ## 11. Dataset consistency (folder check) @@ -407,8 +384,8 @@ instead of keeping copies. | Length units (D-a2) | `space_unit(text)`, `to_micrometers(value, unit)`, `SPACE_UNITS`, `UnsupportedUnitError` (`physical_sizes`) | Microscopy units only: `pm`; `Å`/`angstrom`; `nm`; `µm`/`um`/`micron(s)`/`micrometer(s)`/`micrometre(s)`; `mm`; `cm`; `m`; with each name in US and British spelling and in the plural (`nanometre`, `millimeters`, ...). Symbols of two letters or fewer are matched exactly (`Mm` is not `mm`), longer names in any case; ImageJ's `\u00B5` is decoded. `space_unit` returns the OME-NGFF name (`"micrometer"`), None when no unit is given, and refuses any other unit by name (`UnsupportedUnitError`); `to_micrometers` converts a positive finite length, None when the length or its unit is not known. This brick's readers (§2) use the same table: since D86, ImageJ and OME-NGFF sizes in British spellings (`micrometre`) are calibrated, where they were unknown before. | | Plain-TIFF resolution (D-c) | `read_physical_sizes` | A TIFF's resolution tags give a pixel size only with OME or ImageJ metadata and a unit; a plain TIFF (scanner or image-editor dpi) is uncalibrated, never converted from inches or centimetres. | | Labels (D-b) | `check_labels(pixels, name)`, `LARGEST_LABEL` (`masks`) | A mask holds whole-number labels from 0 (background) to `LARGEST_LABEL` = 2,147,483,647: MorphoBricks labels are int32 in every brick. A negative label, a label above the limit, or a pixel type that is not integer or boolean is refused with a `ValueError` naming the mask and the value. `read_mask`, `piece_counts`, `pair_counts` and `scan_masks` call it. Since D86 the limit is int32; before, this brick refused only labels above 2^63 − 1 (int64), so a `uint32` or wider mask with a label of 3,000,000,000 is now refused. | -| Header axes (D-d) | `header_axes(meta)`, `named_axes(meta)`, `series_axes_from_metadata(series)`, `PLACEHOLDER_AXES`, `OPEN_AXIS` (`metadata`) | The axes an image's own metadata names, as `tifffile` reads them with length-1 axes left out (`tifffile` keeps Y and X even of length 1, so an image one pixel high is `YX`): OME, ImageJ and `tifffile` "shaped" metadata (`{"axes": "CYX"}`, decision D-d), a plain 2D `YX` (or `YXS` RGB) TIFF, and an OME-Zarr image's OME-NGFF axes. None when an axis is a placeholder (`I`: a plain stack of pages; `Q`: shaped without axes, or an OME-NGFF axis of another type), or the image does not open: the file does not say which dimension is depth, time or channel. `named_axes` gives the rule position by position, with `?` (`OPEN_AXIS`) for a dimension the metadata does not name: a placeholder, and, in a plain stack of pages without axis metadata (`IYX`, `QYX`, `QQYX`; `TiffMetadata.axes_from_metadata` False, from `series_axes_from_metadata`; True, the strict reading, for a header built by hand), the `Y` and `X` too, which only say where each page's rows and columns are stored (a (Y, X, C) array written without metadata reads as `QYX`). `header_axes` gives axes exactly when nothing is open, and `check_declared_axes` then accepts those axes, them with `Y` and `X` swapped, and (without a C axis) `S` declared C; nothing else. | -| Declared axes | `check_declared_axes(meta, declared)`, `declared_axes_transposed(meta, declared)` (`axes`) | **Deprecated; removed before the v0.4.0 release** (they warn with a `DeprecationWarning`): bricks take no axis declarations and read the files of a preprocessing run whose axes were checked (§2); nothing is read transposed. Until then: axes a brick or user declares for an image or mask (`image_axes="CYX"`) may name what the header leaves open, never contradict it: one letter per header axis (any case), roles X, Y, Z, C, T each at most once, with `Y` and `X` declared when the header has them (a plain stack too); an axis the metadata names (`named_axes`) keeps its letter, except that the named `Y` and `X` may be declared swapped (header `CYX` declared `CXY`, `ZYX` declared `ZXY`): the image is read transposed, and `declared_axes_transposed` returns True so the caller can record it (owner decision of 31 October 2026; it raises `ValueError` for a declaration that does not fit); any other disagreement is refused; a dimension it leaves open may be declared any role, so a plain `QYX` stack may be declared `YXC` as well as `CYX` (since v0.3.4; before, its positional `Y` and `X` were binding and a placeholder could not be declared X or Y), while an ImageJ `ZYX` declared `CYX`, an OME `CYX` declared `YXC` or a shaped `CYX` declared `YXC` are refused; the RGB samples `S` may be declared C when the header has no C; any other header axis (tile `R`, ...) cannot be declared. Otherwise the error `declared_axes_mismatch` naming the image and both axes (`unreadable_tiff` or `unreadable_ome_zarr` when the header does not open). | +| Header axes (D-d) | `header_axes(meta)`, `named_axes(meta)`, `series_axes_from_metadata(series)`, `PLACEHOLDER_AXES`, `OPEN_AXIS` (`metadata`) | The axes an image's own metadata names, as `tifffile` reads them with length-1 axes left out (`tifffile` keeps Y and X even of length 1, so an image one pixel high is `YX`): OME, ImageJ and `tifffile` "shaped" metadata (`{"axes": "CYX"}`, decision D-d), a plain 2D `YX` (or `YXS` RGB) TIFF, and an OME-Zarr image's OME-NGFF axes. None when an axis is a placeholder (`I`: a plain stack of pages; `Q`: shaped without axes, or an OME-NGFF axis of another type), or the image does not open: the file does not say which dimension is depth, time or channel. `named_axes` gives the rule position by position, with `?` (`OPEN_AXIS`) for a dimension the metadata does not name: a placeholder, and, in a plain stack of pages without axis metadata (`IYX`, `QYX`, `QQYX`; `TiffMetadata.axes_from_metadata` False, from `series_axes_from_metadata`; True, the strict reading, for a header built by hand), the `Y` and `X` too, which only say where each page's rows and columns are stored (a (Y, X, C) array written without metadata reads as `QYX`). `header_axes` gives axes exactly when nothing is open. The axis check (§2) compares confirmed roles with `named_axes`. | +| Declared axes (removed) | `check_declared_axes`, `declared_axes_transposed` | **Removed in v0.4.0** (spec revision 4): they accepted a declaration with Y and X swapped and read it transposed, which v0.4 forbids. Bricks take no axis declarations; preprocessing's axis check (§2) compares the confirmed roles with the header. Former text, for reference: | **Deprecated in v0.4.0, removed in v0.5.0** (they warn with a `DeprecationWarning`): bricks take no axis declarations and read the files of a preprocessing run whose axes were checked (§2); nothing is read transposed. Until then: axes a brick or user declares for an image or mask (`image_axes="CYX"`) may name what the header leaves open, never contradict it: one letter per header axis (any case), roles X, Y, Z, C, T each at most once, with `Y` and `X` declared when the header has them (a plain stack too); an axis the metadata names (`named_axes`) keeps its letter, except that the named `Y` and `X` may be declared swapped (header `CYX` declared `CXY`, `ZYX` declared `ZXY`): the image is read transposed, and `declared_axes_transposed` returns True so the caller can record it (owner decision of 31 October 2026; it raises `ValueError` for a declaration that does not fit); any other disagreement is refused; a dimension it leaves open may be declared any role, so a plain `QYX` stack may be declared `YXC` as well as `CYX` (since v0.3.4; before, its positional `Y` and `X` were binding and a placeholder could not be declared X or Y), while an ImageJ `ZYX` declared `CYX`, an OME `CYX` declared `YXC` or a shaped `CYX` declared `YXC` are refused; the RGB samples `S` may be declared C when the header has no C; any other header axis (tile `R`, ...) cannot be declared. Otherwise the error `declared_axes_mismatch` naming the image and both axes (`unreadable_tiff` or `unreadable_ome_zarr` when the header does not open). | | Mask files | `read_mask_file(path, z=, t=)` (`masks`) | Reads one plane of a TIFF mask or of an OME-Zarr label image (`.zarr/labels/`) given by its path, with `read_mask`'s rules (§15): its axes are its own metadata's, Y and X plus at most a Z and a T axis named so, length-1 axes other than Y and X left out (a mask one pixel high or wide is read as a 2D array, plan §12 question 44); any other or unnamed dimension (a plain stack, `IYX`; a `C` axis) is refused, the message giving the header's axes and shape and the dimension that is not named, never taken for depth or time; `z` and `t` are required for a mask with that axis; integer or boolean pixels; labels checked with `check_labels`. A `ValueError` names the file and the problem. | | Identity (D-e) | `FOV_KEY`, `check_identity_text(value, field)`, `check_plane_index(value, axis)`, `check_identity_case(keys)`, `MISSING_TEXT`, `IdentityError` (`discovery`) | `FOV_KEY` = (`condition`, `biological_repeat`, `technical_repeat`, `position_id`), the position key of §1. `check_identity_text` accepts a label or `position_id` that is text without leading or trailing whitespace and not `""`, `NA`, `nan`, `none`, `null`, `n/a` or `` in any case (`MISSING_TEXT`); `check_plane_index` accepts None or an integer of at least 0, a Python `int` or a NumPy integer (returned as a plain `int`), and refuses everything else (bool, float, `"1"`) without converting it. Both raise `IdentityError` (a `ValueError`) naming the field and the value. Discovery applies the text rule to every folder and image name (`invalid_name`) and refuses names of one level that differ only in case anywhere in the experiment (`case_conflict`, §1). `position_id` keeps its own scope: unique within its technical repeat, ignoring case (`duplicate_position_id`). `check_identity_case` applies the same case rule to identities a brick reads from a table, one key per row in `FOV_KEY` order (or the three labels): a label of one level spelled two ways anywhere in the run, or two `position_id` values of one technical repeat equal ignoring case, raise `IdentityError` naming both spellings; the same key may repeat (several planes of one position). | | Sources of a run | `check_sources(manifest, experiment_root, relative_paths, image_axes=, identities=)`, `source_fingerprint`, `file_sha256` (`manifest`) | The checks a brick makes before using a run's table for some images (formerly CellPatcher `check_images` and MorphoFeature `histogram_fit`). Each image must be in the manifest (`images_not_in_manifest`), still exist (`images_missing`), and be unchanged by `source_fingerprint`, its size and modification time, for an OME-Zarr folder its files' total size and newest time, so a rewritten chunk counts (`images_changed`). Its recorded `position_id` must be the one its name gives; with `identities` (`FOV_KEY` order), its labels and `position_id` must be the caller's (`identity_differs`; `position_id_not_recorded` for a manifest written before positions were recorded). With `image_axes`, the caller's axes must give each axis the role recorded for the file (`FileEntry.roles`, or the run's mapping resolved for the header in older manifests); an axis without a role (length 1) fits any letter (`axis_mapping_mismatch`). All errors, naming their images; the table's own hash is checked with `file_sha256` against `csv_sha256`. | diff --git a/docs/plan.md b/docs/plan.md index 5c926de8..d08d0610 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -296,7 +296,8 @@ API, CLI). The notebook and CLI only call the API. expandable panel with separate storage, compression and preview-resolution cards.* - **OME-Zarr (optional `zarr` extra):** OME-Zarr images (OME-NGFF 0.4 and 0.5) are read as input next to TIFF files, plane by plane from level 0, with their declared axes - (`zarr_input`). For TIFF images the user chooses once per experiment: read them + (`zarr_input`). *TIFF to OME-Zarr conversion, described next, was removed in v0.4.0 + (question 46); TIFF images are read directly.* For TIFF images the user chose once per experiment: read them directly, or convert each to `/ome_zarr/.ome.zarr` (OME-NGFF 0.4) and compute its histograms from the copy, like any OME-Zarr image (`zarr_export`). The copies keep the source values at level 0; smaller levels (2 x 2 means) are for viewers. @@ -307,7 +308,7 @@ API, CLI). The notebook and CLI only call the API. - **Large planes in blocks:** planes over the memory limit are read in blocks of their stored strips, tiles or chunks, sized by the memory limit (`planes`), and their histograms are computed in two passes over the blocks with results identical to whole - planes (`histogram.compute_block_histogram`). OME-Zarr copies of such planes are written + planes (`histogram.compute_block_histogram`). Up to v0.3, OME-Zarr copies of such planes were written block by block, with their smaller levels built from the stored level above (`zarr_export`). The axis preview shows them reduced. Step 3's "What will happen" box names these images before the run. *Implemented, see questions 15 and 16.* @@ -1056,6 +1057,17 @@ Raised during review; to be decided before the milestones that need them: roles (they were compared with the header's letters; the role-based comparison still matters for plain stacks, whose positional Y and X need not be the roles), and not at all for an image with an axis error. `check_declared_axes` and - `declared_axes_transposed` are deprecated, to be removed before the v0.4.0 release - (design v0.4 §6, PP-2). Breaking for v0.4.0: experiments run with roles that swap or - contradict the header no longer run as they are.* + `declared_axes_transposed` are removed in v0.4.0 (question 46). Breaking for + v0.4.0: experiments run with roles that swap or contradict the header no longer run + as they are.* +46. What does v0.4.0 remove (spec revision 4, §8 PP-2)? + *Decided (1 November 2026, owner): `convert_tiffs` with `zarr_export` (OME-Zarr copies in + the output folder; the bricks never read them, and named-axis copies of images with axis + errors are a separate OME-TIFF step), the manifest's `convert_tiffs`, `conversion` and + `ome_zarr_path` (ignored when an older manifest holds them), the CLI `--convert-tiff`, + `--chunk-size`, `--compression`, `--compression-level` and `--pyramid-levels`, the + notebook's conversion card; the deprecated stand-ins of the code moved to the + normalization brick (`_moved` and the `csv_io`, `dtypes`, `histogram`, `normalization`, + `pooling` and `records` modules); and `check_declared_axes` with + `declared_axes_transposed`, removed rather than deprecated because they read a Y/X swap + transposed, which v0.4 forbids. Breaking for v0.4.0.* diff --git a/docs/user_guide.md b/docs/user_guide.md index 6696c0af..56a96840 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -37,8 +37,7 @@ pip install "morphobricks-preprocessing[notebook] @ git+https://github.com/CellM pip install jupyterlab ``` -To read OME-Zarr images, or to convert TIFF images to OME-Zarr (§6), install the `zarr` -extra too: +To read OME-Zarr images, install the `zarr` extra too: `pip install "morphobricks-preprocessing[notebook,zarr] @ git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git"`. Start JupyterLab with `jupyter lab` and open the notebook. Google Colab is detected by the @@ -299,28 +298,6 @@ results or comparing another grouping. same, only reading takes longer. Raise the limit only if the computer has enough memory. - *Recompute every row:* start from scratch instead of updating the earlier table. - - **(Optional) Image conversion:** - - **Read the images directly** (the default and recommended choice) reads TIFF and - OME-Zarr images as they are and saves nothing extra. - - **Convert the TIFF images to OME-Zarr, then process the converted copies** writes - every TIFF image in OME-Zarr format to the **ome_zarr** folder of the output (see - §6) and computes its histograms from that copy, like for any OME-Zarr image; the - results are the same. OME-Zarr images are never converted. - - When converting, an amber box warns that the images are stored again, so the space - they occupy is duplicated. It shows the size of the TIFF images, the most the copies - may need (before compression) and the free space on the output drive. It turns red - if the space may not be enough, but you can still run. - - **OME-Zarr conversion settings** opens when you choose conversion. Click its header - to collapse or expand three cards, each with its own controls and explanation: - **Storage block size** (square chunks, default 1,024 pixels per side), **Lossless - compression** (LZ4 by default, Zstandard, or none, with a level from 1 to 9), and - **Preview resolutions** (smaller copies for fast viewing: automatic, full resolution - only, or a fixed number). Histograms use full resolution. Collapsing the panel keeps - your choices and leaves the disk-space warning visible. Changing the settings later - writes the copies again; the histograms do not change. - - The choice is greyed out, with the install command below it, when the `zarr` extra - is not installed. **What will happen** says which images are converted and with - which settings, and a tile counts the conversions. - Click **Compute histograms**. A progress bar shows which image is being read. Tiles then summarise planes written, images read, kept, new, changed, removed and excluded. Nothing is written unless the whole run succeeds. **Technical details (log)** holds @@ -421,10 +398,6 @@ never writes the file. Other options: | `--image-type fluorescence\|rgb` | The image type (default: the suggested one, printed) | | `--design FILE` | The confirmed experiment design (default: `experiment_design.csv` in the output folder); it is copied to the output folder | | `--force-rebuild` | Recompute every row | -| `--convert-tiff` | Convert the TIFF images to OME-Zarr first and process the copies (needs the `zarr` extra); prints the disk space needed | -| `--chunk-size PX` | With `--convert-tiff`: tile size of the copies (16 to 8192, default 1024) | -| `--compression lz4\|zstd\|none`, `--compression-level N` | With `--convert-tiff`: lossless compression (default lz4, level 5) | -| `--pyramid-levels auto\|none\|N` | With `--convert-tiff`: smaller levels for viewing (default auto) | | `--outlier-limit N`, `--saturation-limit FRACTION`, `--min-images N` | Limits of the intensity checks printed after the run (default 3, 0.01 and 5) | | `--log FILE` | Write technical details to a file | | `--masks FOLDER` | The masks folder (§2). `check` lists the mask types found and checks every mask from its header; `run` stops on a mask error and records the masks in `run_manifest.json` | @@ -502,17 +475,8 @@ The default output folder is next to the experiment folder: fingerprint of the experiment design, and each image's position, size and modification time. Channel names and significant bits are recorded when the metadata provides them, and whether the image's planes were too large to read whole and are read in blocks. Format in [contract.md](contract.md) §9. -- **`ome_zarr/`** (only when TIFF images are converted): one `.ome.zarr` folder per TIFF - image, in the same folder structure as the experiment, e.g. - `ome_zarr/control/bio_rep1/tech_rep1/img_01.ome.zarr`. The full-resolution level holds - exactly the source pixel values; smaller levels are averages for fast viewing (for - example in napari with the `napari-ome-zarr` plugin). Pixel sizes and channel names are - copied from the metadata when present (an unnamed channel stays unnamed), each axis with its own unit. Copies of unchanged - images are kept on later runs, copies of deleted or excluded images are removed. Copies - written by an earlier version with the X unit on every axis are converted again once, - with their own units; the table does not change. Switching back to reading the - TIFF images directly removes the copies and recomputes their rows. Details in - [contract.md](contract.md) §10. +- **`ome_zarr/`**: written only by v0.3 runs that converted TIFF images to OME-Zarr. That + option was removed in v0.4.0; such a folder is left as it is, and may be deleted. - **`mask_relationships.json`** (only from the `masks` command): each kind of mask, the relationship between every two kinds with its counts, and the hierarchy. The rules and @@ -578,7 +542,6 @@ continues. | `axes_not_named` | Warning | The metadata of some images does not name their axes (a plain stack of pages, `IYX` or `QYX`), so which dimension is depth, time or channel comes only from the roles you confirm. The images are read with them, and the roles are recorded as confirmed by you. Check the roles with the preview. Before segmenting these images in other tools, consider saving named-axis copies (OME-TIFF), so every tool reads the same axes. | | `axes_yx_swapped` | Error | The roles you confirmed give the Y and X the image header names the other way round, so the images are not read (never transposed). The message names the likely cause. Two fixes: correct the axes (if the header is right, choose its roles), or convert (save named-axis copies with your roles, then run preprocessing on the copies). | | `axes_contradict_header` | Error | The roles you confirmed contradict the axes the image header names (for example the header says channels, C, and you chose time, T). The images are not read under a contradicted declaration. The message names the likely cause, often the tool that saved the files: tifffile labels a 3-D stack saved with `imagej=True` or `ome=True` as `CYX`, and an ImageJ stack of slices reads as depth even for a time-lapse. The same two fixes as `axes_yx_swapped`. | -| `declared_axes_mismatch` | Error (other bricks; deprecated, removed in v0.4.0) | The axes a brick was told for an image or mask contradict the axes in its metadata (for example a depth stack declared as channels). Declare the axes the metadata gives, or export the image again with the right axes. | | `missing_xy_axis`, `duplicate_axis_role` | Error | X/Y missing or a role used twice. Correct the axis roles. | | `unassigned_axis` | Error | A dimension with several planes has no role. Choose Z, C or T. | | `unsupported_axis` | Error | A position or tile dimension. Export each position separately. | @@ -628,7 +591,7 @@ continues. | `ambiguous_mask` | Error | A mask's name could belong to two images (one image name starts another's) or fit both layouts. Rename the masks or the images so each names one image and one kind. | | `mask_without_image`, `mask_outside_type_folder` | Warning | Files in the masks folder that match no analysed image (check the folder and file names), or other files directly in the masks folder. They are not used. | | `images_not_in_manifest`, `images_missing`, `images_changed`, `position_id_not_recorded`, `identity_differs`, `axis_mapping_mismatch` | Error (other bricks) | A later brick checked its images against this brick's run (`check_sources`): an image the run did not analyse, no longer there, changed since, recorded with other labels or position (or by a version that did not record positions), or read with axes the run gave other roles. Run this brick again on the experiment as it is, or use the run's labels and axis roles. | -| `zarr_not_installed` | Error | OME-Zarr images, or converting TIFF images, need the `zarr` extra. Install it, exclude the OME-Zarr images, or read the TIFF images directly. | +| `zarr_not_installed` | Error | OME-Zarr images need the `zarr` extra. Install it, or exclude the OME-Zarr images. | ## 9. Python interface diff --git a/pyproject.toml b/pyproject.toml index c6743a1a..d0164631 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -123,33 +123,25 @@ baseline = "tools/code_rules_baseline.json" "morphobricks_preprocessing" = "orchestration" "morphobricks_preprocessing.__main__" = "ui" "morphobricks_preprocessing._io" = "io" -"morphobricks_preprocessing._moved" = "pure" "morphobricks_preprocessing.axes" = "pure" "morphobricks_preprocessing.axis_check" = "pure" "morphobricks_preprocessing.mask_checks" = "pure" "morphobricks_preprocessing.notebook_axes" = "ui" "morphobricks_preprocessing.cli" = "ui" -"morphobricks_preprocessing.csv_io" = "pure" "morphobricks_preprocessing.dataset" = "pure" "morphobricks_preprocessing.design" = "io" "morphobricks_preprocessing.discovery" = "io" -"morphobricks_preprocessing.dtypes" = "pure" -"morphobricks_preprocessing.histogram" = "pure" "morphobricks_preprocessing.image_type" = "pure" "morphobricks_preprocessing.intensity" = "io" "morphobricks_preprocessing.manifest" = "io" "morphobricks_preprocessing.mask_scan" = "io" "morphobricks_preprocessing.masks" = "io" "morphobricks_preprocessing.metadata" = "io" -"morphobricks_preprocessing.normalization" = "pure" "morphobricks_preprocessing.notebook_style" = "ui" "morphobricks_preprocessing.notebook_ui" = "ui" "morphobricks_preprocessing.physical_sizes" = "io" "morphobricks_preprocessing.planes" = "io" -"morphobricks_preprocessing.pooling" = "pure" -"morphobricks_preprocessing.records" = "pure" "morphobricks_preprocessing.report" = "pure" "morphobricks_preprocessing.run" = "orchestration" "morphobricks_preprocessing.validation" = "pure" -"morphobricks_preprocessing.zarr_export" = "io" "morphobricks_preprocessing.zarr_input" = "io" diff --git a/src/morphobricks_preprocessing/__init__.py b/src/morphobricks_preprocessing/__init__.py index c57e6ea5..e8f993a2 100644 --- a/src/morphobricks_preprocessing/__init__.py +++ b/src/morphobricks_preprocessing/__init__.py @@ -10,16 +10,11 @@ from importlib.metadata import PackageNotFoundError, version -# The modules that moved to morphobricks_normalization stay importable as -# deprecated stand-ins for one version (see _moved). -from . import _moved, csv_io, dtypes, histogram, normalization, pooling, records # noqa: F401 from .axes import ( AxisMapping, AxisReview, axis_length_ranges, - check_declared_axes, check_mapping, - declared_axes_transposed, declared_mapping, file_mapping_for, prefill_experiment_mapping, @@ -180,13 +175,6 @@ from .report import GroupSummary, group_summaries from .run import Preflight, RunResult, SamplePlane, preflight, run_experiment, sample_planes from .validation import Severity, ValidationIssue, has_errors -from .zarr_export import ( - ConversionEstimate, - ConversionSettings, - OmeZarrWriter, - estimate_conversion, - ome_zarr_relative_path, -) from .zarr_input import ( NgffError, NgffImage, @@ -198,20 +186,6 @@ zarr_available, ) - -def __getattr__(name: str) -> object: - # Called only for names not defined here: the moved ones warn and come from - # the normalization brick, anything else is missing. - module = _moved.TOP_LEVEL_NAMES.get(name) - if module is None: - raise AttributeError(f"module {__name__!r} has no attribute {name!r}") - return _moved.moved_attribute(__name__, module, name) - - -def __dir__() -> list[str]: - return sorted({*globals(), *_moved.TOP_LEVEL_NAMES}) - - try: __version__ = version("morphobricks-preprocessing") except PackageNotFoundError: # running from a source checkout without installing @@ -237,11 +211,9 @@ def __dir__() -> list[str]: "MASK_REPORT_FILENAME", "MISSING_TEXT", "OPEN_AXIS", - "OUTPUT_FILENAME", "PIXEL_SIZE_TOLERANCE", "PLACEHOLDER_AXES", "RGB", - "SINGLE", "SPACE_UNITS", "AxisMapping", "AxisReview", @@ -251,8 +223,6 @@ def __dir__() -> list[str]: "Block", "BlockReader", "ConfirmedDesign", - "ConversionEstimate", - "ConversionSettings", "DatasetReport", "DesignChanges", "DesignCheckError", @@ -275,19 +245,12 @@ def __dir__() -> list[str]: "MetadataReport", "NgffError", "NgffImage", - "NoValidPixelsError", - "NormalizationParameters", - "OmeZarrWriter", "PairRelation", "PhysicalSizes", "Plane", - "PlaneHistogram", "PlaneLayout", "PlaneMemoryError", "PlaneReadError", - "PlaneRecord", - "PooledHistogram", - "PooledStatistics", "Preflight", "RepeatBox", "RunManifest", @@ -296,7 +259,6 @@ def __dir__() -> list[str]: "SamplePlane", "Severity", "TiffMetadata", - "UnsupportedDtypeError", "UnsupportedUnitError", "ValidationIssue", "XycPlane", @@ -306,7 +268,6 @@ def __dir__() -> list[str]: "balance_table", "biological_repeats", "check_dataset", - "check_declared_axes", "check_identity_case", "check_identity_text", "check_image_type", @@ -319,21 +280,15 @@ def __dir__() -> list[str]: "check_planes", "check_sources", "check_table", - "compute_block_histogram", - "compute_plane_histogram", - "declared_axes_transposed", "declared_mapping", - "default_bin_count", "default_design", "default_output_folder", "default_repeat_groups", "design_issues", "discover_experiment", - "estimate_conversion", "file_mapping_for", "file_sha256", "find_masks", - "group_records", "group_summaries", "has_errors", "header_axes", @@ -349,25 +304,20 @@ def __dir__() -> list[str]: "iter_xyc_planes", "likely_axis_cause", "load_design", - "make_plane_id", "images_named_as_masks", "mask_issues", "mask_name_issues", "mask_plane_issues", "mask_size_issues", "mask_thresholds", - "min_max", "named_axes", "ngff_label_names", - "ome_zarr_relative_path", "open_ngff", "open_ngff_labels", "pair_counts", - "percentile_range", "piece_counts", "pixel_size_groups", "plane_layout", - "pooled_statistics", "position_id", "prefill_experiment_mapping", "prefill_mapping", @@ -379,7 +329,6 @@ def __dir__() -> list[str]: "read_mask_file", "read_physical_sizes", "read_plane", - "read_plane_histograms", "read_sample_plane", "reconcile_design", "reference_axes", @@ -387,7 +336,6 @@ def __dir__() -> list[str]: "repeat_distributions", "review_axes", "review_image_type", - "robust_z_score", "run_experiment", "same_pixel_size", "sample_planes", @@ -399,7 +347,5 @@ def __dir__() -> list[str]: "write_experiment_design", "write_manifest", "write_mask_report", - "write_plane_histograms", - "z_score", "zarr_available", ] diff --git a/src/morphobricks_preprocessing/_io.py b/src/morphobricks_preprocessing/_io.py index 0596fbbc..044d6098 100644 --- a/src/morphobricks_preprocessing/_io.py +++ b/src/morphobricks_preprocessing/_io.py @@ -9,6 +9,7 @@ from __future__ import annotations import os +import shutil import tempfile from pathlib import Path @@ -43,3 +44,26 @@ def write_text_atomic(path: str | os.PathLike, text: str, *, parents: bool = Fal Path(temporary).unlink(missing_ok=True) raise return path + + +def free_space(path: str | os.PathLike) -> int | None: + """Return the free space on the drive holding ``path`` (or its nearest existing parent). + + Parameters + ---------- + path + A file or folder, which may not exist yet. + + Returns + ------- + int or None + Free bytes; None when it cannot be determined. + """ + path = Path(path).resolve() + for candidate in (path, *path.parents): + if candidate.exists(): + try: + return shutil.disk_usage(candidate).free + except OSError: + return None + return None diff --git a/src/morphobricks_preprocessing/_moved.py b/src/morphobricks_preprocessing/_moved.py deleted file mode 100644 index ef7efb83..00000000 --- a/src/morphobricks_preprocessing/_moved.py +++ /dev/null @@ -1,152 +0,0 @@ -"""Deprecated access to the code that moved to the normalization brick. - -The modules ``dtypes``, ``histogram``, ``records``, ``csv_io``, ``pooling`` and -``normalization`` moved unchanged to the package ``morphobricks_normalization`` -(normalization plan, decision N6). Their public names stay importable from -``morphobricks_preprocessing`` and from the old submodules for one version. The -first access through each old path emits a ``DeprecationWarning`` that names the -new location; the package's own code imports from ``morphobricks_normalization`` -and never triggers it. -""" - -from __future__ import annotations - -import importlib -import warnings -from collections.abc import Callable - -#: Public names of each moved module (normalization contract §1.1). -MOVED_NAMES: dict[str, tuple[str, ...]] = { - "dtypes": ( - "MAX_BIN_COUNT", - "DEFAULT_FLOAT_BIN_COUNT", - "MIN_FLOAT_BIN_COUNT", - "INTEGER_DTYPES", - "FLOAT_DTYPES", - "UnsupportedDtypeError", - "check_supported", - "is_integer", - "bit_depth", - "default_bin_count", - "validate_bin_count", - ), - "histogram": ( - "PlaneHistogram", - "compute_plane_histogram", - "compute_block_histogram", - "estimate_histogram_memory", - ), - "records": ("SINGLE", "LEVELS", "PlaneRecord", "make_plane_id", "group_records"), - "csv_io": ( - "SCHEMA_VERSION", - "OUTPUT_FILENAME", - "METADATA_COLUMNS", - "BIN_COLUMN_PREFIX", - "CsvFormatError", - "bin_column", - "write_plane_histograms", - "read_plane_histograms", - ), - "pooling": ( - "DEFAULT_PERCENTILES", - "NoValidPixelsError", - "PooledStatistics", - "PooledHistogram", - "pooled_statistics", - ), - "normalization": ( - "NORMAL_CONSISTENCY", - "NormalizationParameters", - "min_max", - "percentile_range", - "z_score", - "robust_z_score", - ), -} - -#: The moved names the package itself exported, and the module each came from. -TOP_LEVEL_NAMES: dict[str, str] = { - "OUTPUT_FILENAME": "csv_io", - "read_plane_histograms": "csv_io", - "write_plane_histograms": "csv_io", - "UnsupportedDtypeError": "dtypes", - "default_bin_count": "dtypes", - "PlaneHistogram": "histogram", - "compute_block_histogram": "histogram", - "compute_plane_histogram": "histogram", - "NormalizationParameters": "normalization", - "min_max": "normalization", - "percentile_range": "normalization", - "robust_z_score": "normalization", - "z_score": "normalization", - "NoValidPixelsError": "pooling", - "PooledHistogram": "pooling", - "PooledStatistics": "pooling", - "pooled_statistics": "pooling", - "SINGLE": "records", - "PlaneRecord": "records", - "group_records": "records", - "make_plane_id": "records", -} - -# Old paths that have already warned in this session. -_warned: set[str] = set() - - -def moved_attribute(old_module: str, module: str, name: str) -> object: - """Return a moved name, warning the first time it is read through its old path. - - Parameters - ---------- - old_module - The module it was read from, e.g. ``"morphobricks_preprocessing"``. - module - The moved module that defines it, e.g. ``"histogram"``. - name - The public name. - - Returns - ------- - object - The object from ``morphobricks_normalization.``, so types and - exceptions are the very same ones. - """ - old_path = f"{old_module}.{name}" - if old_path not in _warned: - _warned.add(old_path) - warnings.warn( - f"{old_path} has moved to morphobricks_normalization.{module}.{name}; import it " - "from there. The old path will be removed in the next version of " - "morphobricks-preprocessing.", - DeprecationWarning, - # Point at the caller's import or attribute access, not at this helper. - stacklevel=3, - ) - return getattr(importlib.import_module(f"morphobricks_normalization.{module}"), name) - - -def module_attributes(module: str) -> tuple[Callable[[str], object], Callable[[], list[str]]]: - """Build the ``__getattr__`` and ``__dir__`` of the stand-in for a moved module. - - Parameters - ---------- - module - The moved module, e.g. ``"histogram"``. - - Returns - ------- - tuple - ``(__getattr__, __dir__)`` for ``morphobricks_preprocessing.``. - """ - old_module = f"morphobricks_preprocessing.{module}" - names = MOVED_NAMES[module] - - def __getattr__(name: str) -> object: - if name in names: - return moved_attribute(old_module, module, name) - raise AttributeError(f"module {old_module!r} has no attribute {name!r}") - - def __dir__() -> list[str]: - return sorted(names) - - return __getattr__, __dir__ diff --git a/src/morphobricks_preprocessing/axes.py b/src/morphobricks_preprocessing/axes.py index 47638a7b..8ad0cbd5 100644 --- a/src/morphobricks_preprocessing/axes.py +++ b/src/morphobricks_preprocessing/axes.py @@ -17,7 +17,6 @@ from __future__ import annotations -import warnings from collections import Counter from collections.abc import Iterable, Mapping from dataclasses import dataclass, field @@ -31,11 +30,9 @@ check_header_axes, ) from .metadata import ( - OPEN_AXIS, MetadataReport, TiffMetadata, channel_count_issues, - named_axes, ) from .validation import ValidationIssue, error, has_errors, plural @@ -286,154 +283,6 @@ def prefill_mapping(axes: str) -> AxisMapping: return AxisMapping(axes, tuple(roles)) -def check_declared_axes(meta: TiffMetadata, declared: str) -> ValidationIssue | None: - """Check the axes a brick or user declares for an image against its header. - - .. deprecated:: 0.4.0 - Bricks no longer take axis declarations: they read the files of a preprocessing - run whose axes were checked (:func:`review_axes`, contract §2). This function is - removed before the v0.4.0 release. - - One rule for every brick that is told an image's (or mask's) axes, for example - ``image_axes="CYX"`` (MorphoBricks D86): a declaration can name what the header leaves - open, never contradict it. Only what the metadata names binds - (:func:`.metadata.named_axes`, the same rule as :func:`.metadata.header_axes`). - Position by position: - - - an axis the metadata names (X, Y, Z, C, T) must be declared with that letter, except - that the named Y and X may be declared swapped (header ``CYX`` declared ``CXY``): the - image is then read transposed, which :func:`declared_axes_transposed` reports so the - caller can record it (owner decision, 31 October 2026); - - a dimension it leaves open may be declared any role: a placeholder axis (``I``, - ``Q``), and in a plain stack of pages without axis metadata (``IYX``, ``QYX``, - ``QQYX``) the ``Y`` and ``X`` too, which only say where the pages' rows and columns - are stored (a (Y, X, C) array written without metadata reads as ``QYX``); - - the RGB samples axis ``S`` may be declared C when the header has no C (as the axis - review's prefill does); - - any other header axis (tile ``R``, angle ``A``, ...) has no role and cannot be - declared. - - The declaration must have one letter per header axis, each role at most once, and - declare Y and X when the header has them (also where a plain stack leaves them open). - - Parameters - ---------- - meta - The image's header (:func:`.metadata.inspect_image`). - declared - The declared axes, one letter per dimension (any case), e.g. ``"ZYX"``. - - Returns - ------- - ValidationIssue or None - The error ``declared_axes_mismatch`` naming the image, the declared and the header - axes (or ``unreadable_tiff`` / ``unreadable_ome_zarr`` when its header could not be - read); None when the declaration fits the header. - """ - _deprecated("check_declared_axes") - return _check_declared_axes(meta, declared) - - -def _check_declared_axes(meta: TiffMetadata, declared: str) -> ValidationIssue | None: - if not meta.readable: - code = "unreadable_ome_zarr" if meta.is_ome_zarr else "unreadable_tiff" - return error( - code, - "The image's header could not be read, so its axes cannot be checked.", - [meta.relative_path], - "Export the image again, or leave it out.", - ) - text = declared.upper() if isinstance(declared, str) else "" - problem = _declared_axes_problem(meta.axes, named_axes(meta), text) - if problem is None: - return None - return error( - "declared_axes_mismatch", - f"The axes declared for {meta.relative_path} ({declared!r}) do not fit the axes in its " - f"header ({meta.axes!r}): {problem}. Planes would be read along the wrong dimension.", - [meta.relative_path], - "Declare the axes the image's metadata gives, or export the image again with the " - "right axes in its metadata.", - ) - - -def declared_axes_transposed(meta: TiffMetadata, declared: str) -> bool: - """Return True when a declaration fits the header only with its named Y and X swapped. - - .. deprecated:: 0.4.0 - Nothing is read transposed any more: a confirmed Y/X swap is the error - ``axes_yx_swapped`` (:func:`review_axes`). This function is removed before the - v0.4.0 release. - - :func:`check_declared_axes` accepts a declaration that differs from the axes the - metadata names only by swapping Y and X (header ``CYX`` declared ``CXY``): the image is - read transposed. A brick that accepts such a declaration records it with this function. - A plain stack's Y and X are not named (:func:`.metadata.named_axes`), so declaring them - the other way round is never a swap. - - Parameters - ---------- - meta - The image's header (:func:`.metadata.inspect_image`). - declared - The declared axes, one letter per dimension (any case). - - Returns - ------- - bool - True when the named Y is declared X (and the named X declared Y). - - Raises - ------ - ValueError - When the declaration does not fit the header at all (:func:`check_declared_axes` - gives an error), with that error's message. - """ - _deprecated("declared_axes_transposed") - issue = _check_declared_axes(meta, declared) - if issue is not None: - raise ValueError(issue.message) - named = named_axes(meta) - return "Y" in named and declared.upper()[named.index("Y")] == "X" - - -def _declared_axes_problem(header: str, named: str, declared: str) -> str | None: - if not declared or len(declared) != len(header): - return f"{len(declared)} declared for {len(header)} dimensions" - unknown = sorted(set(declared) - set(ROLES)) - if unknown: - return f"{', '.join(unknown)} {'is' if len(unknown) == 1 else 'are'} not an axis role" - repeated = sorted({axis for axis in declared if declared.count(axis) > 1}) - if repeated: - return f"{', '.join(repeated)} declared more than once" - # Even where a plain stack leaves its Y and X open, the image has rows and columns: - # a declaration without them would read no plane at all. - undeclared = [axis for axis in "YX" if axis in header and axis not in declared] - if undeclared: - return f"{' and '.join(undeclared)} {'is' if len(undeclared) == 1 else 'are'} not declared" - for index, (have, said) in enumerate(zip(named, declared, strict=True)): - if have in (said, OPEN_AXIS): - continue - # The named Y and X swapped: with one letter per role, the other position must then - # be swapped too, so this accepts a swap of the two and nothing else. - if have in "YX" and said in "YX": - continue - if have == "S" and said == "C" and "C" not in header: - continue - return f"dimension {index} is {_axis_label(have)} in the header, declared {said}" - return None - - -def _deprecated(name: str) -> None: - warnings.warn( - f"morphobricks_preprocessing.{name} is deprecated and will be removed in v0.4.0: " - "bricks read the files of a preprocessing run whose axes were checked, and take no " - "axis declarations.", - DeprecationWarning, - stacklevel=3, - ) - - def reference_axes(files: Iterable[TiffMetadata]) -> str | None: """Return the arrangement the experiment mapping is defined on. diff --git a/src/morphobricks_preprocessing/cli.py b/src/morphobricks_preprocessing/cli.py index 26b94caa..95cd01a2 100644 --- a/src/morphobricks_preprocessing/cli.py +++ b/src/morphobricks_preprocessing/cli.py @@ -55,7 +55,7 @@ scan_masks, write_mask_report, ) -from .planes import DEFAULT_MEMORY_LIMIT, format_bytes +from .planes import DEFAULT_MEMORY_LIMIT from .report import ( axis_check_summary, balance_summary, @@ -69,17 +69,7 @@ run_summary, ) from .run import preflight, run_experiment -from .validation import has_errors, plural -from .zarr_export import ( - CHUNK_SIDE, - COMPRESSIONS, - MAX_CHUNK_SIZE, - MAX_LEVELS, - MIN_CHUNK_SIZE, - ConversionSettings, - estimate_conversion, - free_space, -) +from .validation import has_errors _UNITS = { "": 1, @@ -110,26 +100,6 @@ def parse_size(text: str) -> int: return int(float(match.group(1)) * _UNITS[unit]) -def parse_pyramid_levels(text: str) -> int | None: - """Parse ``--pyramid-levels``: ``auto`` (None), ``none`` (0) or a number of levels. - - Raises - ------ - argparse.ArgumentTypeError - If the text is none of these. - """ - value = text.strip().lower() - if value == "auto": - return None - if value == "none": - return 0 - if value.isdigit() and 0 <= int(value) < MAX_LEVELS: - return int(value) - raise argparse.ArgumentTypeError( - f"{text!r} is not a number of pyramid levels; use auto, none or 1 to {MAX_LEVELS - 1}." - ) - - def build_parser() -> argparse.ArgumentParser: """Build the argument parser.""" parser = argparse.ArgumentParser( @@ -199,45 +169,6 @@ def build_parser() -> argparse.ArgumentParser: run.add_argument( "--force-rebuild", action="store_true", help="recompute every row of the table" ) - run.add_argument( - "--convert-tiff", - action="store_true", - help="convert every TIFF image to OME-Zarr in /ome_zarr first and compute " - "its histograms from the copy (needs the zarr extra); OME-Zarr images are always " - "read directly", - ) - conversion = run.add_argument_group( - "conversion settings", "used with --convert-tiff; the defaults suit most experiments" - ) - conversion.add_argument( - "--chunk-size", - type=int, - default=CHUNK_SIDE, - metavar="PX", - help=f"side of the stored tiles in pixels ({MIN_CHUNK_SIZE} to {MAX_CHUNK_SIZE}; " - f"default {CHUNK_SIDE})", - ) - conversion.add_argument( - "--compression", - choices=sorted(COMPRESSIONS), - default="lz4", - help="lossless compression: lz4 (default, fast), zstd (smaller, slower) or none", - ) - conversion.add_argument( - "--compression-level", - type=int, - default=5, - metavar="N", - help="1 (fastest) to 9 (smallest files); default 5", - ) - conversion.add_argument( - "--pyramid-levels", - type=parse_pyramid_levels, - default=None, - metavar="auto|none|N", - help="smaller levels for viewing: auto (default, halve until a plane fits 512 px), " - f"none, or a number from 1 to {MAX_LEVELS - 1}", - ) checks = run.add_argument_group( "intensity checks", "limits of the warnings printed after the run; the defaults suit most" ) @@ -447,16 +378,6 @@ def _run(args: argparse.Namespace) -> int: # noqa: PLR0915 f"--axes {pre.suggested_mapping.roles_string if pre.suggested_mapping else axes}." ) return 1 - try: - conversion = ConversionSettings( - chunk_size=args.chunk_size, - compression=args.compression, - compression_level=args.compression_level, - pyramid_levels=args.pyramid_levels, - ) - except ValueError as exc: - print(exc) - return 1 try: intensity_settings = IntensityCheckSettings( outlier_limit=args.outlier_limit, @@ -496,9 +417,6 @@ def _run(args: argparse.Namespace) -> int: # noqa: PLR0915 print("Experiment design:") print(design_summary(design)) print() - if args.convert_tiff: - print(_conversion_note(pre, conversion, args.output)) - print() def progress(done: int, total: int, path: str) -> None: if path: @@ -514,8 +432,6 @@ def progress(done: int, total: int, path: str) -> None: exclude=args.exclude, force_rebuild=args.force_rebuild, progress=progress, - convert_tiffs=args.convert_tiff, - conversion=conversion, design=design, masks_folder=args.masks, image_type=image_type, @@ -595,24 +511,6 @@ def _image_type_note(review: ImageTypeReview, given: str | None) -> str: return "\n".join(lines) -def _conversion_note(pre, conversion: ConversionSettings, output) -> str: # noqa: ANN001 - estimate = estimate_conversion(pre.discovery.root, pre.metadata.files, conversion) - if not estimate.image_count: - return "Conversion: there are no TIFF images to convert." - folder = output or default_output_folder(pre.discovery.root) - free = free_space(folder) - lines = [ - f"Conversion ({conversion.describe()}): {plural(estimate.image_count, 'TIFF image')} " - "will be stored again in OME-Zarr format, so the space they take is duplicated.", - f" TIFF images: {format_bytes(estimate.source_bytes)}; copies: up to about " - f"{format_bytes(estimate.copy_bytes)} before compression; free space on the output " - f"drive: {format_bytes(free) if free is not None else 'unknown'}.", - ] - if free is not None and estimate.copy_bytes > free: - lines.append(" Warning: this may not be enough space.") - return "\n".join(lines) - - def _attach_log(path: str) -> logging.Handler: handler = logging.FileHandler(path, encoding="utf-8") handler.setLevel(logging.DEBUG) diff --git a/src/morphobricks_preprocessing/csv_io.py b/src/morphobricks_preprocessing/csv_io.py deleted file mode 100644 index ad75358c..00000000 --- a/src/morphobricks_preprocessing/csv_io.py +++ /dev/null @@ -1,12 +0,0 @@ -"""Deprecated: moved to ``morphobricks_normalization.csv_io``. - -Its public names stay importable from here for one version and warn on first use -(see :mod:`morphobricks_preprocessing._moved`). Import them from -``morphobricks_normalization.csv_io`` instead. -""" - -from __future__ import annotations - -from ._moved import module_attributes - -__getattr__, __dir__ = module_attributes("csv_io") diff --git a/src/morphobricks_preprocessing/dtypes.py b/src/morphobricks_preprocessing/dtypes.py deleted file mode 100644 index b4b83471..00000000 --- a/src/morphobricks_preprocessing/dtypes.py +++ /dev/null @@ -1,12 +0,0 @@ -"""Deprecated: moved to ``morphobricks_normalization.dtypes``. - -Its public names stay importable from here for one version and warn on first use -(see :mod:`morphobricks_preprocessing._moved`). Import them from -``morphobricks_normalization.dtypes`` instead. -""" - -from __future__ import annotations - -from ._moved import module_attributes - -__getattr__, __dir__ = module_attributes("dtypes") diff --git a/src/morphobricks_preprocessing/histogram.py b/src/morphobricks_preprocessing/histogram.py deleted file mode 100644 index fa437c93..00000000 --- a/src/morphobricks_preprocessing/histogram.py +++ /dev/null @@ -1,12 +0,0 @@ -"""Deprecated: moved to ``morphobricks_normalization.histogram``. - -Its public names stay importable from here for one version and warn on first use -(see :mod:`morphobricks_preprocessing._moved`). Import them from -``morphobricks_normalization.histogram`` instead. -""" - -from __future__ import annotations - -from ._moved import module_attributes - -__getattr__, __dir__ = module_attributes("histogram") diff --git a/src/morphobricks_preprocessing/manifest.py b/src/morphobricks_preprocessing/manifest.py index dc2c182e..ddce8e1b 100644 --- a/src/morphobricks_preprocessing/manifest.py +++ b/src/morphobricks_preprocessing/manifest.py @@ -40,7 +40,6 @@ from .discovery import position_id from .metadata import inspect_image from .validation import ValidationIssue, error, plural -from .zarr_export import ConversionSettings MANIFEST_FILENAME = "run_manifest.json" MANIFEST_VERSION = "1" @@ -65,12 +64,6 @@ class RunSettings: them. exclude Relative paths of images the user excluded from the run, sorted. - convert_tiffs - Whether TIFF images are converted to OME-Zarr in the output folder and - their histograms computed from the converted copies. - conversion - How the copies are written (chunk size, compression, smaller levels); - only used when ``convert_tiffs`` is on. It does not change any row. histogram The histogram range of histograms that are not exact, with the limits of the range check (the normalization brick's ``HistogramSettings``). @@ -82,8 +75,6 @@ class RunSettings: memory_limit: int missing_as_singleton: bool = False exclude: tuple[str, ...] = () - convert_tiffs: bool = False - conversion: ConversionSettings = field(default_factory=ConversionSettings) histogram: HistogramSettings = field(default_factory=HistogramSettings) def __post_init__(self) -> None: @@ -116,9 +107,6 @@ class FileEntry: Channel names from metadata; empty when not given. significant_bits Significant bits from metadata; None when not given. - ome_zarr_path - For a TIFF converted to OME-Zarr, the converted copy relative to the - output folder (its rows are computed from it); otherwise None. channel_names_trimmed True when leading or trailing whitespace was removed from the image's channel names as they were read (names are recorded trimmed). @@ -146,7 +134,6 @@ class FileEntry: plane_count: int = 0 channel_names: tuple[str | None, ...] = () significant_bits: int | None = None - ome_zarr_path: str | None = None read_in_blocks: bool = False position_id: str | None = None channel_names_trimmed: bool = False @@ -161,7 +148,6 @@ def same_source(self, other: FileEntry) -> bool: and self.labels == other.labels and self.axes == other.axes and self.singleton_roles == other.singleton_roles - and self.ome_zarr_path == other.ome_zarr_path ) @@ -312,8 +298,6 @@ def to_dict(self) -> dict: self.settings.missing_as_singleton, "missing_as_singleton" ), "exclude": list(self.settings.exclude), - "convert_tiffs": _flag(self.settings.convert_tiffs, "convert_tiffs"), - "conversion": self.settings.conversion.to_dict(), "histogram": self.settings.histogram.to_dict(), }, "axis_mapping": {"axes": self.mapping.axes, "roles": list(self.mapping.roles)}, @@ -370,8 +354,6 @@ def from_dict(cls, data: dict) -> RunManifest: memory_limit=int(settings["memory_limit"]), missing_as_singleton=_flag(settings["missing_as_singleton"]), exclude=tuple(str(p) for p in settings["exclude"]), - convert_tiffs=_flag(settings.get("convert_tiffs", False)), - conversion=ConversionSettings.from_dict(settings.get("conversion")), histogram=( HistogramSettings() if settings.get("histogram") is None @@ -421,7 +403,6 @@ def file_entry_for( # noqa: PLR0913 plane_count: int = 0, channel_names: tuple[str | None, ...] = (), significant_bits: int | None = None, - ome_zarr_path: str | None = None, read_in_blocks: bool = False, channel_names_trimmed: bool = False, ) -> FileEntry: @@ -437,7 +418,6 @@ def file_entry_for( # noqa: PLR0913 plane_count=plane_count, channel_names=tuple(channel_names), significant_bits=significant_bits, - ome_zarr_path=ome_zarr_path, read_in_blocks=read_in_blocks, position_id=position_id(relative_path), channel_names_trimmed=channel_names_trimmed, @@ -686,7 +666,6 @@ def _file_dict(entry: FileEntry) -> dict: "plane_count": entry.plane_count, "channel_names": list(entry.channel_names), "significant_bits": entry.significant_bits, - "ome_zarr_path": entry.ome_zarr_path, "read_in_blocks": _flag(entry.read_in_blocks, "read_in_blocks"), "position_id": entry.position_id, "channel_names_trimmed": _flag(entry.channel_names_trimmed, "channel_names_trimmed"), @@ -710,7 +689,6 @@ def _file_entry(data: dict) -> FileEntry: plane_count=int(data["plane_count"]), channel_names=tuple(None if n is None else str(n) for n in data.get("channel_names", [])), significant_bits=None if bits is None else int(bits), - ome_zarr_path=None if data.get("ome_zarr_path") is None else str(data["ome_zarr_path"]), read_in_blocks=_flag(data.get("read_in_blocks", False)), position_id=None if data.get("position_id") is None else str(data["position_id"]), channel_names_trimmed=_flag(data.get("channel_names_trimmed", False)), diff --git a/src/morphobricks_preprocessing/metadata.py b/src/morphobricks_preprocessing/metadata.py index 9f91a306..b3402bb6 100644 --- a/src/morphobricks_preprocessing/metadata.py +++ b/src/morphobricks_preprocessing/metadata.py @@ -195,7 +195,8 @@ def dtype(self) -> np.dtype | None: def named_axes(meta: TiffMetadata) -> str: """Return the image's axes with every dimension its metadata does not name as ``?``. - The one rule behind :func:`header_axes` and :func:`.axes.check_declared_axes` + The one rule behind :func:`header_axes` and the axis check + (:func:`.axis_check.header_axis_issue`) (MorphoBricks D86): a dimension is named when ``tifffile`` (or OME-NGFF) gives it a letter from the file's metadata. Left open (:data:`OPEN_AXIS`) are @@ -237,10 +238,7 @@ def header_axes(meta: TiffMetadata) -> str | None: the OME-NGFF axes of an OME-Zarr image. When any dimension is left open (:func:`named_axes`: a placeholder axis, so a plain multi-page TIFF ``IYX`` or a shaped one without axes ``QYX``), the file does not say which dimension is depth, time or - channel, so there are no header axes and nothing may be guessed. Exactly when this - gives axes, :func:`.axes.check_declared_axes` accepts those axes, them with Y and X - swapped (read transposed) and, without a C axis, the RGB samples ``S`` declared C; - nothing else. + channel, so there are no header axes and nothing may be guessed. Length-1 axes are left out, as ``tifffile`` reads them. Parameters diff --git a/src/morphobricks_preprocessing/normalization.py b/src/morphobricks_preprocessing/normalization.py deleted file mode 100644 index 36d5cf18..00000000 --- a/src/morphobricks_preprocessing/normalization.py +++ /dev/null @@ -1,12 +0,0 @@ -"""Deprecated: moved to ``morphobricks_normalization.normalization``. - -Its public names stay importable from here for one version and warn on first use -(see :mod:`morphobricks_preprocessing._moved`). Import them from -``morphobricks_normalization.normalization`` instead. -""" - -from __future__ import annotations - -from ._moved import module_attributes - -__getattr__, __dir__ = module_attributes("normalization") diff --git a/src/morphobricks_preprocessing/notebook_ui.py b/src/morphobricks_preprocessing/notebook_ui.py index a327a325..baa12081 100644 --- a/src/morphobricks_preprocessing/notebook_ui.py +++ b/src/morphobricks_preprocessing/notebook_ui.py @@ -114,18 +114,6 @@ ) from .run import Preflight, RunResult, preflight, run_experiment from .validation import ValidationIssue, has_errors, plural -from .zarr_export import ( - CHUNK_SIDE, - COMPRESSIONS, - MAX_CHUNK_SIZE, - MAX_LEVELS, - MIN_CHUNK_SIZE, - OME_ZARR_FOLDER, - ConversionSettings, - estimate_conversion, - free_space, - zarr_available, -) ROLE_OPTIONS = [ ("Z - depth (focal planes)", "Z"), @@ -157,11 +145,6 @@ # "What will happen" is a short summary, so its lists of images fold early. _PLAN_PATHS = 5 -CONVERSION_OPTIONS = [ - ("Read the images directly (recommended; nothing extra is saved)", "read"), - ("Convert the TIFF images to OME-Zarr, then process the converted copies", "convert"), -] - # Step 3's histogram range (follow-up P6, normalization contract §1.4). HISTOGRAM_RANGE_OPTIONS = [ ("Minimum to maximum of each plane (recommended)", "minmax"), @@ -413,7 +396,6 @@ def run_step(self) -> widgets.Widget: [ self.readiness_card, self.settings_card, - self.conversion_card, self.plan_card, self.run_button, self.progress, @@ -1730,7 +1712,6 @@ def _build_run_step(self) -> None: self.settings_card = ui.card( "Settings", [self.recommended, self.recommended_hint, self.advanced] ) - self._build_conversion_card() self.plan = widgets.HTML() self.plan_card = ui.card("What will happen", [self.plan]) self.run_button = ui.button( @@ -1777,211 +1758,6 @@ def _build_run_step(self) -> None: ) self.run_button.on_click(lambda _: self.run()) - def _build_conversion_card(self) -> None: - # Converting is a choice about the data, not a tuning value, so it has its own - # card and the recommended settings leave it alone. - available = zarr_available() - self.conversion = widgets.RadioButtons( - options=CONVERSION_OPTIONS, - value="read", - disabled=not available, - layout=widgets.Layout(width="99%"), - ) - self.conversion_help = ui.help_text(_conversion_help(available)) - self.space_warning = widgets.HTML() - self.chunk_size = widgets.BoundedIntText( - value=CHUNK_SIDE, - min=MIN_CHUNK_SIZE, - max=MAX_CHUNK_SIZE, - step=256, - description="Side length (px)", - style=_label(), - layout=widgets.Layout(width="99%"), - ) - self.compression = widgets.Dropdown( - options=[(label, key) for key, label in COMPRESSIONS.items()], - value="lz4", - style=_label(), - layout=widgets.Layout(width="99%"), - ) - self.compression_level = widgets.BoundedIntText( - value=5, - min=1, - max=9, - description="Level (1–9)", - style=_label(), - layout=widgets.Layout(width="99%"), - ) - self.compression_level_box = widgets.VBox( - [ - self.compression_level, - ui.help_text( - "1 favours speed; 9 favours smaller files. The default is 5. " - "Every level keeps the full-resolution pixel values unchanged." - ), - ] - ) - self.pyramid = widgets.Dropdown( - options=PYRAMID_OPTIONS, - value="auto", - style=_label(), - layout=widgets.Layout(width="99%"), - ) - self.pyramid_count = widgets.BoundedIntText( - value=2, - min=1, - max=MAX_LEVELS - 1, - description="Extra levels", - style=_label(), - layout=widgets.Layout(width="99%"), - ) - self.pyramid_count_box = widgets.VBox( - [ - self.pyramid_count, - ui.help_text( - f"Add 1 to {MAX_LEVELS - 1} smaller copies, each half the width and " - "height of the previous level. Two adds half-size and quarter-size copies." - ), - ] - ) - storage_card = ui.card( - "Storage block size", - [ - ui.help_text( - "Each plane is stored in square blocks called chunks. " - "This changes how the copy is stored and read." - ), - self.chunk_size, - ui.help_text( - f"Default: {CHUNK_SIDE:,} × {CHUNK_SIDE:,} pixels per block, or smaller " - f"at an image edge. Allowed side lengths: {MIN_CHUNK_SIZE}–{MAX_CHUNK_SIZE:,} " - "pixels. Each block belongs to one depth slice, channel and time point." - ), - ui.help_text( - "Smaller blocks let viewers read smaller regions; larger blocks mean fewer " - "blocks to manage. Leave the default if you are unsure." - ), - ], - ) - compression_card = ui.card( - "Lossless compression", - [ - ui.help_text("Reduce the space used by the copies without changing pixel values."), - self.compression, - ui.help_text( - "LZ4 (recommended) favours speed. Zstandard favours smaller files and " - "may take longer. No compression skips compression and uses more disk space." - ), - self.compression_level_box, - ], - ) - preview_card = ui.card( - "Preview resolutions", - [ - ui.help_text( - "Add smaller versions for viewers to zoom out quickly. These versions " - "form an image pyramid; histograms always use the full resolution." - ), - self.pyramid, - ui.help_text( - "Automatic (recommended) halves each plane until it fits within " - "512 × 512 pixels. Full resolution only saves space by omitting smaller " - "versions. Choose the number of copies to set how many to add." - ), - self.pyramid_count_box, - ], - ) - self.conversion_grid = widgets.GridBox( - [storage_card, compression_card, preview_card], - layout=widgets.Layout( - width="99%", - grid_template_columns="repeat(auto-fit, minmax(min(99%, 320px), 1fr))", - grid_gap="10px", - ), - ) - self.conversion_settings = widgets.Accordion( - children=[self.conversion_grid], - titles=("OME-Zarr conversion settings",), - selected_index=None, - layout=widgets.Layout(width="99%"), - ) - self.conversion_card = ui.card( - "(Optional) Image conversion", - [self.conversion, self.conversion_help, self.space_warning, self.conversion_settings], - ) - for control in ( - self.conversion, - self.chunk_size, - self.compression, - self.compression_level, - self.pyramid, - self.pyramid_count, - ): - control.observe(lambda _: self._conversion_changed(), names="value") - self._refresh_conversion() - - def _conversion_changed(self) -> None: - self._refresh_run_step() - - def _refresh_conversion(self) -> None: - converting = self._convert_tiffs() - # Open on choosing conversion, but preserve a deliberate collapse while editing. - if converting and not ui.is_visible(self.conversion_settings): - self.conversion_settings.selected_index = 0 - ui.set_visible(self.conversion_settings, converting) - ui.set_visible(self.compression_level_box, self.compression.value != "none") - ui.set_visible(self.pyramid_count_box, self.pyramid.value == "fixed") - self.space_warning.value = self._space_warning_html() if converting else "" - - def _space_warning_html(self) -> str: - if self.preflight is None: - return ui.status( - "warn", - "Converting stores your images a second time.", - ["Check the experiment folder in step 1 to see how much disk space it needs."], - ) - folder = ui.clean_path(self.folder.value) - estimate = estimate_conversion( - folder, self.preflight.metadata.files, self._conversion_settings() - ) - if not estimate.image_count: - return ui.status( - "info", - "There is nothing to convert.", - ["Every image is already OME-Zarr; they are always read directly."], - ) - output = Path(self._output_folder() or default_output_folder(folder)) - free = free_space(output) - lines = [ - f"All {plural(estimate.image_count, 'TIFF image')} will be stored again in the " - "converted format, so all the space they occupy is duplicated.", - f"The TIFF images take {format_bytes(estimate.source_bytes)}. The copies need up " - f"to about {format_bytes(estimate.copy_bytes)} (before compression, including the " - "smaller levels); compression usually makes them smaller.", - "Free space on the drive of the output folder: " - + (format_bytes(free) if free is not None else "unknown") - + ".", - "The copies are kept in the output folder; choosing Read the images directly " - "later removes them.", - ] - if free is not None and estimate.copy_bytes > free: - lines.append( - "This may not be enough space. Free some space, or choose an output folder " - "on another drive in step 1." - ) - return ui.status("error", "The converted images may not fit on the drive.", lines) - return ui.status("warn", "Converting duplicates the space your images take.", lines) - - def _conversion_settings(self) -> ConversionSettings: - return ConversionSettings( - chunk_size=self.chunk_size.value, - compression=self.compression.value, - compression_level=self.compression_level.value, - pyramid_levels={"auto": None, "none": 0}.get( - self.pyramid.value, self.pyramid_count.value - ), - ) - def _reset_settings(self, pre: Preflight) -> None: dtype = pre.metadata.dtype self._default_bins = None @@ -1997,7 +1773,6 @@ def _reset_settings(self, pre: Preflight) -> None: self.bins.value = self._default_bins self.bins_hint.value = f"
{_bins_help(dtype)}
" self._recommended_changed() - self._refresh_conversion() def _recommended_changed(self) -> None: recommended = self.recommended.value @@ -2061,7 +1836,6 @@ def _refresh_run_step(self) -> None: self.readiness.value = title + ui.checklist(items) self.run_button.disabled = bool(missing) self.plan.value = self._plan_html(review if folder_ok else None) - self._refresh_conversion() def _plan_layouts(self, review: AxisReview) -> dict[str, PlaneLayout]: """Return how each image's planes are read, from the headers only. @@ -2168,14 +1942,6 @@ def _plan_html(self, review: AxisReview | None = None) -> str: f"{plural(zarrs, 'OME-Zarr image')} will be read directly, with the axes " f"{'its' if zarrs == 1 else 'their'} metadata declares." ) - if tiffs and self._convert_tiffs(): - copies = html.escape(str(output / OME_ZARR_FOLDER)) - settings = html.escape(self._conversion_settings().describe()) - lines.append( - f"{plural(tiffs, 'TIFF image')} will be converted to OME-Zarr in " - f"{copies} first ({settings}); their histograms are computed from the " - "copies. Copies of unchanged images are kept." - ) # Which images are read in blocks depends on the axis roles, so it is only # worked out once they are confirmed (``review`` is None until then). blocks_line, over_card = self._memory_plan(review) if review is not None else ("", "") @@ -2233,8 +1999,6 @@ def run(self) -> RunResult | None: exclude=self._excluded(), force_rebuild=self.force_rebuild.value, progress=self._progress, - convert_tiffs=self._convert_tiffs(), - conversion=self._conversion_settings(), design=self.design, image_type=self.confirmed_image_type, intensity_settings=self.run_checks.settings(), @@ -2282,11 +2046,6 @@ def _show_run_result(self, result: RunResult) -> None: lines = [f"Table: {result.csv_path}", f"Manifest: {result.manifest_path}"] if result.design_path is not None: lines.append(f"Experiment design: {result.design_path}") - if result.converted or self._convert_tiffs(): - lines.append( - f"Converted TIFF images: {result.output_folder / OME_ZARR_FOLDER} " - f"({plural(len(result.converted), 'image')} converted in this run)" - ) if result.rebuild_reason: lines.append(f"Every image was read because {result.rebuild_reason}.") self.run_status.value = ui.status( @@ -2307,8 +2066,6 @@ def _show_run_result(self, result: RunResult) -> None: ("🗑️", f"{len(result.removed):,}", "images removed"), ("⛔", f"{len(result.excluded):,}", "excluded images"), ] - if result.converted or self._convert_tiffs(): - tiles.append(("🗂️", f"{len(result.converted):,}", "TIFF images converted")) tiles.append(("⚠️", issue_counts(result.issues), "checks")) self.run_tiles.value = ui.tiles(tiles) @@ -2330,9 +2087,6 @@ def _show_run_checks(self, path: Path) -> None: "MorphoBricks bricks either way.", ) - def _convert_tiffs(self) -> bool: - return self.conversion.value == "convert" - def _progress(self, done: int, total: int, path: str) -> None: self.progress.max = max(total, 1) self.progress.value = done @@ -3278,27 +3032,6 @@ def _folder_tiles(pre: Preflight, issues: Sequence[ValidationIssue]) -> list[tup ] -def _conversion_help(available: bool) -> str: - """Help lines under the choice between reading and converting.""" - items = [ - "Read the images directly (recommended): TIFF and OME-Zarr images are read " - "as they are, and nothing extra is saved.", - "Convert: every TIFF image is first written in OME-Zarr format to the " - f"{OME_ZARR_FOLDER} folder of the output, and its histograms are computed from " - "that copy, like for any OME-Zarr image. The results are the same.", - "OME-Zarr images are always read directly; they are never converted.", - "The copies keep the exact pixel values and can be opened in viewers such as napari. " - "Your images are never changed.", - "Switching between the two later recomputes every TIFF image.", - ] - if not available: - items.append( - "Converting is not available: it needs the zarr package. Install it with " - 'pip install "morphobricks-preprocessing[zarr]" and restart the notebook.' - ) - return "
    " + "".join(f"
  • {item}
    • " for item in items) + "
" - - def _bins_help(dtype) -> str: # noqa: ANN001 if is_integer(dtype): top = min(MAX_BIN_COUNT, 2 ** (dtype.itemsize * 8)) diff --git a/src/morphobricks_preprocessing/pooling.py b/src/morphobricks_preprocessing/pooling.py deleted file mode 100644 index e36e4d2c..00000000 --- a/src/morphobricks_preprocessing/pooling.py +++ /dev/null @@ -1,12 +0,0 @@ -"""Deprecated: moved to ``morphobricks_normalization.pooling``. - -Its public names stay importable from here for one version and warn on first use -(see :mod:`morphobricks_preprocessing._moved`). Import them from -``morphobricks_normalization.pooling`` instead. -""" - -from __future__ import annotations - -from ._moved import module_attributes - -__getattr__, __dir__ = module_attributes("pooling") diff --git a/src/morphobricks_preprocessing/records.py b/src/morphobricks_preprocessing/records.py deleted file mode 100644 index 42489555..00000000 --- a/src/morphobricks_preprocessing/records.py +++ /dev/null @@ -1,12 +0,0 @@ -"""Deprecated: moved to ``morphobricks_normalization.records``. - -Its public names stay importable from here for one version and warn on first use -(see :mod:`morphobricks_preprocessing._moved`). Import them from -``morphobricks_normalization.records`` instead. -""" - -from __future__ import annotations - -from ._moved import module_attributes - -__getattr__, __dir__ = module_attributes("records") diff --git a/src/morphobricks_preprocessing/report.py b/src/morphobricks_preprocessing/report.py index dadf1885..cecd032e 100644 --- a/src/morphobricks_preprocessing/report.py +++ b/src/morphobricks_preprocessing/report.py @@ -227,7 +227,6 @@ def run_summary(result: RunResult) -> str: ("Unchanged (rows kept)", () if result.rebuild_reason else result.unchanged), ("Removed from the table", result.removed), ("Excluded", result.excluded), - ("TIFF images converted to OME-Zarr", result.converted), ): if files: lines.append(f"{label}: {plural(len(files), 'image')}") diff --git a/src/morphobricks_preprocessing/run.py b/src/morphobricks_preprocessing/run.py index 01ffcb49..ebaa2e5f 100644 --- a/src/morphobricks_preprocessing/run.py +++ b/src/morphobricks_preprocessing/run.py @@ -26,7 +26,7 @@ import os import shutil import tempfile -from collections.abc import Callable, Iterable, Iterator, Sequence +from collections.abc import Callable, Iterable, Iterator from contextlib import contextmanager from dataclasses import dataclass, field, replace from datetime import UTC, datetime @@ -76,28 +76,16 @@ ) from .mask_checks import mask_issues from .masks import MaskInventory, find_masks -from .metadata import MetadataReport, TiffMetadata, inspect_images +from .metadata import MetadataReport, inspect_images from .planes import ( DEFAULT_MEMORY_LIMIT, Plane, PlaneReadError, check_planes, - iter_planes, iter_xyc_planes, - plane_layout, read_sample_plane, ) from .validation import ValidationIssue, error, has_errors, plural, warning -from .zarr_export import ( - ConversionSettings, - OmeZarrWriter, - copy_channels_current, - copy_units_current, - is_copy_path, - ome_zarr_relative_path, - read_physical_sizes, - zarr_available, -) logger = logging.getLogger(__name__) @@ -177,8 +165,6 @@ class RunResult: Why every row was recomputed, or None when the earlier table was updated. plane_count Number of rows in the written table. - converted - Files whose OME-Zarr copy was written in this run. design The experiment design written with the table, or None when the run was given none. @@ -201,7 +187,6 @@ class RunResult: excluded: tuple[str, ...] = () rebuild_reason: str | None = None plane_count: int = 0 - converted: tuple[str, ...] = () design: ExperimentDesign | None = None image_type: str | None = None intensity: IntensityReport | None = None @@ -347,8 +332,6 @@ def run_experiment( # noqa: C901, PLR0913, PLR0915 exclude: Iterable[str] = (), force_rebuild: bool = False, progress: Callable[[int, int, str], None] | None = None, - convert_tiffs: bool = False, - conversion: ConversionSettings | None = None, design: ExperimentDesign | None = None, masks_folder: str | os.PathLike | None = None, ome_zarr_labels: bool = False, @@ -381,16 +364,6 @@ def run_experiment( # noqa: C901, PLR0913, PLR0915 progress Called as ``progress(done, total, relative_path)`` before each file is read, and as ``progress(total, total, "")`` when reading is finished. - convert_tiffs - Convert every TIFF image to OME-Zarr in ``/ome_zarr`` and compute - its histograms from the converted copy, as for any OME-Zarr image (needs - the ``zarr`` extra). Copies of unchanged images are kept; copies of - images no longer converted are removed. Switching this recomputes the - rows of every TIFF image. - conversion - How converted copies are written (chunk size, compression, smaller - levels); defaults to :class:`ConversionSettings`. Changing it rewrites - the copies of every TIFF image. design The experiment design the user confirmed: how biological repeats pair across conditions, and the controls. It is brought up to date with the @@ -434,15 +407,12 @@ def run_experiment( # noqa: C901, PLR0913, PLR0915 ------ ValueError For an image type other than ``"fluorescence"`` and ``"rgb"``, or for - ``missing_as_singleton`` or ``convert_tiffs`` that is not True or False + ``missing_as_singleton`` that is not True or False (the manifest records them as JSON booleans, contract §9). """ if image_type is not None: check_image_type(image_type) - for name, value in ( - ("missing_as_singleton", missing_as_singleton), - ("convert_tiffs", convert_tiffs), - ): + for name, value in (("missing_as_singleton", missing_as_singleton),): # Refused before anything is read rather than when the manifest is written. if not isinstance(value, bool): raise ValueError(f"{name} must be True or False, found {value!r}") @@ -484,16 +454,6 @@ def run_experiment( # noqa: C901, PLR0913, PLR0915 histogram_settings=histogram_settings, ) ) - if convert_tiffs and not zarr_available(): - issues.append( - error( - "zarr_not_installed", - "Converting the TIFF images to OME-Zarr was requested, but the zarr package " - "is not installed.", - action='Install it with pip install "morphobricks-preprocessing[zarr]", or ' - "read the TIFF images directly.", - ) - ) def stop(**counts) -> RunResult: # noqa: ANN003 return RunResult(output, False, _sorted(issues), excluded=pre.excluded, **counts) @@ -501,21 +461,16 @@ def stop(**counts) -> RunResult: # noqa: ANN003 if has_errors(issues): return stop() - conversion = conversion or ConversionSettings() settings = RunSettings( bins, memory_limit, missing_as_singleton, pre.excluded, - convert_tiffs, - conversion, histogram=histogram_settings or HistogramSettings(), ) previous, previous_records, reason = _load_previous( output, settings, mapping, dtype.name, force_rebuild ) - # Copies written with other conversion settings are rewritten (their rows are the same). - same_conversion = previous is None or previous.settings.conversion == conversion labels = {image: leaf.labels for leaf in pre.discovery.leaves for image in leaf.images} in_blocks = {path for i in issues if i.code == "planes_read_in_blocks" for path in i.paths} headers = {f.relative_path: f for f in pre.metadata.files} @@ -523,7 +478,6 @@ def stop(**counts) -> RunResult: # noqa: ANN003 new, changed, unchanged = [], [], [] for rel, file_mapping in sorted(review.file_mappings.items()): header = headers[rel] - copy = ome_zarr_relative_path(rel) if convert_tiffs and not header.is_ome_zarr else None entries[rel] = file_entry_for( root, rel, @@ -531,7 +485,6 @@ def stop(**counts) -> RunResult: # noqa: ANN003 file_mapping, channel_names=header.channel_names, significant_bits=header.significant_bits, - ome_zarr_path=copy, read_in_blocks=rel in in_blocks, channel_names_trimmed=header.channel_names_trimmed, ) @@ -546,18 +499,6 @@ def stop(**counts) -> RunResult: # noqa: ANN003 before.same_source(entries[rel]) # During a rebuild there are no rows to check; otherwise the rows must be complete. and (reason is not None or len(previous_records.get(rel, ())) == before.plane_count) - # Rows of a converted TIFF are only kept while its copy is still there, as - # written with the same settings and with the axis units read now (copies - # written before each axis had its own unit are written again). - and ( - copy is None - or ( - (output / copy).is_dir() - and same_conversion - and copy_units_current(output / copy, header.physical_sizes) - and copy_channels_current(output / copy, header.channel_names) - ) - ) ): unchanged.append(rel) else: @@ -576,9 +517,6 @@ def stop(**counts) -> RunResult: # noqa: ANN003 "rebuild_reason": reason, } - to_convert = [rel for rel in to_read if entries[rel].ome_zarr_path] - staging = _CopyStaging(output, root, conversion) if to_convert else None - records: list[PlaneRecord] = [] narrow: list[NarrowRange] = [] for rel in unchanged if not reason else (): @@ -591,32 +529,24 @@ def stop(**counts) -> RunResult: # noqa: ANN003 # Kept rows keep the range checks they were written with. narrow.extend(entry for entry in previous.narrow_ranges if entry.relative_path == rel) failed: list[str] = [] - try: - for done, rel in enumerate(to_read): - if progress is not None: - progress(done, len(to_read), rel) - file_mapping = review.file_mappings[rel] - position = entries[rel].position_id - try: - if rel in to_convert: - # Convert first, then read the copy like any other OME-Zarr image. - copy, copy_mapping = staging.convert(rel, file_mapping, headers[rel], settings) - rows, ranges = _read_file( - copy, rel, labels[rel], copy_mapping, settings, position - ) - else: - rows, ranges = _read_file( - root / rel, rel, labels[rel], file_mapping, settings, position - ) - records.extend(rows) - narrow.extend(ranges) - except PlaneReadError: - logger.warning("Cannot read the planes of %s", rel, exc_info=True) - failed.append(rel) - except BaseException: - if staging is not None: - staging.discard() - raise + for done, rel in enumerate(to_read): + if progress is not None: + progress(done, len(to_read), rel) + try: + rows, ranges = _read_file( + root / rel, + rel, + labels[rel], + review.file_mappings[rel], + settings, + entries[rel].position_id, + ) + except PlaneReadError: + logger.warning("Cannot read the planes of %s", rel, exc_info=True) + failed.append(rel) + continue + records.extend(rows) + narrow.extend(ranges) if progress is not None: progress(len(to_read), len(to_read), "") if failed: @@ -629,16 +559,12 @@ def stop(**counts) -> RunResult: # noqa: ANN003 "Export the images again, or exclude them from the run to continue without them.", ) ) - if staging is not None: - staging.discard() return stop(read=tuple(to_read), **counts) issues.extend(_pixel_issues(records)) narrow.sort(key=lambda entry: (entry.plane_id, entry.channel)) issues.extend(_range_issues(narrow)) if has_errors(issues): - if staging is not None: - staging.discard() return stop(read=tuple(to_read), **counts) records.sort(key=_row_order) @@ -648,23 +574,7 @@ def stop(**counts) -> RunResult: # noqa: ANN003 for record in records: per_file[record.source_relative_path] = per_file.get(record.source_relative_path, 0) + 1 output.mkdir(parents=True, exist_ok=True) - # Copies recorded for images no longer analysed, or no longer converted, are - # removed; a damaged manifest cannot point the removal outside this brick's own - # ome_zarr folder. - removed_copies = [ - f.ome_zarr_path - for f in (previous.files if previous else ()) - if f.ome_zarr_path - and (f.relative_path not in entries or entries[f.relative_path].ome_zarr_path is None) - and is_copy_path(output, f.ome_zarr_path) - ] - with _staged_outputs( - output, - staging.root if staging is not None else None, - copies=[ome_zarr_relative_path(rel) for rel in to_convert], - removed_copies=removed_copies, - with_design=design is not None, - ) as staged: + with _staged_outputs(output, with_design=design is not None) as staged: csv_path = write_plane_histograms(records, staged / OUTPUT_FILENAME) design_sha256 = None if design is not None: @@ -702,7 +612,6 @@ def stop(**counts) -> RunResult: # noqa: ANN003 read=tuple(to_read), excluded=pre.excluded, plane_count=len(records), - converted=tuple(to_convert), design=design, image_type=image_type, intensity=intensity, @@ -715,25 +624,17 @@ class _OutputRollbackError(RuntimeError): @contextmanager -def _staged_outputs( - output: Path, - staged: Path | None = None, - *, - copies: Sequence[str] = (), - removed_copies: Sequence[str] = (), - with_design: bool = False, -) -> Iterator[Path]: +def _staged_outputs(output: Path, *, with_design: bool = False) -> Iterator[Path]: """Prepare all outputs before publishing, and retain backups if recovery fails. - ``staged`` is a temporary folder inside ``output`` that may already hold the - OME-Zarr ``copies`` written while reading; one is created if None. - ``with_design`` publishes ``experiment_design.csv`` too. + The outputs are written to a temporary folder inside ``output``; ``with_design`` + publishes ``experiment_design.csv`` too. """ - staged = staged or Path(tempfile.mkdtemp(prefix=".run-", dir=output)) + staged = Path(tempfile.mkdtemp(prefix=".run-", dir=output)) keep_backups = False try: yield staged - _publish_outputs(output, staged, copies, removed_copies, with_design) + _publish_outputs(output, staged, with_design) except _OutputRollbackError: keep_backups = True raise @@ -746,18 +647,11 @@ def _staged_outputs( logger.warning("Cannot remove temporary output folder %s", staged, exc_info=True) -def _publish_outputs( - output: Path, - staged: Path, - copies: Sequence[str] = (), - removed_copies: Sequence[str] = (), - with_design: bool = False, -) -> None: +def _publish_outputs(output: Path, staged: Path, with_design: bool = False) -> None: """Replace the outputs, restoring their previous state on a publication failure. - OME-Zarr copies are published first and the manifest last, so a manifest never - names a copy or design that is not in place. Earlier copies are moved into the - backup folder rather than copied, because they can be large. + The manifest is published last, so a manifest never names a design that is not in + place. """ filenames = (OUTPUT_FILENAME, *((DESIGN_FILENAME,) if with_design else ()), MANIFEST_FILENAME) backup = staged / "previous" @@ -768,34 +662,18 @@ def _publish_outputs( previous = output / name if previous.exists() or previous.is_symlink(): shutil.copy2(previous, backup / name, follow_symlinks=False) - # Undo steps in order: ("moved", copy) went to the backup, ("placed", copy) is new, - # ("file", name) is a published table or manifest. - done: list[tuple[str, str]] = [] + published: list[str] = [] try: - for target in removed_copies: - if (output / target).exists(): - _move(output / target, backup / target) - done.append(("moved", target)) - for target in copies: - if (output / target).exists(): - _move(output / target, backup / target) - done.append(("moved", target)) - _move(staged / target, output / target) - done.append(("placed", target)) for name in filenames: os.replace(staged / name, output / name) - done.append(("file", name)) + published.append(name) except BaseException: try: - for kind, item in reversed(done): - if kind == "placed": - shutil.rmtree(output / item) - elif kind == "moved": - _move(backup / item, output / item) - elif (backup / item).exists() or (backup / item).is_symlink(): - os.replace(backup / item, output / item) + for name in reversed(published): + if (backup / name).exists() or (backup / name).is_symlink(): + os.replace(backup / name, output / name) else: - (output / item).unlink() + (output / name).unlink() except BaseException as exc: raise _OutputRollbackError( "The output update failed and the earlier files could not be fully restored. " @@ -845,64 +723,6 @@ def _read_file( return records, narrow -def _move(source: Path, destination: Path) -> None: - """Rename a file or folder, creating the destination's parent folders.""" - destination.parent.mkdir(parents=True, exist_ok=True) - os.replace(source, destination) - - -class _CopyStaging: - """The run's temporary folder while TIFF images are converted to OME-Zarr. - - The copies are published with the table and manifest by :func:`_staged_outputs`, - which reuses this folder; :meth:`discard` removes it when the run stops early. - """ - - def __init__(self, output: Path, experiment_root: Path, settings: ConversionSettings): # noqa: ANN204 - self.output = output - self.experiment_root = experiment_root - self.settings = settings - self.created_output = not output.exists() - output.mkdir(parents=True, exist_ok=True) - self.root = Path(tempfile.mkdtemp(prefix=".run-", dir=output)) - - def writer( - self, rel: str, mapping: AxisMapping, header: TiffMetadata, memory_limit: int - ) -> OmeZarrWriter: - """Create the writer of one image's copy inside the temporary folder.""" - source = self.experiment_root / rel - return OmeZarrWriter.create( - self.root / ome_zarr_relative_path(rel), - mapping, - plane_layout(source, mapping), - sizes=read_physical_sizes(source), - channel_names=header.channel_names, - name=rel, - settings=self.settings, - memory_limit=memory_limit, - ) - - def convert( - self, rel: str, mapping: AxisMapping, header: TiffMetadata, settings: RunSettings - ) -> tuple[Path, AxisMapping]: - """Convert one TIFF plane by plane; return the copy and its own axis mapping.""" - writer = self.writer(rel, mapping, header, settings.memory_limit) - for plane in iter_planes( - self.experiment_root / rel, - mapping, - memory_limit=settings.memory_limit, - bin_count=settings.bin_count, - ): - writer.write(plane) - return writer.finish(), writer.mapping - - def discard(self) -> None: - """Remove the temporary folder, and the output folder if this run created it.""" - shutil.rmtree(self.root, ignore_errors=True) - if self.created_output and not any(self.output.iterdir()): - self.output.rmdir() - - def _load_previous( output: Path, settings: RunSettings, mapping: AxisMapping, dtype: str, force: bool ) -> tuple[RunManifest | None, dict[str, list[PlaneRecord]], str | None]: diff --git a/src/morphobricks_preprocessing/zarr_export.py b/src/morphobricks_preprocessing/zarr_export.py deleted file mode 100644 index 69b14b6a..00000000 --- a/src/morphobricks_preprocessing/zarr_export.py +++ /dev/null @@ -1,622 +0,0 @@ -"""Write OME-Zarr copies of the source images (OME-NGFF 0.4). - -An optional part of a run: each analysed TIFF is copied to -``/ome_zarr/.ome.zarr`` so that later bricks and viewers -can read it lazily, chunk by chunk. The source images are never changed. - -The copy is written plane by plane, so the image is never held in memory as a -whole. A plane too large for the memory limit is written block by block; its -smaller levels are then built from the stored level above, a region at a time, -so they equal those of a plane written whole. Axes follow the -OME-NGFF order ``t, c, z, y, x`` with the roles of the confirmed axis mapping -(axes without a role are dropped; they have length 1). Pixel values are copied -unchanged at level 0; smaller levels for viewers are 2 x 2 averages, rounded to -the pixel type. Physical pixel sizes and channel names are taken from OME or -ImageJ metadata when present. - -Needs the ``zarr`` extra (``pip install "morphobricks-preprocessing[zarr]"``). -Zarr 2.18 and 3.x are supported; the files are always written in Zarr format 2, -which OME-NGFF 0.4 uses. -""" - -from __future__ import annotations - -import json -import logging -import math -import os -import shutil -from collections.abc import Iterable, Sequence -from dataclasses import dataclass, field -from pathlib import Path, PurePosixPath - -import numpy as np -from morphobricks_normalization.dtypes import is_integer - -from .axes import AxisMapping -from .physical_sizes import ( # noqa: F401 - moved there, still importable from here - UNKNOWN_SCALE_KEY, - PhysicalSizes, - read_physical_sizes, -) -from .planes import DEFAULT_MEMORY_LIMIT, Plane, PlaneLayout -from .zarr_input import ( # noqa: F401 - re-exported for callers of this module - ZarrUnavailableError, - _zarr, - _zarr_major, - canonical_dtype, - zarr_available, -) - -logger = logging.getLogger(__name__) - -NGFF_VERSION = "0.4" -OME_ZARR_FOLDER = "ome_zarr" -OME_ZARR_SUFFIX = ".ome.zarr" - -# OME-NGFF 0.4 axis order, and the axis type of each role. -AXIS_ORDER = ("T", "C", "Z", "Y", "X") -_AXIS_TYPES = {"T": "time", "C": "channel", "Z": "space", "Y": "space", "X": "space"} - -# Levels are added for viewers until the plane fits in this many pixels per side. -PYRAMID_TARGET = 512 -MAX_LEVELS = 8 -CHUNK_SIDE = 1024 - - -def ome_zarr_relative_path(source_relative_path: str) -> str: - """Return where the copy of an image goes, relative to the output folder. - - ``"ctrl/rep1/img_01.tif"`` becomes ``"ome_zarr/ctrl/rep1/img_01.ome.zarr"``; - an ``.ome.tif`` or ``.ome.tiff`` file loses its whole suffix. - """ - path = PurePosixPath(source_relative_path) - name = path.name - for suffix in (".ome.tiff", ".ome.tif", ".tiff", ".tif"): - if name.lower().endswith(suffix): - name = name[: -len(suffix)] - break - return (PurePosixPath(OME_ZARR_FOLDER) / path.parent / (name + OME_ZARR_SUFFIX)).as_posix() - - -COMPRESSIONS = { - "lz4": "Blosc LZ4 (fast)", - "zstd": "Blosc Zstandard (smaller, slower)", - "none": "no compression", -} -"""Lossless compressions offered for converted images; pixel values never change.""" - -MIN_CHUNK_SIZE = 16 -MAX_CHUNK_SIZE = 8192 - - -@dataclass(frozen=True) -class ConversionSettings: - """How TIFF images are written when they are converted to OME-Zarr. - - Attributes - ---------- - chunk_size - Side of the stored tiles in X and Y, in pixels (16 to 8,192). Along Z, C - and T every chunk holds one plane. - compression - ``"lz4"`` (Blosc LZ4, default), ``"zstd"`` (Blosc Zstandard) or ``"none"``. - All are lossless. - compression_level - 1 (fastest) to 9 (smallest); ignored without compression. - pyramid_levels - Smaller levels added for viewers: None to halve until a plane fits 512 - pixels (default), 0 for none, or a number of levels (1 to 7). - - Raises - ------ - ValueError - If a setting is outside its allowed range. - """ - - chunk_size: int = CHUNK_SIDE - compression: str = "lz4" - compression_level: int = 5 - pyramid_levels: int | None = None - - def __post_init__(self) -> None: - if not MIN_CHUNK_SIZE <= int(self.chunk_size) <= MAX_CHUNK_SIZE: - raise ValueError( - f"The chunk size must be between {MIN_CHUNK_SIZE} and {MAX_CHUNK_SIZE:,} " - f"pixels; got {self.chunk_size}." - ) - if self.compression not in COMPRESSIONS: - raise ValueError( - f"Unknown compression {self.compression!r}; choose one of {sorted(COMPRESSIONS)}." - ) - if not 1 <= int(self.compression_level) <= 9: - raise ValueError( - f"The compression level must be between 1 and 9; got {self.compression_level}." - ) - if self.pyramid_levels is not None and not 0 <= int(self.pyramid_levels) < MAX_LEVELS: - raise ValueError( - f"The number of pyramid levels must be between 0 and {MAX_LEVELS - 1}; " - f"got {self.pyramid_levels}." - ) - - def describe(self) -> str: - """Return a plain-language summary, e.g. for the run summary.""" - compression = COMPRESSIONS[self.compression].split(" (")[0] - if self.compression != "none": - compression += f" level {self.compression_level}" - if self.pyramid_levels is None: - pyramid = "smaller levels until a plane fits 512 px" - elif self.pyramid_levels == 0: - pyramid = "no smaller levels" - else: - pyramid = f"{self.pyramid_levels} smaller level(s)" - side = self.chunk_size - return f"chunks of {side} x {side} px, {compression}, {pyramid}" - - def to_dict(self) -> dict: - """Return the settings as JSON-compatible data.""" - return { - "chunk_size": self.chunk_size, - "compression": self.compression, - "compression_level": self.compression_level, - "pyramid_levels": self.pyramid_levels, - } - - @classmethod - def from_dict(cls, data: dict | None) -> ConversionSettings: - """Build settings from :meth:`to_dict` data; missing keys get the defaults.""" - data = data or {} - levels = data.get("pyramid_levels") - return cls( - chunk_size=int(data.get("chunk_size", CHUNK_SIDE)), - compression=str(data.get("compression", "lz4")), - compression_level=int(data.get("compression_level", 5)), - pyramid_levels=None if levels is None else int(levels), - ) - - -@dataclass(frozen=True) -class ConversionEstimate: - """How much disk space converting images to OME-Zarr may take. - - Attributes - ---------- - image_count - TIFF images to convert. - source_bytes - Size of these TIFF files on disk. - copy_bytes - Uncompressed size of the copies, including the smaller levels. It is an - upper estimate: compression usually makes the copies smaller. - """ - - image_count: int - source_bytes: int - copy_bytes: int - - -def estimate_conversion( - experiment_root: str | os.PathLike, - files: Iterable, - settings: ConversionSettings | None = None, -) -> ConversionEstimate: - """Estimate the disk space for converting the TIFF images among ``files``. - - Parameters - ---------- - experiment_root - The experiment folder. - files - Image headers (:class:`~morphobricks_preprocessing.metadata.TiffMetadata`); - OME-Zarr images and unreadable files are left out, as they are not converted. - settings - The conversion settings; they decide the smaller levels. - """ - settings = settings or ConversionSettings() - root = Path(experiment_root) - count = source = copy = 0 - for f in files: - if not f.readable or f.is_ome_zarr or f.dtype is None or len(f.shape) < 2: - continue - count += 1 - try: - source += (root / f.relative_path).stat().st_size - except OSError: - pass - y, x = f.shape[-2], f.shape[-1] - levels = pyramid_shapes((y, x), settings.pyramid_levels) - factor = sum(ly * lx for ly, lx in levels) / max(y * x, 1) - copy += int(math.prod(f.shape) * np.dtype(f.dtype).itemsize * factor) - return ConversionEstimate(count, source, copy) - - -def free_space(path: str | os.PathLike) -> int | None: - """Return the free space on the drive holding ``path`` (or its nearest existing parent). - - Returns None when it cannot be determined. - """ - path = Path(path).resolve() - for candidate in (path, *path.parents): - if candidate.exists(): - try: - return shutil.disk_usage(candidate).free - except OSError: - return None - return None - - -def pyramid_shapes( - plane_shape: tuple[int, int], levels: int | None = None -) -> list[tuple[int, int]]: - """Return (Y, X) of every level: level 0, then halved planes. - - With ``levels`` None, planes are halved until they fit ``PYRAMID_TARGET`` - (at most ``MAX_LEVELS`` levels in all); otherwise ``levels`` smaller levels - are added, stopping early once a plane is a single pixel. - """ - shapes = [tuple(plane_shape)] - while len(shapes) < MAX_LEVELS: - if levels is None and max(shapes[-1]) <= PYRAMID_TARGET: - break - if levels is not None and (len(shapes) > levels or max(shapes[-1]) <= 1): - break - y, x = shapes[-1] - shapes.append((math.ceil(y / 2), math.ceil(x / 2))) - return shapes - - -def downsample(pixels: np.ndarray) -> np.ndarray: - """Average 2 x 2 blocks (edge pixels repeated for odd sizes), keeping the dtype. - - Integer results are rounded and stay within the type's range; NaN and - infinity propagate as in a plain average. - """ - y, x = pixels.shape - padded = np.pad(pixels, ((0, y % 2), (0, x % 2)), mode="edge").astype(np.float64) - means = padded.reshape(padded.shape[0] // 2, 2, padded.shape[1] // 2, 2).mean(axis=(1, 3)) - if is_integer(pixels.dtype): - info = np.iinfo(pixels.dtype) - return np.clip(np.rint(means), info.min, info.max).astype(pixels.dtype) - return means.astype(pixels.dtype) - - -@dataclass -class OmeZarrWriter: - """Writes one image as OME-Zarr, one plane at a time. - - Use :meth:`create`, then :meth:`write` every plane, then :meth:`finish`. - Planes read in blocks are written block by block. - - Attributes - ---------- - path - The ``.ome.zarr`` folder being written. - roles - Roles of the array axes, in OME-NGFF order (e.g. ``("C", "Z", "Y", "X")``). - shape - Shape of level 0. - memory_limit - Working memory allowed while the smaller levels of a plane read in - blocks are built, in bytes. - """ - - path: Path - roles: tuple[str, ...] - shape: tuple[int, ...] - dtype: np.dtype - sizes: PhysicalSizes - channel_names: tuple[str | None, ...] = () - name: str = "" - settings: ConversionSettings = field(default_factory=ConversionSettings) - memory_limit: int = DEFAULT_MEMORY_LIMIT - _levels: list = field(default_factory=list, repr=False) - _channel_ranges: dict = field(default_factory=dict, repr=False) - - @classmethod - def create( - cls, - path: str | os.PathLike, - mapping: AxisMapping, - layout: PlaneLayout, - *, - sizes: PhysicalSizes | None = None, - channel_names: Sequence[str | None] = (), - name: str = "", - settings: ConversionSettings | None = None, - memory_limit: int = DEFAULT_MEMORY_LIMIT, - ) -> OmeZarrWriter: - """Create the empty arrays of an image with ``mapping`` and ``layout``.""" - roles = tuple(r for r in AXIS_ORDER if r in ("Y", "X") or r in mapping.plane_roles) - lengths = {"Y": layout.plane_shape[0], "X": layout.plane_shape[1]} - for role in mapping.plane_roles: - dim = mapping.index_of(role) - lengths[role] = 1 if dim is None else layout.shape[dim] - writer = cls( - path=Path(path), - roles=roles, - shape=tuple(lengths[r] for r in roles), - dtype=canonical_dtype(layout.dtype), - sizes=sizes or PhysicalSizes(), - channel_names=tuple(channel_names), - name=name, - settings=settings or ConversionSettings(), - memory_limit=memory_limit, - ) - writer._open() - return writer - - @property - def mapping(self) -> AxisMapping: - """Axis mapping of the copy itself, for reading it back plane by plane.""" - return AxisMapping("".join(self.roles), self.roles) - - def write(self, plane: Plane) -> None: - """Store one plane at its coordinates, at every level.""" - index = tuple( - (plane.coordinates[r.lower()] or 0) for r in self.roles if r not in ("Y", "X") - ) - if plane.in_blocks: - self._write_blocks(index, plane) - return - pixels = plane.pixels.view(self.dtype) - for level, array in enumerate(self._levels): - if level: - pixels = downsample(pixels) - array[index] = pixels - if "C" in self.roles: - self._track_channel(plane.c or 0, plane.pixels) - - def _write_blocks(self, index: tuple[int, ...], plane: Plane) -> None: - """Write a plane read in blocks, then build its smaller levels from the stored ones.""" - level0 = self._levels[0] - for block in plane.blocks(): - rows, cols = block.pixels.shape - region = (slice(block.y, block.y + rows), slice(block.x, block.x + cols)) - level0[(*index, *region)] = block.pixels.view(self.dtype) - if "C" in self.roles: - self._track_channel(plane.c or 0, block.pixels) - for above, array in zip(self._levels, self._levels[1:], strict=False): - height, width = above.shape[-2:] - # Regions of even size at even positions halve exactly like the whole plane; - # only the last row and column of the plane are repeated for odd sizes. - side = self._region_side() - for top in range(0, height, side): - for left in range(0, width, side): - pixels = np.asarray( - above[(*index, slice(top, top + side), slice(left, left + side))] - ) - smaller = downsample(pixels) - rows, cols = smaller.shape - target = (slice(top // 2, top // 2 + rows), slice(left // 2, left // 2 + cols)) - array[(*index, *target)] = smaller - - def _region_side(self) -> int: - """Side of the regions read to build a smaller level: even, within the memory limit.""" - side = 2 * self.settings.chunk_size - # The region, its float64 copy and means, and the chunks read and written. - chunk = self.settings.chunk_size**2 * self.dtype.itemsize - while ( - side > 2 - and side * side * (2 * self.dtype.itemsize + 12) + 5 * chunk > self.memory_limit - ): - side = max(2, side // 4 * 2) - return side - - def finish(self) -> Path: - """Write the OME-NGFF metadata; return the image folder.""" - self._group.attrs.update(self.attributes()) - return self.path - - def attributes(self) -> dict: - """Return the OME-NGFF 0.4 ``multiscales`` (and ``omero`` for channels) metadata.""" - axes = [] - for role in self.roles: - axis = {"name": role.lower(), "type": _AXIS_TYPES[role]} - unit = self.sizes.unit(role) - if unit: - axis["unit"] = unit - axes.append(axis) - datasets = [] - for level in range(len(self._levels)): - factor = 2**level - scale = [ - self.sizes.scale(r) * (factor if r in ("Y", "X") else 1) if r != "C" else 1.0 - for r in self.roles - ] - datasets.append( - { - "path": str(level), - "coordinateTransformations": [{"type": "scale", "scale": scale}], - } - ) - multiscale = { - "version": NGFF_VERSION, - "axes": axes, - "datasets": datasets, - "type": "mean", - "metadata": {"description": "2 x 2 block mean; level 0 holds the source values"}, - } - # OME-NGFF 0.4 requires a scale for every axis, so an unknown size is written as 1; - # say which, so that 1 is never read as a real size (issue #39). - unknown = _unknown_scale(self.roles, self.sizes) - if unknown: - multiscale["metadata"][UNKNOWN_SCALE_KEY] = unknown - if self.name: - multiscale["name"] = self.name - attrs = {"multiscales": [multiscale]} - if "C" in self.roles: - attrs["omero"] = self._omero() - return attrs - - def _open(self) -> None: - self._group = _create_group(self.path) - self._levels = [] - side = self.settings.chunk_size - compressor = _compressor(self.settings) - for level, (y, x) in enumerate( - pyramid_shapes(self.shape[-2:], self.settings.pyramid_levels) - ): - shape = (*self.shape[:-2], y, x) - chunks = (*(1,) * (len(shape) - 2), min(y, side), min(x, side)) - self._levels.append( - _create_array(self._group, str(level), shape, chunks, self.dtype, compressor) - ) - - def _track_channel(self, channel: int, pixels: np.ndarray) -> None: - finite = pixels[np.isfinite(pixels)] if pixels.dtype.kind == "f" else pixels - if finite.size == 0: - return - low, high = float(finite.min()), float(finite.max()) - old = self._channel_ranges.get(channel) - self._channel_ranges[channel] = ( - (low, high) - if old is None - else ( - min(old[0], low), - max(old[1], high), - ) - ) - - def _omero(self) -> dict: - channels = [] - for c in range(self.shape[self.roles.index("C")]): - low, high = self._channel_ranges.get(c, (0.0, 1.0)) - channel = { - "active": True, - "color": "FFFFFF", - "window": {"min": low, "max": high, "start": low, "end": high}, - } - # Only names from the source's metadata are written. An unnamed channel gets - # no label rather than a made-up one, so the copy reads back unnamed, like its - # source, and never fails the exact channel-name check (issue #37). - label = _channel_label(self.channel_names, c) - if label is not None: - channel["label"] = label - channels.append(channel) - return {"version": NGFF_VERSION, "channels": channels} - - -def _channel_label(names: Sequence[str | None], channel: int) -> str | None: - """Return the name of ``channel`` in ``names``; None when it is unnamed.""" - return (names[channel] if channel < len(names) else None) or None - - -def is_copy_path(output_folder: str | os.PathLike, relative_path: str) -> bool: - """Return True if ``relative_path`` names an OME-Zarr copy inside ``/ome_zarr``. - - Used before moving or removing a copy recorded in a manifest, so that a damaged - manifest cannot point outside this brick's own copies. - """ - output = Path(output_folder) - base = (output / OME_ZARR_FOLDER).resolve() - target = (output / relative_path).resolve() - return base in target.parents and target.name.endswith(OME_ZARR_SUFFIX) - - -def _unknown_scale(roles: Iterable[str], sizes: PhysicalSizes) -> list[str]: - """Return the axes (``"x"``, ...) whose size is not known, so their scale 1 is a placeholder.""" - return [role.lower() for role in roles if role in "XYZT" and not sizes.known(role)] - - -def copy_units_current(copy: str | os.PathLike, sizes: PhysicalSizes) -> bool: - """Return True if a copy's axis units are those its source's sizes give now. - - Copies written before each axis had its own unit (plan §12 question 24) may - carry X's unit on Y or Z, and copies written before the unknown sizes were marked - (``unknown_scale``, issue #39) read as size 1; such copies are written again on the next - run. Only - the copy's metadata file is read. A copy whose metadata cannot be read counts - as out of date, so it is written again rather than trusted. - - Parameters - ---------- - copy - The copy's folder (``/ome_zarr/....ome.zarr``). - sizes - The source image's physical sizes, as :func:`read_physical_sizes` gives them. - """ - try: - attrs = json.loads((Path(copy) / ".zattrs").read_text(encoding="utf-8")) - axes = attrs["multiscales"][0]["axes"] - roles = [str(axis["name"]).upper() for axis in axes] - marked = attrs["multiscales"][0].get("metadata", {}).get(UNKNOWN_SCALE_KEY, []) - return marked == _unknown_scale(roles, sizes) and all( - axis.get("unit") == sizes.unit(role) for axis, role in zip(axes, roles, strict=True) - ) - except (OSError, ValueError, KeyError, IndexError, TypeError, AttributeError): - logger.debug("Cannot read the axis units of the copy %s", copy, exc_info=True) - return False - - -def copy_channels_current(copy: str | os.PathLike, channel_names: Sequence[str | None]) -> bool: - """Return True if a copy's channel labels are exactly the names its source gives now. - - Copies written before unnamed channels were left without a label carry the - placeholder ``Channel ``; such copies are written again on the next run, so the - copy never names a channel its source does not. Only the copy's metadata file is - read. A copy whose metadata cannot be read counts as out of date. - - Parameters - ---------- - copy - The copy's folder (``/ome_zarr/....ome.zarr``). - channel_names - The source image's channel names, as its header gives them (None when unnamed). - """ - try: - attrs = json.loads((Path(copy) / ".zattrs").read_text(encoding="utf-8")) - if "omero" not in attrs: - return True - channels = attrs["omero"]["channels"] - return all( - channel.get("label") == _channel_label(channel_names, c) - for c, channel in enumerate(channels) - ) - except (OSError, ValueError, KeyError, IndexError, TypeError, AttributeError): - logger.debug("Cannot read the channel labels of the copy %s", copy, exc_info=True) - return False - - -# zarr 2 / 3 --------------------------------------------------------------------------- - - -def _create_group(path: Path): # noqa: ANN202 - zarr = _zarr() - if _zarr_major() >= 3: - return zarr.open_group(str(path), mode="w", zarr_format=2) - return zarr.open_group(str(path), mode="w") - - -def _compressor(settings: ConversionSettings): # noqa: ANN202 - """Return the numcodecs compressor for ``settings``; None means no compression.""" - if settings.compression == "none": - return None - import numcodecs # installed with zarr - - return numcodecs.Blosc( - cname=settings.compression, - clevel=settings.compression_level, - shuffle=numcodecs.Blosc.SHUFFLE, - ) - - -def _create_array(group, name: str, shape, chunks, dtype, compressor): # noqa: ANN001, ANN202 - if _zarr_major() >= 3: - return group.create_array( - name, - shape=shape, - chunks=chunks, - dtype=dtype, - fill_value=0, - compressors=compressor, - chunk_key_encoding={"name": "v2", "separator": "/"}, - ) - return group.create_dataset( - name, - shape=shape, - chunks=chunks, - dtype=dtype, - fill_value=0, - compressor=compressor, - dimension_separator="/", - ) diff --git a/tests/simulation.py b/tests/simulation.py index 70309e89..488c4661 100644 --- a/tests/simulation.py +++ b/tests/simulation.py @@ -9,8 +9,8 @@ from __future__ import annotations import numpy as np - -from morphobricks_preprocessing import PlaneRecord, compute_plane_histogram +from morphobricks_normalization.histogram import compute_plane_histogram +from morphobricks_normalization.records import PlaneRecord def simulate_plane(rng, dtype, shape=(64, 64), brightness=1.0, n_cells=6): diff --git a/tests/test_cli.py b/tests/test_cli.py index 573d1c86..59222a4f 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -4,13 +4,13 @@ import numpy as np import pytest +from morphobricks_normalization.csv_io import read_plane_histograms from morphobricks_preprocessing import ( default_design, default_output_folder, preflight, read_manifest, - read_plane_histograms, write_experiment_design, ) from morphobricks_preprocessing.cli import main, parse_size @@ -251,41 +251,3 @@ def test_python_dash_m(experiment): ) assert completed.returncode == 0, completed.stderr assert "--axes ZCYX" in completed.stdout - - -@pytest.mark.parametrize( - ("text", "levels"), [("auto", None), ("none", 0), ("3", 3), ("AUTO", None)] -) -def test_parse_pyramid_levels(text, levels): - from morphobricks_preprocessing.cli import parse_pyramid_levels - - assert parse_pyramid_levels(text) == levels - - -@pytest.mark.parametrize("text", ["8", "-1", "many"]) -def test_parse_pyramid_levels_rejects_bad_values(text): - from morphobricks_preprocessing.cli import parse_pyramid_levels - - with pytest.raises(argparse.ArgumentTypeError): - parse_pyramid_levels(text) - - -def test_conversion_options(experiment, tmp_path, capsys): - pytest.importorskip("zarr") - out_dir = tmp_path / "out" - args = ["run", str(experiment), "--axes", "ZCYX", "--output", str(out_dir), "--convert-tiff"] - args += ["--chunk-size", "64", "--compression", "zstd", "--compression-level", "7"] - args += ["--pyramid-levels", "none"] - confirm_design(experiment, out_dir) - assert main(args) == 0 - out = capsys.readouterr().out - assert "Conversion (chunks of 64 x 64 px, Blosc Zstandard level 7, no smaller levels)" in out - assert "so the space they take is duplicated" in out and "free space on the output" in out - settings = read_manifest(out_dir / "run_manifest.json").settings - assert settings.convert_tiffs - assert (settings.conversion.chunk_size, settings.conversion.compression) == (64, "zstd") - - -def test_invalid_chunk_size_is_refused(experiment, capsys): - assert main(["run", str(experiment), "--axes", "ZCYX", "--chunk-size", "4"]) == 1 - assert "chunk size must be between 16" in capsys.readouterr().out diff --git a/tests/test_discovery.py b/tests/test_discovery.py index 55b32244..8e29019f 100644 --- a/tests/test_discovery.py +++ b/tests/test_discovery.py @@ -2,9 +2,9 @@ import numpy as np import pytest +from morphobricks_normalization.records import SINGLE from morphobricks_preprocessing import ( - SINGLE, AxisMapping, Severity, check_output_folder, diff --git a/tests/test_header_axes.py b/tests/test_header_axes.py index 66720573..8664b911 100644 --- a/tests/test_header_axes.py +++ b/tests/test_header_axes.py @@ -1,7 +1,11 @@ -"""The shared header-axes rules (MorphoBricks D86, API A4, decision D-d).""" +"""The shared header-axes rules (MorphoBricks D86, API A4, decision D-d). + +``check_declared_axes`` and ``declared_axes_transposed`` were removed in v0.4.0 (spec +revision 4): bricks take no axis declarations, and the axis check +(``tests/test_axis_check.py``) compares confirmed roles with these rules. +""" import dataclasses -import itertools import numpy as np import pytest @@ -11,8 +15,6 @@ OPEN_AXIS, PLACEHOLDER_AXES, TiffMetadata, - check_declared_axes, - declared_axes_transposed, header_axes, inspect_image, named_axes, @@ -21,23 +23,6 @@ ) from tiff_tree import write_data -# check_declared_axes and declared_axes_transposed are deprecated (removed before the v0.4.0 -# release, design v0.4 §6); their rules stay tested until then, and the warning has its own -# test below. -pytestmark = pytest.mark.filterwarnings( - "ignore:morphobricks_preprocessing.(check_declared_axes|declared_axes_transposed) is " - "deprecated:DeprecationWarning" -) - - -@pytest.mark.filterwarnings("error") -def test_declared_axes_checks_are_deprecated(tmp_path): - meta = inspect_image(write_data(tmp_path / "a.tif", np.zeros((2, 4, 5), np.uint8), "CYX")) - with pytest.warns(DeprecationWarning, match="check_declared_axes is deprecated.*v0.4.0"): - assert check_declared_axes(meta, "CYX") is None - with pytest.warns(DeprecationWarning, match="declared_axes_transposed is deprecated"): - assert declared_axes_transposed(meta, "CXY") - def imagej(path, data, axes): path.parent.mkdir(parents=True, exist_ok=True) @@ -45,6 +30,11 @@ def imagej(path, data, axes): return path +def ome(path, data, axes): + tifffile.imwrite(path, data, ome=True, metadata={"axes": axes}) + return path + + def plain(path, data): path.parent.mkdir(parents=True, exist_ok=True) tifffile.imwrite(path, data, photometric="minisblack", metadata=None) @@ -138,9 +128,8 @@ def test_a_header_built_by_hand_reads_strictly(): # Without the flag, a placeholder stack's Y and X stay binding (refuse rather than guess). meta = TiffMetadata("a.tif", True, (3, 8, 8), np.dtype("uint8"), "QYX", 1, 3) assert named_axes(meta) == "?YX" - assert check_declared_axes(meta, "YXC").code == "declared_axes_mismatch" plain_stack = dataclasses.replace(meta, axes_from_metadata=False) - assert check_declared_axes(plain_stack, "YXC") is None + assert named_axes(plain_stack) == "???" def test_header_axes_of_an_unreadable_file(tmp_path): @@ -157,222 +146,6 @@ def test_header_axes_of_an_ome_zarr_image(tmp_path): assert header_axes(inspect_image(path, "a.ome.zarr")) == "TCYX" -# check_declared_axes: the report's probes --------------------------------------------------- - - -def test_a_depth_stack_declared_as_channels_is_refused(tmp_path): - # MF-1: an ImageJ ZYX image declared image_axes="CYX" had its Z slices read as channels. - meta = inspect_image(imagej(tmp_path / "a.tif", np.zeros((3, 8, 8), np.uint8), "ZYX"), "a.tif") - issue = check_declared_axes(meta, "CYX") - assert issue.code == "declared_axes_mismatch" and issue.is_error - assert issue.paths == ("a.tif",) - assert "'CYX'" in issue.message and "'ZYX'" in issue.message - assert check_declared_axes(meta, "ZYX") is None - assert check_declared_axes(meta, "zyx") is None - - -@pytest.mark.parametrize( - ("header", "declared"), - [ - ("ZYX", "TYX"), # CP-4: a ZYX mask declared TYX (Z slices used as time points) - ("ZYX", "CYX"), # MR-1: a ZYX mask written into the project as CYX - ("TYX", "ZYX"), # MF-1: a TYX mask declared ZYX - ], -) -def test_a_mask_declared_with_other_axes_is_refused(tmp_path, header, declared): - meta = inspect_image(imagej(tmp_path / "m.tif", labels((3, 8, 8)), header), "m.tif") - assert check_declared_axes(meta, declared).code == "declared_axes_mismatch" - assert check_declared_axes(meta, header) is None - - -def test_shaped_axes_are_trusted(tmp_path): - # MR-4 and decision D-d: a shaped CYX TIFF is CYX, and only CYX. - meta = inspect_image(write_data(tmp_path / "a.tif", np.zeros((3, 8, 8), np.uint8), "CYX")) - assert check_declared_axes(meta, "CYX") is None - assert check_declared_axes(meta, "ZYX").code == "declared_axes_mismatch" - - -def test_a_placeholder_axis_may_be_declared_depth_time_or_channel(tmp_path): - meta = inspect_image(plain(tmp_path / "a.tif", np.zeros((3, 8, 8), np.uint8)), "a.tif") - assert meta.axes == "IYX" - for declared in ("ZYX", "TYX", "CYX"): - assert check_declared_axes(meta, declared) is None - - -def shaped_without_axes(path, data): - return write_data(path, data) - - -@pytest.mark.parametrize("write", [plain, shaped_without_axes]) -@pytest.mark.parametrize("declared", ["YXC", "CYX", "ZYX", "TYX", "XYZ", "yxc"]) -def test_a_plain_stack_leaves_every_dimension_open(tmp_path, write, declared): - # The Y and X tifffile reports for a plain stack are only where each page's rows and - # columns are stored: a (Y, X, C) array written without metadata reads as QYX or IYX. - meta = inspect_image(write(tmp_path / "a.tif", np.zeros((8, 8, 3), np.uint8)), "a.tif") - assert meta.axes in ("QYX", "IYX") - assert check_declared_axes(meta, declared) is None - - -def test_a_plain_stack_of_two_unnamed_dimensions_is_open(tmp_path): - meta = inspect_image(write_data(tmp_path / "a.tif", np.zeros((2, 3, 8, 8), np.uint8))) - assert meta.axes == "QQYX" - for declared in ("ZCYX", "TZYX", "YXZC", "CYXT"): - assert check_declared_axes(meta, declared) is None - - -@pytest.mark.parametrize("declared", ["YX", "ZCYX", "ZZX", "ZYQ", "YXS", "", None]) -def test_a_plain_stack_still_needs_one_role_per_dimension(tmp_path, declared): - meta = inspect_image(plain(tmp_path / "a.tif", np.zeros((3, 8, 8), np.uint8)), "a.tif") - assert check_declared_axes(meta, declared).code == "declared_axes_mismatch" - - -@pytest.mark.parametrize("declared", ["CTZ", "ZCX", "YCT"]) -def test_a_plain_stack_must_still_declare_y_and_x(tmp_path, declared): - # Its Y and X are open, but the image still has rows and columns: a declaration - # without them reads no plane at all. - meta = inspect_image(plain(tmp_path / "a.tif", np.zeros((3, 8, 8), np.uint8)), "a.tif") - issue = check_declared_axes(meta, declared) - assert issue.code == "declared_axes_mismatch" - assert "not declared" in issue.message - - -def test_named_axes_stay_binding_beside_a_placeholder(tmp_path): - # Axes metadata that names Y and X but leaves one dimension open: only that one is open. - path = write_data(tmp_path / "a.tif", np.zeros((3, 8, 8), np.uint8), "QYX") - meta = inspect_image(path, "a.tif") - assert check_declared_axes(meta, "CYX") is None - assert check_declared_axes(meta, "YXC").code == "declared_axes_mismatch" - - -def test_rgb_samples_of_a_plain_stack_stay_binding(tmp_path): - # S comes from the SamplesPerPixel tag, not from where the pages are stored. - path = tmp_path / "rgb.tif" - tifffile.imwrite(path, np.zeros((2, 8, 8, 3), np.uint8), photometric="rgb", metadata=None) - meta = inspect_image(path, "rgb.tif") - assert meta.axes == "IYXS" - assert check_declared_axes(meta, "TYXC") is None - assert check_declared_axes(meta, "TYXZ").code == "declared_axes_mismatch" - - -def test_an_ome_channel_stack_declared_samples_last_is_refused(tmp_path): - path = tmp_path / "a.ome.tif" - tifffile.imwrite(path, np.zeros((3, 8, 8), np.uint8), ome=True, metadata={"axes": "CYX"}) - meta = inspect_image(path, "a.ome.tif") - assert meta.axes == "CYX" - assert check_declared_axes(meta, "CYX") is None - assert check_declared_axes(meta, "YXC").code == "declared_axes_mismatch" - - -def test_shaped_axes_declared_in_another_order_are_refused(tmp_path): - # Decision D-d: tifffile "shaped" axes are header axes, so they bind like OME's. - meta = inspect_image(write_data(tmp_path / "a.tif", np.zeros((3, 8, 8), np.uint8), "CYX")) - assert check_declared_axes(meta, "YXC").code == "declared_axes_mismatch" - - -@pytest.mark.parametrize( - "write", - [ - lambda p: plain(p, np.zeros((3, 8, 8), np.uint8)), - lambda p: write_data(p, np.zeros((3, 8, 8), np.uint8)), - lambda p: write_data(p, np.zeros((3, 8, 8), np.uint8), "CYX"), - lambda p: write_data(p, np.zeros((3, 8, 8), np.uint8), "QYX"), - lambda p: imagej(p, np.zeros((3, 8, 8), np.uint8), "ZYX"), - lambda p: plain(p, np.zeros((8, 8), np.uint8)), - lambda p: write_data(p, np.zeros((8, 8), np.uint8)), - ], -) -def test_header_axes_are_exactly_what_check_declared_axes_binds(tmp_path, write): - path = tmp_path / "a.tif" - write(path) - meta = inspect_image(path, "a.tif") - roles = "XYZCT" - accepted = { - "".join(order) - for order in itertools.permutations(roles, len(meta.axes)) - if check_declared_axes(meta, "".join(order)) is None - } - named = header_axes(meta) - if named is None: - # Something is left open, so more than one declaration fits. - assert len(accepted) > 1 - else: - # Every axis is named, so only the header's own axes fit, or them with Y and X - # swapped (read transposed, and reported as such). - swapped = named.translate(str.maketrans("XY", "YX")) - assert accepted == {named, swapped} - assert [declared_axes_transposed(meta, d) for d in (named, swapped)] == [False, True] - - -def ome(path, data, axes): - tifffile.imwrite(path, data, ome=True, metadata={"axes": axes}) - return path - - -@pytest.mark.parametrize( - ("write", "header", "declared"), - [ - (lambda p: imagej(p, np.zeros((3, 8, 6), np.uint8), "CYX"), "CYX", "CXY"), - (lambda p: imagej(p, np.zeros((3, 8, 6), np.uint8), "ZYX"), "ZYX", "ZXY"), - (lambda p: ome(p, np.zeros((3, 8, 6), np.uint8), "CYX"), "CYX", "cxy"), - (lambda p: write_data(p, np.zeros((3, 8, 6), np.uint8), "CYX"), "CYX", "CXY"), - (lambda p: plain(p, np.zeros((8, 6), np.uint8)), "YX", "XY"), - ], -) -def test_named_y_and_x_declared_swapped_are_read_transposed(tmp_path, write, header, declared): - # Owner decision (31 October 2026): only a swap of the named Y and X is accepted, and - # the caller is told so it can record that the image is read transposed. - path = tmp_path / "a.tif" - write(path) - meta = inspect_image(path, "a.tif") - assert meta.axes == header - assert check_declared_axes(meta, declared) is None - assert declared_axes_transposed(meta, declared) is True - assert declared_axes_transposed(meta, header) is False - - -@pytest.mark.parametrize( - ("header", "declared"), - [ - ("CYX", "YXC"), # the named C moved: not a swap of Y and X - ("ZYX", "CYX"), - ("ZYX", "CXY"), # a swap together with another change - ("ZYX", "XZY"), - ], -) -def test_any_other_disagreement_stays_refused(tmp_path, header, declared): - meta = inspect_image(imagej(tmp_path / "a.tif", np.zeros((3, 8, 6), np.uint8), header)) - assert check_declared_axes(meta, declared).code == "declared_axes_mismatch" - with pytest.raises(ValueError, match="do not fit"): - declared_axes_transposed(meta, declared) - - -def test_a_plain_stack_is_never_reported_transposed(tmp_path): - # Its Y and X are open, not named: there is nothing to swap. - meta = inspect_image(plain(tmp_path / "a.tif", np.zeros((3, 8, 6), np.uint8)), "a.tif") - assert declared_axes_transposed(meta, "CXY") is False - assert declared_axes_transposed(meta, "YXC") is False - - -def test_rgb_samples_may_be_declared_channels(tmp_path): - path = tmp_path / "rgb.tif" - tifffile.imwrite(path, np.zeros((8, 8, 3), np.uint8), photometric="rgb") - meta = inspect_image(path, "rgb.tif") - assert check_declared_axes(meta, "YXC") is None - assert check_declared_axes(meta, "YXZ").code == "declared_axes_mismatch" - - -@pytest.mark.parametrize("declared", ["YX", "ZCYX", "ZZX", "ZYQ", "", None]) -def test_malformed_declarations_are_refused(tmp_path, declared): - meta = inspect_image(imagej(tmp_path / "a.tif", np.zeros((3, 8, 8), np.uint8), "ZYX"), "a.tif") - assert check_declared_axes(meta, declared).code == "declared_axes_mismatch" - - -def test_an_unreadable_image_cannot_be_checked(tmp_path): - path = tmp_path / "a.tif" - path.write_bytes(b"not a tiff") - assert check_declared_axes(inspect_image(path, "a.tif"), "YX").code == "unreadable_tiff" - - # read_mask_file -------------------------------------------------------------------------------- @@ -511,8 +284,6 @@ def test_header_axes_of_an_image_one_pixel_high_or_wide(tmp_path, shape): meta = inspect_image(writer(tmp_path / "a.tif", labels(shape)), "a.tif") assert (meta.axes, meta.shape) == ("YX", shape) assert named_axes(meta) == header_axes(meta) == "YX" - assert check_declared_axes(meta, "YX") is None - assert check_declared_axes(meta, "ZX").code == "declared_axes_mismatch" def test_read_mask_file_reads_a_1d_array_saved_as_one_row(tmp_path): diff --git a/tests/test_manifest.py b/tests/test_manifest.py index 7decf4b9..712d6b5c 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -256,7 +256,6 @@ def test_bad_role_in_manifest_is_refused(): ("where", "key"), [ ("settings", "missing_as_singleton"), - ("settings", "convert_tiffs"), ("file", "read_in_blocks"), ("file", "channel_names_trimmed"), ("top", "channels_from_rgb_samples"), @@ -283,7 +282,6 @@ def _with_flag(key, value): "key", [ "missing_as_singleton", - "convert_tiffs", "read_in_blocks", "channel_names_trimmed", "channels_from_rgb_samples", diff --git a/tests/test_moved.py b/tests/test_moved.py deleted file mode 100644 index 5936ad8d..00000000 --- a/tests/test_moved.py +++ /dev/null @@ -1,148 +0,0 @@ -"""The code moved to morphobricks_normalization stays importable here, with a warning.""" - -import importlib -import re -import warnings -from pathlib import Path - -import numpy as np -import pytest - -import morphobricks_preprocessing as mbp -from morphobricks_preprocessing import _moved -from morphobricks_preprocessing.cli import main -from tiff_tree import write_data - -PACKAGE = Path(mbp.__file__).parent -MOVED_MODULES = sorted(_moved.MOVED_NAMES) -SUBMODULE_NAMES = [(m, n) for m in MOVED_MODULES for n in _moved.MOVED_NAMES[m]] - - -@pytest.fixture(autouse=True) -def fresh_warnings(monkeypatch): - # Other tests (and their helpers) already read some old paths in this session. - monkeypatch.setattr(_moved, "_warned", set()) - - -def new_object(module, name): - return getattr(importlib.import_module(f"morphobricks_normalization.{module}"), name) - - -@pytest.mark.parametrize("module", MOVED_MODULES) -def test_moved_names_still_exist_in_the_normalization_brick(module): - # Every name that moved (normalization contract §1.1) is forwarded to the brick's - # current object. Names the brick added later (Milestone 2b: channel_column, - # WeightedHistogram, ...) were never here, so they are not forwarded. - new = importlib.import_module(f"morphobricks_normalization.{module}") - defined = { - name - for name, value in vars(new).items() - if not name.startswith("_") - and not isinstance(value, type(np)) - and getattr(value, "__module__", new.__name__) == new.__name__ - } - assert set(_moved.MOVED_NAMES[module]) <= defined - - -def test_top_level_names_are_the_ones_the_package_exported(): - for name, module in _moved.TOP_LEVEL_NAMES.items(): - assert name in _moved.MOVED_NAMES[module] - assert name in mbp.__all__ - assert name in dir(mbp) - - -@pytest.mark.parametrize(("name", "module"), sorted(_moved.TOP_LEVEL_NAMES.items())) -def test_top_level_name_warns_once_and_is_the_new_object(name, module): - with pytest.warns(DeprecationWarning) as record: - value = getattr(mbp, name) - (warning,) = record - message = str(warning.message) - assert f"morphobricks_preprocessing.{name} has moved to" in message - assert f"morphobricks_normalization.{module}.{name}" in message - assert warning.filename == __file__ # points at the caller - assert value is new_object(module, name) - with warnings.catch_warnings(): - warnings.simplefilter("error") - assert getattr(mbp, name) is value - - -@pytest.mark.parametrize(("module", "name"), SUBMODULE_NAMES) -def test_old_submodule_name_warns_once_and_is_the_new_object(module, name): - with warnings.catch_warnings(): - warnings.simplefilter("error") - old = importlib.import_module(f"morphobricks_preprocessing.{module}") - with pytest.warns(DeprecationWarning) as record: - value = getattr(old, name) - (warning,) = record - message = str(warning.message) - assert f"morphobricks_preprocessing.{module}.{name} has moved to" in message - assert f"morphobricks_normalization.{module}.{name}" in message - assert warning.filename == __file__ - assert value is new_object(module, name) - with warnings.catch_warnings(): - warnings.simplefilter("error") - assert getattr(old, name) is value - assert name in dir(old) - - -def test_from_imports_keep_working(): - with pytest.warns(DeprecationWarning, match="csv_io.SCHEMA_VERSION"): - from morphobricks_preprocessing.csv_io import SCHEMA_VERSION - with pytest.warns(DeprecationWarning, match="morphobricks_preprocessing.PlaneRecord"): - from morphobricks_preprocessing import PlaneRecord - assert SCHEMA_VERSION == "4" # the brick's current value (Milestone 5) - from morphobricks_normalization import PlaneRecord as new_record - - assert PlaneRecord is new_record - - -def test_each_old_path_warns_separately(): - with pytest.warns(DeprecationWarning, match=r"preprocessing\.PooledHistogram"): - _ = mbp.PooledHistogram - with pytest.warns(DeprecationWarning, match=r"preprocessing\.pooling\.PooledHistogram"): - _ = mbp.pooling.PooledHistogram - - -def test_exceptions_are_caught_through_the_old_names(): - with pytest.warns(DeprecationWarning): - old_error = mbp.UnsupportedDtypeError - from morphobricks_normalization.dtypes import check_supported - - with pytest.raises(old_error): - check_supported(np.dtype(bool)) - - -@pytest.mark.parametrize("module", ["morphobricks_preprocessing", *MOVED_MODULES]) -def test_unknown_names_are_still_missing(module): - target = mbp if module == "morphobricks_preprocessing" else getattr(mbp, module) - with pytest.raises(AttributeError, match="no_such_name"): - _ = target.no_such_name - - -def test_package_code_imports_from_the_normalization_brick(): - old_import = re.compile( - r"^\s*from \.(dtypes|histogram|records|csv_io|pooling|normalization) import" - r"|^\s*from morphobricks_preprocessing\.(dtypes|histogram|records|csv_io|pooling|" - r"normalization) import", - re.MULTILINE, - ) - for path in PACKAGE.glob("*.py"): - assert not old_import.search(path.read_text(encoding="utf-8")), path.name - - -def test_the_package_itself_never_warns(tmp_path, capsys): - root = tmp_path / "exp" - rng = np.random.default_rng(0) - for rel in ["ctrl/rep1/a.tif", "drug/rep1/b.tif"]: - write_data(root / rel, rng.integers(0, 200, (2, 8, 8), dtype=np.uint8), axes="ZYX") - assert main(["check", str(root)]) == 0 - # The first run writes the experiment design for the user to confirm. - assert main(["run", str(root), "--axes", "ZYX"]) == 1 - path = mbp.default_output_folder(root) / mbp.DESIGN_FILENAME - design = mbp.read_experiment_design(path) - for group in design.groups: - design = design.with_confirmed(group, True, "2026-10-26") - mbp.write_experiment_design(design, path) - assert main(["run", str(root), "--axes", "ZYX"]) == 0 - capsys.readouterr() - assert _moved._warned == set() diff --git a/tests/test_notebook_ui.py b/tests/test_notebook_ui.py index 62762972..b7ee4988 100644 --- a/tests/test_notebook_ui.py +++ b/tests/test_notebook_ui.py @@ -10,13 +10,9 @@ pytest.importorskip("ipywidgets") pytest.importorskip("matplotlib") -from morphobricks_preprocessing import ( # noqa: E402 - Plane, - notebook_ui, - read_experiment_design, - read_manifest, - read_plane_histograms, -) +from morphobricks_normalization.csv_io import read_plane_histograms + +from morphobricks_preprocessing import Plane, notebook_ui, read_experiment_design, read_manifest from morphobricks_preprocessing import notebook_style as ui # noqa: E402 from morphobricks_preprocessing.axes import AxisMapping # noqa: E402 from morphobricks_preprocessing.manifest import file_sha256 # noqa: E402 @@ -32,7 +28,6 @@ ) from morphobricks_preprocessing.planes import DEFAULT_MEMORY_LIMIT, plane_layout # noqa: E402 from morphobricks_preprocessing.validation import error, warning # noqa: E402 -from morphobricks_preprocessing.zarr_export import ConversionSettings # noqa: E402 from tiff_tree import write_data, write_file # noqa: E402 NOTEBOOK = Path(__file__).parents[1] / "notebooks" / "MorphoBricks_Preprocessing.ipynb" @@ -1238,133 +1233,6 @@ def recomputed(): assert recomputed() == 2 # a new check of the folder -def test_conversion_card_reads_directly_by_default(experiment): - nb = checked(experiment) - assert nb.conversion.value == "read" - assert not ui.is_visible(nb.conversion_settings) and nb.space_warning.value == "" - help_text = nb.conversion_help.value - assert help_text.count("
  • ") >= 4 and "nothing extra is saved" in help_text - assert "OME-Zarr images are always read directly" in help_text - assert "" not in help_text - page = nb.run_step() - root = page.children[1] - assert nb.conversion_card in root.children - assert root.children.index(nb.conversion_card) == root.children.index(nb.settings_card) + 1 - - -def test_conversion_card_survives_the_recommended_settings(experiment): - nb = checked(experiment) - if nb.conversion.disabled: - pytest.skip("zarr is not installed") - nb.conversion.value = "convert" - nb.recommended.value = False - nb.recommended.value = True - assert nb.conversion.value == "convert" # a choice about the data, not a tuning value - - -def test_conversion_shows_settings_and_the_disk_space_warning(experiment): - nb = checked(experiment) - if nb.conversion.disabled: - pytest.skip("zarr is not installed") - nb.conversion.value = "convert" - assert ui.is_visible(nb.conversion_settings) - warning = nb.space_warning.value - assert "Converting duplicates the space your images take" in warning - assert "All 3 TIFF images will be stored again in the converted format" in warning - assert "all the space they occupy is duplicated" in warning - assert "Free space on the drive of the output folder" in warning - nb.compression.value = "none" - assert not ui.is_visible(nb.compression_level_box) - nb.pyramid.value = "fixed" - assert ui.is_visible(nb.pyramid_count_box) - nb.pyramid_count.value = 3 - nb.chunk_size.value = 256 - assert nb._conversion_settings() == ConversionSettings(256, "none", 5, 3) - assert "chunks of 256 x 256 px, no compression, 3 smaller level(s)" in nb.plan.value - nb.conversion.value = "read" - assert not ui.is_visible(nb.conversion_settings) and nb.space_warning.value == "" - - -def test_conversion_cards_can_be_collapsed_without_losing_choices(experiment): - nb = checked(experiment) - if nb.conversion.disabled: - pytest.skip("zarr is not installed") - nb.conversion.value = "convert" - assert nb.conversion_settings.selected_index == 0 - assert nb.conversion_settings.get_title(0) == "OME-Zarr conversion settings" - cards = nb.conversion_grid.children - assert len(cards) == 3 - assert "Storage block size" in cards[0].children[0].value - assert nb.chunk_size in cards[0].children - assert nb.compression in cards[1].children - assert nb.pyramid in cards[2].children - nb.conversion_settings.selected_index = None - nb.chunk_size.value = 512 - nb.compression.value = "zstd" - nb.compression_level.value = 7 - nb.pyramid.value = "none" - nb.recommended.value = False - nb.recommended.value = True - assert nb.conversion_settings.selected_index is None - assert "Free space on the drive" in nb.space_warning.value - settings = nb._conversion_settings() - assert settings == ConversionSettings(512, "zstd", 7, 0) - nb.conversion.value = "read" - nb.conversion.value = "convert" - assert nb.conversion_settings.selected_index == 0 - assert nb._conversion_settings() == settings - - -def test_disk_space_warning_turns_red_when_space_is_short(experiment, monkeypatch): - from morphobricks_preprocessing import notebook_ui - - nb = checked(experiment) - if nb.conversion.disabled: - pytest.skip("zarr is not installed") - monkeypatch.setattr(notebook_ui, "free_space", lambda path: 10) - nb = confirmed(nb.folder.value) - nb.conversion.value = "convert" - assert "may not fit on the drive" in nb.space_warning.value - assert "choose an output folder on another drive" in nb.space_warning.value - assert not nb.run_button.disabled # a warning, never a block - - -def test_nothing_to_convert_in_an_ome_zarr_experiment(tmp_path): - pytest.importorskip("zarr") - from zarr_tree import write_ngff - - write_ngff(tmp_path / "exp/a.ome.zarr", np.zeros((2, 8, 8), np.uint8), "zyx") - nb = checked(tmp_path / "exp") - nb.conversion.value = "convert" - assert "There is nothing to convert" in nb.space_warning.value - - -def test_conversion_without_zarr(experiment, monkeypatch): - from morphobricks_preprocessing import notebook_ui - - monkeypatch.setattr(notebook_ui, "zarr_available", lambda: False) - nb = checked(experiment) - assert nb.conversion.disabled and nb.conversion.value == "read" - assert "Converting is not available" in nb.conversion_help.value - assert 'pip install "morphobricks-preprocessing[zarr]"' in nb.conversion_help.value - - -def test_conversion_from_the_notebook(experiment, tmp_path): - pytest.importorskip("zarr") - nb = confirmed(experiment, output_folder=str(tmp_path / "out")) - assert "converted to OME-Zarr" not in nb.plan.value - nb.conversion.value = "convert" - nb.chunk_size.value = 16 - assert "3 TIFF images will be converted to OME-Zarr" in nb.plan.value - result = nb.run() - assert result.converted == ("ctrl/rep1/a.tif", "ctrl/rep1/b.tif", "drug/rep1/a.tif") - assert "TIFF images converted" in nb.run_tiles.value - assert "3 images converted in this run" in nb.run_status.value - assert (tmp_path / "out" / "ome_zarr" / "ctrl" / "rep1" / "a.ome.zarr").is_dir() - settings = read_manifest(result.manifest_path).settings - assert settings.convert_tiffs and settings.conversion.chunk_size == 16 - - def test_output_folder_edits_refresh_the_plan(experiment, tmp_path): nb = confirmed(experiment) nb.output.value = str(tmp_path / "elsewhere") diff --git a/tests/test_physical_sizes.py b/tests/test_physical_sizes.py index 5d64ffa4..638126f0 100644 --- a/tests/test_physical_sizes.py +++ b/tests/test_physical_sizes.py @@ -1,11 +1,10 @@ -import pickle from dataclasses import replace import numpy as np import pytest import tifffile -from morphobricks_preprocessing import PhysicalSizes, read_physical_sizes, zarr_export +from morphobricks_preprocessing import PhysicalSizes, read_physical_sizes from morphobricks_preprocessing import physical_sizes as physical_sizes_module # PhysicalSizes(x=0.1, y=0.2, z=0.5, space_unit="micrometer") pickled before per-axis units. @@ -126,15 +125,6 @@ def test_replace_keeps_units_consistent(): assert replace(sizes, space_unit=None).unit("X") is None -def test_sizes_pickled_before_per_axis_units_keep_their_unit(): - # PhysicalSizes(x=0.1, y=0.2, z=0.5, space_unit="micrometer") pickled by the version - # before per-axis units (as morphobricks_preprocessing.zarr_export.PhysicalSizes). - sizes = pickle.loads(OLD_PICKLE) - assert [sizes.unit(role) for role in "XYZ"] == ["micrometer"] * 3 - assert sizes == PhysicalSizes(x=0.1, y=0.2, z=0.5, space_unit="micrometer") - assert pickle.loads(pickle.dumps(sizes)) == sizes - - def test_equal_sizes_hash_alike(): given = PhysicalSizes(x=0.1, space_unit="micrometer") spelled_out = PhysicalSizes( @@ -151,11 +141,6 @@ def test_units_of_unknown_sizes_and_channels_are_none(): assert sizes.scale("Y") == 1.0 -def test_still_importable_from_zarr_export(): - assert zarr_export.PhysicalSizes is physical_sizes_module.PhysicalSizes is PhysicalSizes - assert zarr_export.read_physical_sizes is read_physical_sizes - - # Unit spellings ------------------------------------------------------------------------ diff --git a/tests/test_planes.py b/tests/test_planes.py index 1a553a10..22dbb9b0 100644 --- a/tests/test_planes.py +++ b/tests/test_planes.py @@ -4,6 +4,7 @@ import numpy as np import pytest import tifffile +from morphobricks_normalization.histogram import compute_plane_histogram, estimate_histogram_memory from morphobricks_preprocessing import ( DEFAULT_MEMORY_LIMIT, @@ -12,7 +13,6 @@ PlaneReadError, Severity, check_planes, - compute_plane_histogram, inspect_images, inspect_tiff, iter_planes, @@ -24,7 +24,6 @@ review_axes, ) from morphobricks_preprocessing import planes as planes_module -from morphobricks_preprocessing.histogram import estimate_histogram_memory from tiff_tree import write_data, write_file DTYPES = ["int8", "int16", "int32", "int64", "uint8", "uint16", "uint32", "uint64"] diff --git a/tests/test_removed_in_v040.py b/tests/test_removed_in_v040.py new file mode 100644 index 00000000..dddfbebd --- /dev/null +++ b/tests/test_removed_in_v040.py @@ -0,0 +1,96 @@ +"""What v0.4.0 removes (spec revision 4, §8 PP-2), and what still reads. + +Removed: ``convert_tiffs`` with ``zarr_export`` (OME-Zarr copies in the output folder), the +deprecated stand-ins of the code moved to the normalization brick, and +``check_declared_axes``/``declared_axes_transposed``, which read a Y/X swap transposed. +""" + +from __future__ import annotations + +import importlib +import json + +import numpy as np +import pytest + +import morphobricks_preprocessing as pp +from morphobricks_preprocessing import prefill_mapping, read_manifest, run_experiment +from morphobricks_preprocessing._io import free_space +from morphobricks_preprocessing.cli import main +from tiff_tree import write_data + +REMOVED_NAMES = [ + "check_declared_axes", + "declared_axes_transposed", + "ConversionSettings", + "ConversionEstimate", + "OmeZarrWriter", + "estimate_conversion", + "ome_zarr_relative_path", + # The deprecated stand-ins of morphobricks_normalization's names. + "PlaneRecord", + "compute_plane_histogram", + "PooledHistogram", + "read_plane_histograms", + "min_max", + "SINGLE", +] + + +@pytest.mark.parametrize("name", REMOVED_NAMES) +def test_removed_names_are_gone(name): + assert not hasattr(pp, name) + assert name not in pp.__all__ + + +@pytest.mark.parametrize( + "module", + [ + "zarr_export", + "_moved", + "csv_io", + "dtypes", + "histogram", + "normalization", + "pooling", + "records", + ], +) +def test_removed_modules_are_gone(module): + with pytest.raises(ModuleNotFoundError): + importlib.import_module(f"morphobricks_preprocessing.{module}") + + +def test_the_run_takes_no_conversion(tmp_path): + write_data(tmp_path / "exp/a.tif", np.zeros((2, 6, 8), np.uint8), "ZYX") + with pytest.raises(TypeError, match="convert_tiffs"): + run_experiment(tmp_path / "exp", prefill_mapping("ZYX"), convert_tiffs=True) + with pytest.raises(TypeError, match="conversion"): + run_experiment(tmp_path / "exp", prefill_mapping("ZYX"), conversion=None) + + +def test_the_cli_takes_no_conversion(tmp_path): + with pytest.raises(SystemExit) as stopped: + main(["run", str(tmp_path), "--axes", "ZYX", "--convert-tiff"]) + assert stopped.value.code == 2 + + +def test_a_v03_manifest_with_conversion_keys_is_still_read(tmp_path): + root = tmp_path / "exp" + write_data(root / "a.tif", np.zeros((2, 6, 8), np.uint8), "ZYX") + result = run_experiment(root, prefill_mapping("ZYX")) + data = json.loads(result.manifest_path.read_text(encoding="utf-8")) + data["settings"]["convert_tiffs"] = True + data["settings"]["conversion"] = {"chunk_size": 1024} + data["files"][0]["ome_zarr_path"] = "ome_zarr/a.ome.zarr" + result.manifest_path.write_text(json.dumps(data), encoding="utf-8") + manifest = read_manifest(result.manifest_path) + assert manifest.file("a.tif") is not None + assert "convert_tiffs" not in json.dumps(manifest.to_dict()) + # The rows of the earlier run are kept: they were the TIFF's own. + again = run_experiment(root, prefill_mapping("ZYX")) + assert again.unchanged == ("a.tif",) and again.read == () + + +def test_free_space_is_kept_for_the_disk_check(tmp_path): + assert free_space(tmp_path / "not" / "yet") > 0 diff --git a/tests/test_report.py b/tests/test_report.py index 9b2072d2..f14112f3 100644 --- a/tests/test_report.py +++ b/tests/test_report.py @@ -1,14 +1,14 @@ import numpy as np import pytest +from morphobricks_normalization.csv_io import read_plane_histograms +from morphobricks_normalization.pooling import PooledHistogram +from morphobricks_normalization.records import group_records from morphobricks_preprocessing import ( - PooledHistogram, discover_experiment, - group_records, group_summaries, prefill_mapping, preflight, - read_plane_histograms, run_experiment, ) from morphobricks_preprocessing.report import ( diff --git a/tests/test_run.py b/tests/test_run.py index 9d727e72..b52f9ac3 100644 --- a/tests/test_run.py +++ b/tests/test_run.py @@ -7,22 +7,23 @@ import numpy as np import pytest import tifffile -from morphobricks_normalization.csv_io import write_plane_histograms +from morphobricks_normalization.csv_io import ( + OUTPUT_FILENAME, + read_plane_histograms, + write_plane_histograms, +) +from morphobricks_normalization.pooling import PooledHistogram +from morphobricks_normalization.records import SINGLE, group_records from morphobricks_preprocessing import ( MANIFEST_FILENAME, - OUTPUT_FILENAME, - SINGLE, AxisMapping, PlaneReadError, - PooledHistogram, Severity, - group_records, plane_layout, prefill_mapping, preflight, read_manifest, - read_plane_histograms, run_experiment, ) from morphobricks_preprocessing import planes as planes_module @@ -941,11 +942,10 @@ def test_sample_planes_report_problems(tmp_path): assert "limit" in too_big[0].error and too_big[0].plane is None -@pytest.mark.parametrize("flag", ["missing_as_singleton", "convert_tiffs"]) +@pytest.mark.parametrize("flag", ["missing_as_singleton"]) @pytest.mark.parametrize("value", [1, "false", np.bool_(True)]) def test_run_refuses_flags_that_are_not_booleans_before_reading(experiment, tmp_path, flag, value): - # Review of #44: the manifest writer refuses such flags; the run refuses them up - # front instead of after every image has been read. + # Review of #44: the run refuses such flags before reading, as the manifest writer does. output = tmp_path / "out" with pytest.raises(ValueError, match=f"{flag} must be True or False"): run_experiment(experiment, ZYX, output_folder=output, **{flag: value}) diff --git a/tests/test_zarr_export.py b/tests/test_zarr_export.py deleted file mode 100644 index c9e44952..00000000 --- a/tests/test_zarr_export.py +++ /dev/null @@ -1,976 +0,0 @@ -import json -import os -import shutil -from pathlib import Path - -import numpy as np -import pytest -import tifffile - -from morphobricks_preprocessing import ( - AxisMapping, - PhysicalSizes, - PlaneReadError, - check_metadata, - default_output_folder, - inspect_ome_zarr, - ome_zarr_relative_path, - plane_layout, - prefill_mapping, - read_experiment_design, - read_manifest, - read_physical_sizes, - read_plane_histograms, - run_experiment, - write_experiment_design, -) -from morphobricks_preprocessing import planes as planes_module -from morphobricks_preprocessing import run as run_module -from morphobricks_preprocessing.cli import main -from morphobricks_preprocessing.zarr_export import ( - PYRAMID_TARGET, - ConversionSettings, - downsample, - estimate_conversion, - free_space, - is_copy_path, - pyramid_shapes, -) -from tiff_tree import write_data, write_file - -ZYX = prefill_mapping("ZYX") - -needs_zarr = pytest.mark.skipif( - not run_module.zarr_available(), reason="the zarr extra is not installed" -) - - -def random_data(shape, dtype="uint16", seed=0): - dt = np.dtype(dtype) - rng = np.random.default_rng(seed) - if dt.kind == "f": - return rng.normal(100, 30, shape).astype(dt) - info = np.iinfo(dt) - return rng.integers(info.min, info.max, shape, dtype=dt, endpoint=True) - - -def open_copy(output, rel): - import zarr - - return zarr.open_group(str(output / ome_zarr_relative_path(rel)), mode="r") - - -def multiscale(group): - return dict(group.attrs)["multiscales"][0] - - -# Pure helpers (no zarr needed) -------------------------------------------------------- - - -@pytest.mark.parametrize( - ("source", "copy"), - [ - ("ctrl/rep1/img_01.tif", "ome_zarr/ctrl/rep1/img_01.ome.zarr"), - ("a.TIFF", "ome_zarr/a.ome.zarr"), - ("x/b.ome.tif", "ome_zarr/x/b.ome.zarr"), - ("x/c.OME.TIFF", "ome_zarr/x/c.ome.zarr"), - ], -) -def test_copy_location_mirrors_the_experiment(source, copy): - assert ome_zarr_relative_path(source) == copy - - -def test_pyramid_shapes(): - assert pyramid_shapes((100, 80)) == [(100, 80)] - assert pyramid_shapes((1100, 700)) == [(1100, 700), (550, 350), (275, 175)] - assert max(pyramid_shapes((3000, 10))[-1]) <= PYRAMID_TARGET - assert len(pyramid_shapes((10**7, 10**7))) == 8 - - -def test_pyramid_shapes_with_a_number_of_levels(): - assert pyramid_shapes((1100, 700), 0) == [(1100, 700)] - assert pyramid_shapes((100, 80), 2) == [(100, 80), (50, 40), (25, 20)] - assert pyramid_shapes((2, 2), 5) == [(2, 2), (1, 1)] # stops at a single pixel - - -# Conversion settings -------------------------------------------------------------------- - - -def test_conversion_settings_defaults_and_round_trip(): - default = ConversionSettings() - assert (default.chunk_size, default.compression, default.compression_level) == (1024, "lz4", 5) - assert default.pyramid_levels is None - custom = ConversionSettings(256, "zstd", 9, 3) - assert ConversionSettings.from_dict(custom.to_dict()) == custom - assert ConversionSettings.from_dict(None) == default - assert ConversionSettings.from_dict({"compression": "none"}).compression == "none" - - -@pytest.mark.parametrize( - "kwargs", - [ - {"chunk_size": 8}, - {"chunk_size": 9000}, - {"compression": "gzip"}, - {"compression_level": 0}, - {"compression_level": 10}, - {"pyramid_levels": -1}, - {"pyramid_levels": 8}, - ], -) -def test_conversion_settings_reject_bad_values(kwargs): - with pytest.raises(ValueError): - ConversionSettings(**kwargs) - - -def test_conversion_settings_describe(): - assert ConversionSettings().describe() == ( - "chunks of 1024 x 1024 px, Blosc LZ4 level 5, smaller levels until a plane fits 512 px" - ) - assert ConversionSettings(256, "none", 5, 0).describe() == ( - "chunks of 256 x 256 px, no compression, no smaller levels" - ) - - -def test_estimate_conversion(tmp_path): - from morphobricks_preprocessing import TiffMetadata - - path = write_data(tmp_path / "a.tif", random_data((2, 1024, 600), "uint16"), axes="ZYX") - files = [ - TiffMetadata("a.tif", True, (2, 1024, 600), np.dtype("uint16"), "ZYX", 1, 2), - TiffMetadata( - "b.ome.zarr", True, (2, 8, 8), np.dtype("uint16"), "ZYX", 1, 0, kind="ome-zarr" - ), - TiffMetadata("bad.tif", False), - ] - level0 = 2 * 1024 * 600 * 2 - flat = estimate_conversion(tmp_path, files, ConversionSettings(pyramid_levels=0)) - assert (flat.image_count, flat.copy_bytes) == (1, level0) - assert flat.source_bytes == path.stat().st_size - pyramid = estimate_conversion(tmp_path, files) # 1024 x 600 -> 512 x 300 - assert pyramid.copy_bytes == level0 + 2 * 512 * 300 * 2 - - -def test_free_space_uses_the_nearest_existing_folder(tmp_path, monkeypatch): - # Compare the folder asked about, not two readings of the free space: other processes - # can write to the drive in between (seen on CI runners). - real = shutil.disk_usage - asked = [] - - def spy(path): - asked.append(Path(path)) - return real(path) - - monkeypatch.setattr(shutil, "disk_usage", spy) - assert free_space(tmp_path / "not" / "yet" / "there") > 0 - assert asked == [tmp_path.resolve()] - - -def test_downsample_integers_round_and_keep_type(): - pixels = np.array([[0, 1, 255], [1, 1, 255], [7, 7, 9]], dtype=np.uint8) - small = downsample(pixels) - assert small.dtype == np.uint8 - # Blocks: [[0,1],[1,1]] -> 0.75 -> 1; [[255],[255]] padded -> 255; [7,7] row padded -> 7. - np.testing.assert_array_equal(small, [[1, 255], [7, 9]]) - - -def test_downsample_floats_propagate_nan(): - pixels = np.array([[1.0, 3.0], [np.nan, 0.0]], dtype=np.float32) - assert np.isnan(downsample(pixels)[0, 0]) - np.testing.assert_allclose(downsample(np.full((4, 4), 2.5)), np.full((2, 2), 2.5)) - - -def test_physical_sizes_from_ome(tmp_path): - path = tmp_path / "a.ome.tif" - tifffile.imwrite( - path, - np.zeros((2, 3, 8, 8), np.uint16), - ome=True, - photometric="minisblack", - metadata={ - "axes": "TZYX", - "PhysicalSizeX": 120.0, - "PhysicalSizeXUnit": "nm", - "PhysicalSizeY": 120.0, - "PhysicalSizeYUnit": "nm", - "PhysicalSizeZ": 500.0, - "TimeIncrement": 30.0, - "TimeIncrementUnit": "s", - }, - ) - # PhysicalSizeZUnit is left out, so Z is in the OME default, micrometres. - assert read_physical_sizes(path) == PhysicalSizes( - x=120.0, - y=120.0, - z=500.0, - t=30.0, - space_unit="nanometer", - time_unit="second", - x_unit="nanometer", - y_unit="nanometer", - z_unit="micrometer", - ) - - -def test_physical_sizes_from_imagej(tmp_path): - path = tmp_path / "a.tif" - tifffile.imwrite( - path, - np.zeros((3, 8, 8), np.uint16), - imagej=True, - resolution=(1 / 0.25, 1 / 0.25), - metadata={"axes": "ZYX", "unit": "um", "spacing": 1.5}, - ) - sizes = read_physical_sizes(path) - assert (sizes.x, sizes.y, sizes.z, sizes.space_unit) == (0.25, 0.25, 1.5, "micrometer") - - -def test_physical_sizes_unknown(tmp_path): - plain = write_data(tmp_path / "a.tif", np.zeros((8, 8), np.uint8)) - assert read_physical_sizes(plain) == PhysicalSizes() - no_unit = tmp_path / "b.tif" - tifffile.imwrite(no_unit, np.zeros((8, 8), np.uint8), imagej=True, resolution=(2.0, 2.0)) - assert read_physical_sizes(no_unit).x is None - assert read_physical_sizes(write_file(tmp_path / "bad.tif", b"garbage")) == PhysicalSizes() - assert PhysicalSizes().scale("X") == 1.0 and PhysicalSizes().unit("X") is None - - -def test_only_copies_inside_the_ome_zarr_folder_count(tmp_path): - assert is_copy_path(tmp_path, "ome_zarr/a/x.ome.zarr") - assert not is_copy_path(tmp_path, "../keep.ome.zarr") - assert not is_copy_path(tmp_path, "ome_zarr/../../x.ome.zarr") - assert not is_copy_path(tmp_path, "ome_zarr/a/x.zarr") - assert not is_copy_path(tmp_path, "plane_histograms.csv") - - -def test_missing_zarr_is_a_clear_error(tmp_path, monkeypatch): - write_data(tmp_path / "exp/a.tif", random_data((2, 8, 8)), axes="ZYX") - monkeypatch.setattr(run_module, "zarr_available", lambda: False) - result = run_experiment(tmp_path / "exp", ZYX, convert_tiffs=True) - (found,) = [i for i in result.issues if i.code == "zarr_not_installed"] - assert "[zarr]" in found.action and not result.written - assert run_experiment(tmp_path / "exp", ZYX).written - - -# Writing copies ----------------------------------------------------------------------- - - -LAYOUTS = [ - pytest.param((6, 7), "YX", {}, None, ("Y", "X"), id="2d"), - pytest.param((3, 6, 7), "ZYX", {}, None, ("Z", "Y", "X"), id="zstack"), - pytest.param((3, 2, 6, 7), "ZCYX", {}, None, ("C", "Z", "Y", "X"), id="zcyx"), - pytest.param( - (2, 3, 2, 6, 7), "TZCYX", {"imagej": True}, None, ("T", "C", "Z", "Y", "X"), id="imagej" - ), - pytest.param( - (2, 6, 7, 3), "ZYXS", {"photometric": "rgb"}, None, ("C", "Z", "Y", "X"), id="rgb" - ), - pytest.param((3, 6, 7), "ZXY", {}, None, ("Z", "Y", "X"), id="x-before-y"), - pytest.param((1, 3, 6, 7), "QZYX", {}, None, ("Z", "Y", "X"), id="unassigned-singleton"), - pytest.param((4, 6, 7), None, {}, "TYX", ("T", "Y", "X"), id="pages-as-time"), -] - - -@needs_zarr -@pytest.mark.parametrize(("shape", "axes", "kwargs", "roles", "ngff"), LAYOUTS) -def test_copy_holds_the_source_values(tmp_path, shape, axes, kwargs, roles, ngff): - data = random_data(shape, "uint8" if "photometric" in kwargs else "uint16") - write_data(tmp_path / "exp/a.tif", data, axes=axes, **kwargs) - info = run_module.preflight(tmp_path / "exp").metadata.files[0] - mapping = AxisMapping(info.axes, roles) if roles else prefill_mapping(info.axes) - result = run_experiment(tmp_path / "exp", mapping, convert_tiffs=True) - assert result.written and result.converted == ("a.tif",) - - group = open_copy(result.output_folder, "a.tif") - meta = multiscale(group) - assert [a["name"] for a in meta["axes"]] == [r.lower() for r in ngff] - assert meta["version"] == "0.4" - # Reorder the source to OME-NGFF order with NumPy and compare with level 0. - present = [r for r in mapping.roles if r is not None] - keep = tuple(slice(None) if r is not None else 0 for r in mapping.roles) - expected = np.moveaxis(data[keep], [present.index(r) for r in ngff], range(len(ngff))) - np.testing.assert_array_equal(group["0"][:], expected) - assert group["0"].dtype == data.dtype - - -@needs_zarr -@pytest.mark.parametrize( - "dtype", - [ - "int8", - "int16", - "int32", - "int64", - "uint8", - "uint16", - "uint32", - "uint64", - "float32", - "float64", - ], -) -def test_every_dtype_is_copied_unchanged(tmp_path, dtype): - data = random_data((2, 9, 11), dtype) - if data.dtype.kind == "f": - data[0, 0, :2] = [np.nan, np.inf] - write_data(tmp_path / "exp/a.tif", data, axes="ZYX") - result = run_experiment(tmp_path / "exp", ZYX, convert_tiffs=True) - np.testing.assert_array_equal(open_copy(result.output_folder, "a.tif")["0"][:], data) - - -@needs_zarr -def test_copies_give_each_axis_its_own_unit(tmp_path): - path = tmp_path / "exp/a.ome.tif" - path.parent.mkdir(parents=True) - tifffile.imwrite( - path, - random_data((2, 8, 8), "uint16"), - ome=True, - photometric="minisblack", - metadata={ - "axes": "ZYX", - "PhysicalSizeX": 120.0, - "PhysicalSizeXUnit": "nm", - "PhysicalSizeY": 0.25, - "PhysicalSizeYUnit": "µm", - "PhysicalSizeZ": 2.0, - "PhysicalSizeZUnit": "mm", - }, - ) - result = run_experiment(tmp_path / "exp", ZYX, convert_tiffs=True) - meta = multiscale(open_copy(result.output_folder, "a.ome.tif")) - assert [axis["unit"] for axis in meta["axes"]] == ["millimeter", "micrometer", "nanometer"] - assert meta["datasets"][0]["coordinateTransformations"][0]["scale"] == [2.0, 0.25, 120.0] - - -@needs_zarr -def test_pyramid_levels_and_scales(tmp_path): - data = random_data((2, 1100, 700), "uint16") - path = tmp_path / "exp/a.ome.tif" - path.parent.mkdir(parents=True) - tifffile.imwrite( - path, - data, - ome=True, - photometric="minisblack", - metadata={"axes": "ZYX", "PhysicalSizeX": 0.5, "PhysicalSizeY": 0.5, "PhysicalSizeZ": 2}, - ) - result = run_experiment(tmp_path / "exp", ZYX, convert_tiffs=True) - group = open_copy(result.output_folder, "a.ome.tif") - meta = multiscale(group) - assert [d["path"] for d in meta["datasets"]] == ["0", "1", "2"] - scales = [d["coordinateTransformations"][0]["scale"] for d in meta["datasets"]] - assert scales == [[2.0, 0.5, 0.5], [2.0, 1.0, 1.0], [2.0, 2.0, 2.0]] - assert meta["axes"][2] == {"name": "x", "type": "space", "unit": "micrometer"} - assert group["1"].shape == (2, 550, 350) and group["2"].shape == (2, 275, 175) - np.testing.assert_array_equal(group["1"][1], downsample(data[1])) - assert group["0"].chunks == (1, 1024, 700) - - -@needs_zarr -def test_channel_names_and_display_ranges(tmp_path): - data = random_data((2, 8, 8), "uint16") - path = tmp_path / "exp/a.ome.tif" - path.parent.mkdir(parents=True) - tifffile.imwrite( - path, - data, - ome=True, - photometric="minisblack", - metadata={"axes": "CYX", "Channel": {"Name": ["DAPI", "GFP"]}}, - ) - result = run_experiment(tmp_path / "exp", prefill_mapping("CYX"), convert_tiffs=True) - omero = dict(open_copy(result.output_folder, "a.ome.tif").attrs)["omero"] - assert [c["label"] for c in omero["channels"]] == ["DAPI", "GFP"] - window = omero["channels"][1]["window"] - assert (window["min"], window["max"]) == (float(data[1].min()), float(data[1].max())) - - -@needs_zarr -def test_copies_of_unnamed_channels_read_back_unnamed(tmp_path): - # Issue #37 review: channel names are compared exactly, so a copy must not name a - # channel its source leaves unnamed (it used to write the placeholder "Channel "). - root = tmp_path / "exp" - root.mkdir() - tifffile.imwrite( - root / "a.ome.tif", - random_data((2, 8, 8)), - ome=True, - photometric="minisblack", - metadata={"axes": "CYX", "Channel": {"Name": ["DAPI", "GFP"]}}, - ) - write_data(root / "b.tif", random_data((2, 8, 8), seed=1), axes="CYX") - result = run_experiment(root, prefill_mapping("CYX"), convert_tiffs=True) - assert result.written - omero = dict(open_copy(result.output_folder, "b.tif").attrs)["omero"] - assert all("label" not in channel for channel in omero["channels"]) - copies = result.output_folder / "ome_zarr" - headers = [inspect_ome_zarr(copies / name, name) for name in ("a.ome.zarr", "b.ome.zarr")] - assert [h.channel_names for h in headers] == [("DAPI", "GFP"), ()] - assert "channel_names_differ" not in {i.code for i in check_metadata(headers)} - - -@needs_zarr -def test_copies_with_placeholder_channel_names_are_written_again(tmp_path): - root = tmp_path / "exp" - root.mkdir() - write_data(root / "a.tif", random_data((2, 8, 8)), axes="CYX") - first = run_experiment(root, prefill_mapping("CYX"), convert_tiffs=True) - table = first.csv_path.read_bytes() - # Make the copy look like one written with placeholder names. - attrs_path = first.output_folder / ome_zarr_relative_path("a.tif") / ".zattrs" - attrs = json.loads(attrs_path.read_text(encoding="utf-8")) - for c, channel in enumerate(attrs["omero"]["channels"]): - channel["label"] = f"Channel {c}" - attrs_path.write_text(json.dumps(attrs), encoding="utf-8") - second = run_experiment(root, prefill_mapping("CYX"), convert_tiffs=True) - assert second.converted == ("a.tif",) - assert second.csv_path.read_bytes() == table # same pixels, same rows - omero = dict(open_copy(second.output_folder, "a.tif").attrs)["omero"] - assert all("label" not in channel for channel in omero["channels"]) - assert run_experiment(root, prefill_mapping("CYX"), convert_tiffs=True).converted == () - - -@needs_zarr -def test_missing_axis_becomes_length_one(tmp_path): - write_data(tmp_path / "exp/a.tif", random_data((2, 3, 8, 8)), axes="ZTYX") - data = random_data((2, 8, 8), seed=1) - write_data(tmp_path / "exp/b.tif", data, axes="ZYX") - result = run_experiment( - tmp_path / "exp", prefill_mapping("ZTYX"), convert_tiffs=True, missing_as_singleton=True - ) - group = open_copy(result.output_folder, "b.tif") - assert group["0"].shape == (1, 2, 8, 8) - np.testing.assert_array_equal(group["0"][0], data) - - -# Runs and reruns ---------------------------------------------------------------------- - - -@pytest.fixture -def experiment(tmp_path): - root = tmp_path / "exp" - for i, rel in enumerate(["ctrl/a.tif", "ctrl/b.tif", "drug/a.tif"]): - write_data(root / rel, random_data((2, 16, 16), "uint8", seed=i), axes="ZYX") - return root - - -def copy_mtimes(output): - base = output / "ome_zarr" - return { - p.relative_to(base).as_posix(): p.stat().st_mtime_ns for p in base.rglob("*") if p.is_file() - } - - -@needs_zarr -def test_manifest_records_the_copies(experiment): - result = run_experiment(experiment, ZYX, convert_tiffs=True) - manifest = read_manifest(result.manifest_path) - assert manifest.settings.convert_tiffs - assert [f.ome_zarr_path for f in manifest.files] == [ - "ome_zarr/ctrl/a.ome.zarr", - "ome_zarr/ctrl/b.ome.zarr", - "ome_zarr/drug/a.ome.zarr", - ] - assert not [p for p in result.output_folder.iterdir() if p.name.startswith(".staging")] - - -@needs_zarr -def test_rerun_keeps_unchanged_copies(experiment): - first = run_experiment(experiment, ZYX, convert_tiffs=True) - before = copy_mtimes(first.output_folder) - second = run_experiment(experiment, ZYX, convert_tiffs=True) - assert second.read == second.converted == () - assert copy_mtimes(second.output_folder) == before - - -@needs_zarr -def test_changed_image_is_copied_again(experiment): - first = run_experiment(experiment, ZYX, convert_tiffs=True) - path = experiment / "ctrl/b.tif" - new_data = random_data((2, 16, 16), "uint8", seed=42) - stat = os.stat(path) - write_data(path, new_data, axes="ZYX") - os.utime(path, ns=(stat.st_atime_ns, stat.st_mtime_ns + 10**9)) - second = run_experiment(experiment, ZYX, convert_tiffs=True) - assert second.converted == ("ctrl/b.tif",) - np.testing.assert_array_equal(open_copy(first.output_folder, "ctrl/b.tif")["0"][:], new_data) - - -@needs_zarr -def test_removed_and_excluded_images_lose_their_copies(experiment): - first = run_experiment(experiment, ZYX, convert_tiffs=True) - (experiment / "ctrl/a.tif").unlink() - run_experiment(experiment, ZYX, convert_tiffs=True, exclude=["drug/a.tif"]) - remaining = sorted( - p.relative_to(first.output_folder).as_posix() - for p in (first.output_folder / "ome_zarr").rglob("*.ome.zarr") - ) - assert remaining == ["ome_zarr/ctrl/b.ome.zarr"] - - -@needs_zarr -def test_switching_to_conversion_recomputes_the_tiff_rows_from_copies(experiment): - first = run_experiment(experiment, ZYX) - table = first.csv_path.read_bytes() - second = run_experiment(experiment, ZYX, convert_tiffs=True) - assert second.rebuild_reason is None - assert second.changed == ("ctrl/a.tif", "ctrl/b.tif", "drug/a.tif") == second.converted - # The copies hold the source values, so the rows are byte-identical. - assert second.csv_path.read_bytes() == table - # A copy deleted by hand is converted again, with its rows. - import shutil - - shutil.rmtree(second.output_folder / "ome_zarr/drug/a.ome.zarr") - third = run_experiment(experiment, ZYX, convert_tiffs=True) - assert third.changed == ("drug/a.tif",) == third.converted - - -@needs_zarr -def test_histograms_are_computed_from_the_converted_copies(experiment, monkeypatch): - read_paths = [] - - def spy(name): - original = getattr(run_module, name) - - def reading(path, *args, **kwargs): - read_paths.append(Path(path)) - return original(path, *args, **kwargs) - - monkeypatch.setattr(run_module, name, reading) - - spy("iter_planes") # conversion - spy("iter_xyc_planes") # histograms - run_experiment(experiment, ZYX, convert_tiffs=True) - copies = [p for p in read_paths if p.name.endswith(".ome.zarr")] - sources = [p for p in read_paths if p.suffix == ".tif"] - # Each TIFF is read once to convert it, and its copy once for the histograms. - assert len(sources) == len(copies) == 3 - assert all(".run-" in str(p) for p in copies) - - -@needs_zarr -def test_switching_back_to_reading_tiffs_removes_the_copies(experiment): - run_experiment(experiment, ZYX, convert_tiffs=True) - result = run_experiment(experiment, ZYX) - assert result.changed == ("ctrl/a.tif", "ctrl/b.tif", "drug/a.tif") - assert result.converted == () - assert all(f.ome_zarr_path is None for f in read_manifest(result.manifest_path).files) - assert not list((result.output_folder / "ome_zarr").rglob("*.ome.zarr")) - - -@needs_zarr -def test_failed_run_writes_no_copies(experiment, monkeypatch): - original = planes_module._read_page - - def broken(series, index, layout, path): - if str(path).endswith("b.tif"): - raise PlaneReadError("cannot be decoded") - return original(series, index, layout, path) - - monkeypatch.setattr(planes_module, "_read_page", broken) - result = run_experiment(experiment, ZYX, convert_tiffs=True) - assert not result.written - assert not result.output_folder.exists() - - -@needs_zarr -def test_failed_rerun_keeps_earlier_copies(experiment, monkeypatch): - first = run_experiment(experiment, ZYX, convert_tiffs=True) - before = copy_mtimes(first.output_folder) - write_data(experiment / "ctrl/c.tif", random_data((2, 16, 16), "uint8", seed=7), axes="ZYX") - original = planes_module._read_page - - def broken(series, index, layout, path): - if str(path).endswith("c.tif"): - raise PlaneReadError("cannot be decoded") - return original(series, index, layout, path) - - monkeypatch.setattr(planes_module, "_read_page", broken) - assert not run_experiment(experiment, ZYX, convert_tiffs=True).written - assert copy_mtimes(first.output_folder) == before - assert sorted(p.name for p in first.output_folder.iterdir()) == [ - "ome_zarr", - "plane_histograms.csv", - "run_manifest.json", - ] - - -@needs_zarr -def test_table_is_the_same_with_or_without_copies(experiment, tmp_path): - with_copies = run_experiment(experiment, ZYX, convert_tiffs=True, output_folder=tmp_path / "a") - without = run_experiment(experiment, ZYX, output_folder=tmp_path / "b") - a = read_plane_histograms(with_copies.csv_path) - b = read_plane_histograms(without.csv_path) - assert [r.plane_id for r in a] == [r.plane_id for r in b] - assert all( - np.array_equal(x.channel_histogram(0).counts, y.channel_histogram(0).counts) - for x, y in zip(a, b, strict=True) - ) - - -@needs_zarr -def test_cli_option(experiment, capsys): - # The first run asks for the experiment design; confirm it as a user would. - assert main(["run", str(experiment), "--axes", "ZYX", "--convert-tiff"]) == 1 - path = default_output_folder(experiment) / "experiment_design.csv" - design = read_experiment_design(path) - for group in design.groups: - design = design.with_confirmed(group, True, "2026-10-26") - write_experiment_design(design, path) - assert main(["run", str(experiment), "--axes", "ZYX", "--convert-tiff"]) == 0 - assert "TIFF images converted to OME-Zarr: 3 images" in capsys.readouterr().out - - -def test_old_manifest_without_ome_zarr_fields_loads(experiment): - result = run_experiment(experiment, ZYX) - data = json.loads(result.manifest_path.read_text(encoding="utf-8")) - del data["settings"]["convert_tiffs"] - for entry in data["files"]: - del entry["ome_zarr_path"] - result.manifest_path.write_text(json.dumps(data), encoding="utf-8") - manifest = read_manifest(result.manifest_path) - assert not manifest.settings.convert_tiffs and manifest.files[0].ome_zarr_path is None - assert run_experiment(experiment, ZYX).read == () - - -@pytest.mark.parametrize("code", ["q", "Q", "l", "L", "i", "h", "b", "B", "f", "d"]) -def test_canonical_dtype(code): - from morphobricks_preprocessing.zarr_export import canonical_dtype - - dtype = np.dtype(code) - canonical = canonical_dtype(dtype) - assert canonical == dtype and canonical is np.dtype(dtype.name) - - -@needs_zarr -@pytest.mark.parametrize("code", ["q", "Q"]) -def test_long_long_planes_are_written(tmp_path, code): - # On Linux and macOS, tifffile can return 64-bit integers as numpy "long long", - # a dtype class zarr 3 does not accept although it equals int64. - from dataclasses import replace - - from morphobricks_preprocessing import Plane, plane_layout - from morphobricks_preprocessing.zarr_export import OmeZarrWriter - - data = random_data((2, 6, 7), np.dtype(code).name).astype(np.dtype(code)) - path = write_data(tmp_path / "a.tif", data, axes="ZYX") - layout = replace(plane_layout(path, ZYX), dtype=np.dtype(code)) - writer = OmeZarrWriter.create(tmp_path / "a.ome.zarr", ZYX, layout) - for z in range(2): - writer.write(Plane(z, None, None, np.ascontiguousarray(data[z]))) - writer.finish() - import zarr - - np.testing.assert_array_equal(zarr.open_group(str(tmp_path / "a.ome.zarr"))["0"][:], data) - - -# Publication with copies (same protocol as the table and manifest) ---------------------- - - -def output_state(output): - """Every file under the output folder with its bytes, ignoring temporary folders.""" - return { - p.relative_to(output).as_posix(): p.read_bytes() - for p in output.rglob("*") - if p.is_file() and not p.relative_to(output).parts[0].startswith(".run-") - } - - -def change(path, seed): - stat = os.stat(path) - write_data(path, random_data((2, 16, 16), "uint8", seed=seed), axes="ZYX") - os.utime(path, ns=(stat.st_atime_ns, stat.st_mtime_ns + 10**9)) - - -@needs_zarr -@pytest.mark.parametrize("failed", ["copy", "csv", "manifest"]) -def test_failed_publication_restores_earlier_copies(experiment, monkeypatch, failed): - first = run_experiment(experiment, ZYX, convert_tiffs=True) - (experiment / "drug/a.tif").unlink() # its copy is removed by the next run - change(experiment / "ctrl/b.tif", 42) # its copy is replaced by the next run - before = output_state(first.output_folder) - targets = { - "copy": first.output_folder / "ome_zarr/ctrl/b.ome.zarr", - "csv": first.csv_path, - "manifest": first.manifest_path, - } - original_replace = os.replace - - def broken_replace(source, destination): - staged_new = ".run-" in str(source) and "previous" not in Path(source).parts - if Path(destination) == targets[failed] and staged_new: - raise PermissionError("simulated publication failure") - return original_replace(source, destination) - - monkeypatch.setattr(run_module.os, "replace", broken_replace) - with pytest.raises(PermissionError, match="simulated"): - run_experiment(experiment, ZYX, convert_tiffs=True) - monkeypatch.undo() - assert output_state(first.output_folder) == before - assert not list(first.output_folder.glob(".run-*")) - retried = run_experiment(experiment, ZYX, convert_tiffs=True) - assert retried.converted == ("ctrl/b.tif",) and retried.removed == ("drug/a.tif",) - assert not (first.output_folder / "ome_zarr/drug/a.ome.zarr").exists() - - -@needs_zarr -def test_failed_table_write_keeps_earlier_copies(experiment, monkeypatch): - first = run_experiment(experiment, ZYX, convert_tiffs=True) - change(experiment / "ctrl/b.tif", 42) - before = output_state(first.output_folder) - - def broken_csv(*args, **kwargs): - raise OSError("simulated CSV failure") - - monkeypatch.setattr(run_module, "write_plane_histograms", broken_csv) - with pytest.raises(OSError, match="simulated"): - run_experiment(experiment, ZYX, convert_tiffs=True) - assert output_state(first.output_folder) == before - assert not list(first.output_folder.glob(".run-*")) - - -@needs_zarr -def test_failed_rollback_keeps_the_earlier_copy_in_the_backup(experiment, monkeypatch): - first = run_experiment(experiment, ZYX, convert_tiffs=True) - change(experiment / "ctrl/b.tif", 42) - old_copy = output_state(first.output_folder / "ome_zarr/ctrl/b.ome.zarr") - original_replace = os.replace - - def broken_replace(source, destination): - source, destination = Path(source), Path(destination) - if destination == first.manifest_path or "previous" in source.parts: - raise PermissionError("simulated locked output") - return original_replace(source, destination) - - monkeypatch.setattr(run_module.os, "replace", broken_replace) - with pytest.raises(RuntimeError, match="could not be fully restored"): - run_experiment(experiment, ZYX, convert_tiffs=True) - (staged,) = first.output_folder.glob(".run-*") - backup = staged / "previous" / "ome_zarr" / "ctrl" / "b.ome.zarr" - assert output_state(backup) == old_copy - - -# Conversion settings in a run ------------------------------------------------------------ - - -@needs_zarr -@pytest.mark.parametrize( - ("settings", "compressor"), - [ - (ConversionSettings(), {"cname": "lz4", "clevel": 5}), - ( - ConversionSettings(compression="zstd", compression_level=9), - {"cname": "zstd", "clevel": 9}, - ), - (ConversionSettings(compression="none"), None), - ], -) -def test_copies_follow_the_conversion_settings(tmp_path, settings, compressor): - data = random_data((2, 100, 80), "uint16") - write_data(tmp_path / "exp/a.tif", data, axes="ZYX") - custom = ConversionSettings( - chunk_size=32, - compression=settings.compression, - compression_level=settings.compression_level, - pyramid_levels=2, - ) - result = run_experiment(tmp_path / "exp", ZYX, convert_tiffs=True, conversion=custom) - group = open_copy(result.output_folder, "a.tif") - assert sorted(group.array_keys()) == ["0", "1", "2"] - assert group["0"].chunks == (1, 32, 32) and group["2"].shape == (2, 25, 20) - np.testing.assert_array_equal(group["0"][:], data) - zarray = json.loads( - (result.output_folder / "ome_zarr/a.ome.zarr/0/.zarray").read_text(encoding="utf-8") - ) - if compressor is None: - assert zarray["compressor"] is None - else: - assert {k: zarray["compressor"][k] for k in compressor} == compressor - assert read_manifest(result.manifest_path).settings.conversion == custom - - -@needs_zarr -def test_copies_with_outdated_axis_units_are_written_again(tmp_path): - root = tmp_path / "exp" - for name in ("a.ome.tif", "b.ome.tif"): - (root / "ctrl").mkdir(parents=True, exist_ok=True) - tifffile.imwrite( - root / "ctrl" / name, - random_data((2, 8, 8), "uint8"), - ome=True, - photometric="minisblack", - metadata={ - "axes": "ZYX", - "PhysicalSizeX": 120.0, - "PhysicalSizeXUnit": "nm", - "PhysicalSizeY": 0.25, - "PhysicalSizeYUnit": "µm", - "PhysicalSizeZ": 2.0, - }, - ) - first = run_experiment(root, ZYX, convert_tiffs=True) - table = first.csv_path.read_bytes() - # Make a.ome.tif's copy look like one written before per-axis units: X's unit - # on every space axis. - attrs_path = first.output_folder / ome_zarr_relative_path("ctrl/a.ome.tif") / ".zattrs" - attrs = json.loads(attrs_path.read_text(encoding="utf-8")) - for axis in attrs["multiscales"][0]["axes"]: - axis["unit"] = "nanometer" - attrs_path.write_text(json.dumps(attrs), encoding="utf-8") - second = run_experiment(root, ZYX, convert_tiffs=True) - assert second.converted == ("ctrl/a.ome.tif",) - assert second.csv_path.read_bytes() == table # same pixels, same rows - units = [ - a["unit"] for a in multiscale(open_copy(second.output_folder, "ctrl/a.ome.tif"))["axes"] - ] - assert units == ["micrometer", "micrometer", "nanometer"] - assert run_experiment(root, ZYX, convert_tiffs=True).converted == () - - -@needs_zarr -def test_changed_conversion_settings_rewrite_the_copies(experiment): - first = run_experiment(experiment, ZYX, convert_tiffs=True) - table = first.csv_path.read_bytes() - same = run_experiment(experiment, ZYX, convert_tiffs=True) - assert same.converted == () - changed = run_experiment( - experiment, ZYX, convert_tiffs=True, conversion=ConversionSettings(chunk_size=16) - ) - assert changed.converted == ("ctrl/a.tif", "ctrl/b.tif", "drug/a.tif") - assert changed.csv_path.read_bytes() == table # same pixels, same rows - assert open_copy(changed.output_folder, "ctrl/a.tif")["0"].chunks == (1, 16, 16) - # Without conversion the settings are recorded but change nothing. - plain = run_experiment(experiment, ZYX, conversion=ConversionSettings(chunk_size=64)) - assert plain.converted == () - - -# Planes too large for the memory limit ------------------------------------------------- - - -def levels(group): - return {name: group[name][:] for name in sorted(group.array_keys())} - - -@needs_zarr -@pytest.mark.parametrize( - ("dtype", "storage"), - [ - ("uint16", {}), - ("uint16", {"compression": "zlib", "rowsperstrip": 5}), - ("int32", {"tile": (16, 32)}), - ("float32", {"compression": "zlib", "rowsperstrip": 3}), - ], -) -def test_copies_of_planes_in_blocks_equal_whole_copies(tmp_path, dtype, storage): - data = random_data((2, 91, 70), dtype) # odd sizes: the edges are repeated when halving - write_data(tmp_path / "exp/a.tif", data, axes="ZYX", **storage) - settings = ConversionSettings(chunk_size=16, pyramid_levels=3) - whole = run_experiment( - tmp_path / "exp", - ZYX, - output_folder=tmp_path / "whole", - convert_tiffs=True, - conversion=settings, - ) - limit = plane_layout(tmp_path / "exp/a.tif", ZYX, memory_limit=1).block_memory_estimate - blocks = run_experiment( - tmp_path / "exp", - ZYX, - output_folder=tmp_path / "blocks", - convert_tiffs=True, - conversion=settings, - memory_limit=limit + 4096, - ) - assert whole.written and blocks.written - assert [i.code for i in blocks.issues] == ["planes_read_in_blocks"] - assert read_manifest(blocks.manifest_path).files[0].read_in_blocks - assert not read_manifest(whole.manifest_path).files[0].read_in_blocks - assert blocks.csv_path.read_bytes() == whole.csv_path.read_bytes() - copied, expected = ( - open_copy(blocks.output_folder, "a.tif"), - open_copy(whole.output_folder, "a.tif"), - ) - np.testing.assert_array_equal(copied["0"][:], data) - got, want = levels(copied), levels(expected) - assert list(got) == ["0", "1", "2", "3"] and list(want) == list(got) - for name in want: - np.testing.assert_array_equal(got[name], want[name]) - assert dict(copied.attrs) == dict(expected.attrs) # same display ranges and scales - - -@needs_zarr -@pytest.mark.parametrize("memory_limit", [2**30, 6000]) -def test_writer_builds_smaller_levels_of_planes_in_blocks(tmp_path, memory_limit): - from morphobricks_preprocessing import Block, BlockReader, OmeZarrWriter, Plane - - data = random_data((1, 77, 53), "uint16") - mapping = prefill_mapping("ZYX") - layout = plane_layout(write_data(tmp_path / "a.tif", data, axes="ZYX"), mapping) - settings = ConversionSettings(chunk_size=16, pyramid_levels=4) - - def uneven_blocks(): - for top, bottom in [(0, 5), (5, 40), (40, 77)]: - for left, right in [(0, 33), (33, 53)]: - yield Block(top, left, data[0, top:bottom, left:right]) - - reader = BlockReader((77, 53), data.dtype, uneven_blocks) - in_blocks = OmeZarrWriter.create( - tmp_path / "blocks.ome.zarr", mapping, layout, settings=settings, memory_limit=memory_limit - ) - in_blocks.write(Plane(0, None, None, None, reader=reader)) - in_blocks.finish() - whole = OmeZarrWriter.create(tmp_path / "whole.ome.zarr", mapping, layout, settings=settings) - whole.write(Plane(0, None, None, data[0])) - whole.finish() - import zarr - - got = levels(zarr.open_group(str(tmp_path / "blocks.ome.zarr"), mode="r")) - want = levels(zarr.open_group(str(tmp_path / "whole.ome.zarr"), mode="r")) - assert list(got) == list(want) == ["0", "1", "2", "3", "4"] - for name in want: - np.testing.assert_array_equal(got[name], want[name]) - assert in_blocks._region_side() == (32 if memory_limit == 2**30 else 8) - tiny = OmeZarrWriter.create( - tmp_path / "tiny.ome.zarr", mapping, layout, settings=settings, memory_limit=1 - ) - assert tiny._region_side() == 2 # the smallest region that halves exactly - - -@needs_zarr -def test_unknown_pixel_sizes_are_marked_not_read_as_1(tmp_path): - # Issue #39: OME-NGFF 0.4 needs a scale per axis, so an unknown size is written as 1; - # the copy says which axes those are, and reading the copy gives no size for them. - from morphobricks_preprocessing import open_ngff - - write_data(tmp_path / "exp" / "a.tif", random_data((2, 8, 8), "uint8"), axes="ZYX") - result = run_experiment(tmp_path / "exp", ZYX, convert_tiffs=True) - copy = result.output_folder / ome_zarr_relative_path("a.tif") - meta = multiscale(open_copy(result.output_folder, "a.tif")) - assert meta["datasets"][0]["coordinateTransformations"][0]["scale"] == [1.0, 1.0, 1.0] - assert meta["metadata"]["unknown_scale"] == ["z", "y", "x"] - assert open_ngff(copy).physical_sizes == PhysicalSizes() - # A copy written before the marker counts as out of date and is written again. - attrs_path = copy / ".zattrs" - attrs = json.loads(attrs_path.read_text(encoding="utf-8")) - del attrs["multiscales"][0]["metadata"]["unknown_scale"] - attrs_path.write_text(json.dumps(attrs), encoding="utf-8") - again = run_experiment(tmp_path / "exp", ZYX, convert_tiffs=True) - assert again.converted == ("a.tif",) - assert run_experiment(tmp_path / "exp", ZYX, convert_tiffs=True).converted == () - - -@needs_zarr -def test_known_pixel_sizes_are_not_marked(tmp_path): - path = tmp_path / "exp/a.ome.tif" - path.parent.mkdir(parents=True) - tifffile.imwrite( - path, - random_data((2, 8, 8), "uint8"), - ome=True, - photometric="minisblack", - metadata={"axes": "ZYX", "PhysicalSizeX": 0.5, "PhysicalSizeY": 0.5}, - ) - result = run_experiment(tmp_path / "exp", ZYX, convert_tiffs=True) - meta = multiscale(open_copy(result.output_folder, "a.ome.tif")) - assert meta["metadata"]["unknown_scale"] == ["z"] # no z-step in the metadata diff --git a/tests/test_zarr_input.py b/tests/test_zarr_input.py index 178a2859..1a9fd307 100644 --- a/tests/test_zarr_input.py +++ b/tests/test_zarr_input.py @@ -3,6 +3,8 @@ import numpy as np import pytest import tifffile +from morphobricks_normalization.csv_io import read_plane_histograms +from morphobricks_normalization.histogram import compute_plane_histogram from morphobricks_preprocessing import ( AxisMapping, @@ -16,15 +18,12 @@ inspect_image, inspect_images, iter_planes, - ome_zarr_relative_path, open_ngff, plane_layout, prefill_mapping, preflight, read_manifest, - read_physical_sizes, read_plane, - read_plane_histograms, read_sample_plane, reference_axes, review_axes, @@ -33,8 +32,6 @@ ) from morphobricks_preprocessing import run as run_module from morphobricks_preprocessing.cli import main -from morphobricks_preprocessing.histogram import compute_plane_histogram -from morphobricks_preprocessing.zarr_export import ConversionSettings from morphobricks_preprocessing.zarr_input import ngff_axes from tiff_tree import write_data, write_file from zarr_tree import write_bioformats2raw, write_ngff, write_plate, zarr_major @@ -285,30 +282,6 @@ def test_header_of_a_store_carries_its_physical_sizes(tmp_path, version): assert (info.physical_sizes.x, info.physical_sizes.unit("Z")) == (0.25, "micrometer") -def test_converted_copies_give_the_sizes_of_their_tiff(tmp_path): - source = tmp_path / "exp/a.ome.tif" - source.parent.mkdir(parents=True) - tifffile.imwrite( - source, - np.zeros((2, 3, 8, 8), np.uint16), - ome=True, - photometric="minisblack", - metadata={ - "axes": "TZYX", - "PhysicalSizeX": 120.0, - "PhysicalSizeXUnit": "nm", - "PhysicalSizeY": 0.25, - "PhysicalSizeYUnit": "µm", - "PhysicalSizeZ": 2.0, - "TimeIncrement": 30.0, - "TimeIncrementUnit": "s", - }, - ) - result = run_experiment(tmp_path / "exp", prefill_mapping("TZYX"), convert_tiffs=True) - copy = result.output_folder / ome_zarr_relative_path("a.ome.tif") - assert open_ngff(copy).physical_sizes == read_physical_sizes(source) - - def test_bioformats2raw_layout(tmp_path): one = write_bioformats2raw(tmp_path / "one.ome.zarr", [random_data((2, 8, 8))]) image = open_ngff(one) @@ -683,7 +656,7 @@ def test_ome_zarr_experiment_end_to_end(zarr_experiment): pre = preflight(zarr_experiment) assert pre.ok and pre.suggested_mapping == prefill_mapping("CZYX") result = run_experiment(zarr_experiment, pre.suggested_mapping) - assert result.written and result.converted == () + assert result.written records = read_plane_histograms(result.csv_path) assert len(records) == 9 # 3 stores x 3 depth slices, each row with 2 channels import zarr @@ -755,20 +728,6 @@ def test_mixed_tiff_and_ome_zarr_experiment(tmp_path): assert {r.z for r in rows.values() if r.source_relative_path == "drug/d.ome.zarr"} == {0, 1, 2} -def test_conversion_leaves_ome_zarr_images_alone(tmp_path): - root = tmp_path / "exp" - write_data(root / "a.tif", random_data((3, 8, 8), "uint8"), axes="ZYX") - write_ngff(root / "b.ome.zarr", random_data((3, 8, 8), "uint8", seed=1), "zyx") - result = run_experiment(root, prefill_mapping("ZYX"), convert_tiffs=True) - assert result.converted == ("a.tif",) - manifest = read_manifest(result.manifest_path) - assert manifest.file("b.ome.zarr").ome_zarr_path is None - assert manifest.file("a.tif").ome_zarr_path == "ome_zarr/a.ome.zarr" - # A converted image keeps the position identifier of its source. - assert {r.position_id for r in read_plane_histograms(result.csv_path)} == {"a", "b"} - assert manifest.file("a.tif").position_id == "a" - - def test_sample_planes_cover_both_formats(tmp_path): root = tmp_path / "exp" write_data(root / "a.tif", random_data((3, 2, 8, 8), "uint8"), axes="ZCYX", imagej=True) @@ -855,30 +814,6 @@ def test_transformations_that_are_not_a_list_give_no_sizes(tmp_path, where): assert open_ngff(path).physical_sizes == PhysicalSizes() -def test_sizes_come_from_level_0_of_a_pyramid(tmp_path): - # The brick's own copy has three levels with scales 0.5, 1 and 2 along X and Y. - source = tmp_path / "exp/a.ome.tif" - source.parent.mkdir(parents=True) - tifffile.imwrite( - source, - np.zeros((2, 64, 64), np.uint16), - ome=True, - photometric="minisblack", - metadata={"axes": "ZYX", "PhysicalSizeX": 0.5, "PhysicalSizeY": 0.5, "PhysicalSizeZ": 2}, - ) - result = run_experiment( - tmp_path / "exp", - prefill_mapping("ZYX"), - convert_tiffs=True, - conversion=ConversionSettings(chunk_size=16, pyramid_levels=2), - ) - copy = result.output_folder / ome_zarr_relative_path("a.ome.tif") - image = open_ngff(copy) - assert image.dataset_path == "0" and image.shape == (2, 64, 64) - sizes = image.physical_sizes - assert (sizes.x, sizes.y, sizes.z) == (0.5, 0.5, 2.0) - - # Pixel sizes compared in the folder check ------------------------------------------------- diff --git a/tools/code_rules_baseline.json b/tools/code_rules_baseline.json index 301152e4..9297cceb 100644 --- a/tools/code_rules_baseline.json +++ b/tools/code_rules_baseline.json @@ -1,6 +1,6 @@ { "noqa": { - "ANN": 63, + "ANN": 57, "C901": 2, "PLR0913": 3, "PLR0915": 2 @@ -9,20 +9,15 @@ "identical bodies: tests/test_axis_check.py:# imagej, tests/test_header_axes.py:# imagej": 0, "identical bodies: tests/test_axis_check.py:# plain, tests/test_header_axes.py:# plain": 0, "identical bodies: tests/test_cli.py:# bright_tree, tests/test_notebook_ui.py:# bright_tree": 0, - "identical bodies: tests/test_zarr_export.py:# random_data, tests/test_zarr_input.py:# random_data": 0, - "src/morphobricks_preprocessing/axes.py _check_declared_axes: private function of # lines without a docstring": 20, - "src/morphobricks_preprocessing/axes.py _declared_axes_problem: private function of # lines without a docstring": 20, "src/morphobricks_preprocessing/axes.py check_mapping: # executable lines (max #)": 72, "src/morphobricks_preprocessing/axes.py review_axes: # executable lines (max #)": 108, - "src/morphobricks_preprocessing/axes.py: # lines (max #)": 868, + "src/morphobricks_preprocessing/axes.py: # lines (max #)": 717, "src/morphobricks_preprocessing/cli.py _check: private function of # lines without a docstring": 52, "src/morphobricks_preprocessing/cli.py _common: private function of # lines without a docstring": 19, - "src/morphobricks_preprocessing/cli.py _conversion_note: private function of # lines without a docstring": 15, "src/morphobricks_preprocessing/cli.py _masks: private function of # lines without a docstring": 39, - "src/morphobricks_preprocessing/cli.py _run: # executable lines (max #)": 93, - "src/morphobricks_preprocessing/cli.py _run: private function of # lines without a docstring": 93, - "src/morphobricks_preprocessing/cli.py build_parser: # executable lines (max #)": 147, - "src/morphobricks_preprocessing/cli.py: # lines (max #)": 637, + "src/morphobricks_preprocessing/cli.py _run: # executable lines (max #)": 78, + "src/morphobricks_preprocessing/cli.py _run: private function of # lines without a docstring": 78, + "src/morphobricks_preprocessing/cli.py build_parser: # executable lines (max #)": 108, "src/morphobricks_preprocessing/dataset.py _missing_issues: private function of # lines without a docstring": 32, "src/morphobricks_preprocessing/dataset.py _unbalanced_issues: private function of # lines without a docstring": 19, "src/morphobricks_preprocessing/design.py _check_strict_values: private function of # lines without a docstring": 19, @@ -33,10 +28,10 @@ "src/morphobricks_preprocessing/discovery.py finish: # executable lines (max #)": 145, "src/morphobricks_preprocessing/discovery.py: # lines (max #)": 847, "src/morphobricks_preprocessing/image_type.py _rgb_problem: private function of # lines without a docstring": 12, - "src/morphobricks_preprocessing/manifest.py _file_dict: private function of # lines without a docstring": 18, - "src/morphobricks_preprocessing/manifest.py _file_entry: private function of # lines without a docstring": 21, - "src/morphobricks_preprocessing/manifest.py file_entry_for: # parameters (max #)": 10, - "src/morphobricks_preprocessing/manifest.py: # lines (max #)": 754, + "src/morphobricks_preprocessing/manifest.py _file_dict: private function of # lines without a docstring": 17, + "src/morphobricks_preprocessing/manifest.py _file_entry: private function of # lines without a docstring": 20, + "src/morphobricks_preprocessing/manifest.py file_entry_for: # parameters (max #)": 9, + "src/morphobricks_preprocessing/manifest.py: # lines (max #)": 732, "src/morphobricks_preprocessing/masks.py _duplicate_issues: private function of # lines without a docstring": 19, "src/morphobricks_preprocessing/masks.py _folder_issues: private function of # lines without a docstring": 19, "src/morphobricks_preprocessing/masks.py _name_issues: private function of # lines without a docstring": 15, @@ -47,14 +42,12 @@ "src/morphobricks_preprocessing/metadata.py _check_pixel_sizes: # executable lines (max #)": 71, "src/morphobricks_preprocessing/metadata.py _check_plane_counts: private function of # lines without a docstring": 27, "src/morphobricks_preprocessing/metadata.py check_metadata: # executable lines (max #)": 98, - "src/morphobricks_preprocessing/metadata.py: # lines (max #)": 936, + "src/morphobricks_preprocessing/metadata.py: # lines (max #)": 934, "src/morphobricks_preprocessing/notebook_ui.py __init__: # executable lines (max #)": 110, "src/morphobricks_preprocessing/notebook_ui.py _bins_help: private function of # lines without a docstring": 12, "src/morphobricks_preprocessing/notebook_ui.py _browse_csv: private function of # lines without a docstring": 11, "src/morphobricks_preprocessing/notebook_ui.py _build_axis_step: # executable lines (max #)": 82, "src/morphobricks_preprocessing/notebook_ui.py _build_axis_step: private function of # lines without a docstring": 82, - "src/morphobricks_preprocessing/notebook_ui.py _build_conversion_card: # executable lines (max #)": 139, - "src/morphobricks_preprocessing/notebook_ui.py _build_conversion_card: private function of # lines without a docstring": 139, "src/morphobricks_preprocessing/notebook_ui.py _build_design_card: private function of # lines without a docstring": 53, "src/morphobricks_preprocessing/notebook_ui.py _build_design_editor: private function of # lines without a docstring": 46, "src/morphobricks_preprocessing/notebook_ui.py _build_folder_step: # executable lines (max #)": 140, @@ -62,8 +55,8 @@ "src/morphobricks_preprocessing/notebook_ui.py _build_image_type_card: private function of # lines without a docstring": 30, "src/morphobricks_preprocessing/notebook_ui.py _build_results_step: # executable lines (max #)": 92, "src/morphobricks_preprocessing/notebook_ui.py _build_results_step: private function of # lines without a docstring": 92, - "src/morphobricks_preprocessing/notebook_ui.py _build_run_step: # executable lines (max #)": 120, - "src/morphobricks_preprocessing/notebook_ui.py _build_run_step: private function of # lines without a docstring": 120, + "src/morphobricks_preprocessing/notebook_ui.py _build_run_step: # executable lines (max #)": 119, + "src/morphobricks_preprocessing/notebook_ui.py _build_run_step: private function of # lines without a docstring": 119, "src/morphobricks_preprocessing/notebook_ui.py _check: private function of # lines without a docstring": 29, "src/morphobricks_preprocessing/notebook_ui.py _design_load_note: private function of # lines without a docstring": 27, "src/morphobricks_preprocessing/notebook_ui.py _draw: private function of # lines without a docstring": 11, @@ -71,7 +64,7 @@ "src/morphobricks_preprocessing/notebook_ui.py _folder_edited: private function of # lines without a docstring": 19, "src/morphobricks_preprocessing/notebook_ui.py _folder_tiles: private function of # lines without a docstring": 18, "src/morphobricks_preprocessing/notebook_ui.py _mapping_changed: private function of # lines without a docstring": 42, - "src/morphobricks_preprocessing/notebook_ui.py _plan_html: private function of # lines without a docstring": 57, + "src/morphobricks_preprocessing/notebook_ui.py _plan_html: private function of # lines without a docstring": 49, "src/morphobricks_preprocessing/notebook_ui.py _recommended_changed: private function of # lines without a docstring": 17, "src/morphobricks_preprocessing/notebook_ui.py _refresh_axis_next: private function of # lines without a docstring": 22, "src/morphobricks_preprocessing/notebook_ui.py _refresh_axis_step: # executable lines (max #)": 90, @@ -79,18 +72,17 @@ "src/morphobricks_preprocessing/notebook_ui.py _refresh_design_status: private function of # lines without a docstring": 51, "src/morphobricks_preprocessing/notebook_ui.py _refresh_folder_next: private function of # lines without a docstring": 19, "src/morphobricks_preprocessing/notebook_ui.py _refresh_missing_card: private function of # lines without a docstring": 29, - "src/morphobricks_preprocessing/notebook_ui.py _refresh_run_step: private function of # lines without a docstring": 41, + "src/morphobricks_preprocessing/notebook_ui.py _refresh_run_step: private function of # lines without a docstring": 40, "src/morphobricks_preprocessing/notebook_ui.py _render_preview: private function of # lines without a docstring": 28, - "src/morphobricks_preprocessing/notebook_ui.py _reset_settings: private function of # lines without a docstring": 15, + "src/morphobricks_preprocessing/notebook_ui.py _reset_settings: private function of # lines without a docstring": 14, "src/morphobricks_preprocessing/notebook_ui.py _save_design: private function of # lines without a docstring": 15, "src/morphobricks_preprocessing/notebook_ui.py _show_image_type_note: private function of # lines without a docstring": 19, "src/morphobricks_preprocessing/notebook_ui.py _show_issue_summary: private function of # lines without a docstring": 40, - "src/morphobricks_preprocessing/notebook_ui.py _show_run_result: private function of # lines without a docstring": 42, - "src/morphobricks_preprocessing/notebook_ui.py _space_warning_html: private function of # lines without a docstring": 37, + "src/morphobricks_preprocessing/notebook_ui.py _show_run_result: private function of # lines without a docstring": 35, "src/morphobricks_preprocessing/notebook_ui.py _toggle_confirmation: private function of # lines without a docstring": 48, "src/morphobricks_preprocessing/notebook_ui.py _update_output_hint: private function of # lines without a docstring": 20, - "src/morphobricks_preprocessing/notebook_ui.py run: # executable lines (max #)": 65, - "src/morphobricks_preprocessing/notebook_ui.py: # lines (max #)": 3318, + "src/morphobricks_preprocessing/notebook_ui.py run: # executable lines (max #)": 63, + "src/morphobricks_preprocessing/notebook_ui.py: # lines (max #)": 3051, "src/morphobricks_preprocessing/physical_sizes.py _imagej_sizes: private function of # lines without a docstring": 27, "src/morphobricks_preprocessing/physical_sizes.py _ome_sizes: private function of # lines without a docstring": 20, "src/morphobricks_preprocessing/physical_sizes.py: # lines (max #)": 706, @@ -102,22 +94,16 @@ "src/morphobricks_preprocessing/run.py _bin_count: private function of # lines without a docstring": 12, "src/morphobricks_preprocessing/run.py _exclusion_issues: private function of # lines without a docstring": 34, "src/morphobricks_preprocessing/run.py _pixel_issues: private function of # lines without a docstring": 38, - "src/morphobricks_preprocessing/run.py run_experiment: # executable lines (max #)": 250, - "src/morphobricks_preprocessing/run.py run_experiment: # parameters (max #)": 17, - "src/morphobricks_preprocessing/run.py: # lines (max #)": 1075, - "src/morphobricks_preprocessing/zarr_export.py _create_array: private function of # lines without a docstring": 19, - "src/morphobricks_preprocessing/zarr_export.py _omero: private function of # lines without a docstring": 13, - "src/morphobricks_preprocessing/zarr_export.py _open: private function of # lines without a docstring": 12, - "src/morphobricks_preprocessing/zarr_export.py _track_channel: private function of # lines without a docstring": 13, - "src/morphobricks_preprocessing/zarr_export.py: # lines (max #)": 622, + "src/morphobricks_preprocessing/run.py run_experiment: # executable lines (max #)": 195, + "src/morphobricks_preprocessing/run.py run_experiment: # parameters (max #)": 15, + "src/morphobricks_preprocessing/run.py: # lines (max #)": 895, "tests/test_discovery.py: # lines (max #)": 848, "tests/test_masks.py: # lines (max #)": 618, "tests/test_metadata.py: # lines (max #)": 629, - "tests/test_notebook_ui.py: # lines (max #)": 1736, - "tests/test_planes.py: # lines (max #)": 819, + "tests/test_notebook_ui.py: # lines (max #)": 1604, + "tests/test_planes.py: # lines (max #)": 818, "tests/test_run.py: # lines (max #)": 982, - "tests/test_zarr_export.py: # lines (max #)": 976, - "tests/test_zarr_input.py: # lines (max #)": 934, + "tests/test_zarr_input.py: # lines (max #)": 869, "tests/zarr_tree.py write_ngff: # parameters (max #)": 12 } }