diff --git a/AGENTS.md b/AGENTS.md index e70cf30f..d54a8bc5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,13 +25,15 @@ brick**. - **Input:** one experiment folder of TIFF images and/or OME-Zarr images (OME-NGFF 0.4/0.5, `zarr` extra). -- **Output:** one intensity histogram per 2D plane in `plane_histograms.csv`, plus +- **Output:** `plane_histograms.csv` (schema version 3): one row per XYC plane (an image at + one Z and T), with one intensity histogram per channel in its own columns, plus `experiment_design.csv` (repeat groups, confirmation, controls) and `run_manifest.json`, all in an output folder outside the experiment. - **Statistics code (from the normalization brick):** the per-plane histogram, the CSV reader and writer, pooling (`PooledHistogram`) and normalization parameters (`min_max`, `percentile_range`, `z_score`, `robust_z_score`) live in the required dependency - `morphobricks_normalization` (pinned to its release v0.1.0). This brick runs them over + `morphobricks_normalization` (pinned by commit to its Milestone 2b merge `0e53e67`; no + tags until its v0.2.0, plan §12 question 27). This brick runs them over the experiment. Later bricks such as patch extraction use them to normalise any group of planes without rereading pixels. The old names stay importable from here, deprecated, for one version (follow-up P1, normalization plan N6). @@ -74,7 +76,9 @@ src/morphobricks_preprocessing/ validation.py ValidationIssue (severity, code, message, paths, action), error()/warning() builders axes.py AxisMapping (role per dimension), prefill, check_mapping, review_axes (one mapping applied to every file, missing axes as singletons only when confirmed) - planes.py plane-by-plane reader (iter_planes, read_plane, read_sample_plane), memory estimate, + planes.py plane-by-plane reader (iter_planes, read_plane, read_sample_plane), table rows as + XYC planes whose channels are read one at a time (iter_xyc_planes, XycPlane), + memory estimate, check_planes (preflight memory / layout errors); planes over the limit read in blocks of TIFF strips/tiles or zarr chunks (Plane.blocks, BlockReader), reduced previews @@ -98,7 +102,8 @@ src/morphobricks_preprocessing/ folder_step/axis_step take title, purpose, intro and show_next (and show_output for step 1) so other bricks can embed them (plan §12 Q26) notebooks/MorphoBricks_Preprocessing.ipynb thin notebook: every code cell calls notebook_ui only -tests/ pytest with simulated TIFFs (tests/tiff_tree.py, tests/simulation.py) +tests/ pytest with simulated TIFFs (tests/tiff_tree.py, tests/simulation.py); the + schema-version-2 fixture (tests/v2_fixture.py, tests/data) experiments/binning_study/ the study behind the binning decisions (not part of the package) ``` @@ -109,7 +114,8 @@ Moved to [morphobricks-normalization](https://github.com/CellMigrationLab/morpho dtypes.py supported dtypes, default and allowed bin counts histogram.py per-plane histogram (PlaneHistogram, compute_plane_histogram, and compute_block_histogram: two passes over blocks, identical result), memory estimate - records.py PlaneRecord (one CSV row), plane_id, group_records + records.py PlaneRecord (one CSV row: an XYC plane, one histogram per channel), + make_plane_id, group_records csv_io.py write/read plane_histograms.csv (atomic write, labels always text) pooling.py PooledHistogram: pooled counts, never mean-of-means normalization.py NormalizationParameters (offset, scale) and the four methods @@ -196,9 +202,15 @@ functions the tests use. - **Series and pyramids:** only the first image series, at full-resolution level 0. A file with more than one series is an error (`multiple_series`). OME pyramids are one series; non-OME SubIFD pyramids appear as two series in tifffile and are therefore blocked. -- **Coordinates:** plane coordinates `z, c, t` are zero-based and always written: 0 when - the image has no such axis (question 17). `PlaneRecord` stores None as 0; `Plane` - coordinates stay None for axes the file lacks. +- **Coordinates:** a row is an XYC plane; its coordinates `z, t` are zero-based and always + written: 0 when the image has no such axis (question 17). Channels are columns, + numbered from 0 (question 27). `PlaneRecord` stores None as 0; `Plane` (2D, with + `z, c, t`) and `XycPlane` coordinates stay None for axes the file lacks. +- **Channel counts (N43, question 27):** every image has the same number of channels (the + C axis, 1 without one, also for a confirmed missing C), else the error + `different_channel_counts`: from the headers in `metadata.check_metadata` (C, or S + without C) and from the confirmed roles in `review_axes`, which supersedes it + (`axes._SUPERSEDED_METADATA_CODES`). Shared builder `metadata.channel_count_issues`. - **OME-Zarr axes:** translated from OME-NGFF metadata (`zarr_input.ngff_axes`: time -> T, channel -> C, space z/y/x -> Z/Y/X, anything else -> Q). The experiment mapping is defined on the TIFF arrangement if there are TIFFs; OME-Zarr images with another order @@ -241,15 +253,26 @@ functions the tests use. Percentile range defaults to 1–99.8. Robust z-score uses MAD × 1.4826. A zero or non-finite scale is an error. -### CSV (normalization contract §6.2, schema version 2) +### CSV (normalization contract §6.2, schema version 3, follow-up P3b) +- **Rows:** one per XYC plane (image, Z, T), written with the brick's writer from + `PlaneRecord(path, *labels, histograms=…, z=, t=, position_id=)`. - **Columns:** - `position_id, image_file, z, c, t, condition, biological_repeat, technical_repeat, - source_relative_path, plane_id, dtype, bit_depth, pixel_min, pixel_max, total_pixel_count, - valid_pixel_count, binning_min, binning_max, bin_count, schema_version, histogram_bin_00000…`. -- **`plane_id`** = relative path plus `::z=…,c=…,t=…`, always all three - (`img.tif::z=0,c=0,t=0` for a 2D image). It is the row key. -- **Rows** are sorted by path, then z, c, t. + `position_id, image_file, z, t, condition, biological_repeat, technical_repeat, + source_relative_path, plane_id, dtype, bit_depth, channel_count, total_pixel_count, + bin_count, schema_version`, then per channel `i`: `Intensity_MinIntensity_C`, + `Intensity_MaxIntensity_C`, `Histogram_ValidPixelCount_C`, + `Histogram_BinningMin_C`, `Histogram_BinningMax_C`, `Histogram_Count_C_00000…`. +- **`plane_id`** = relative path plus `::z=…,t=…`, always both (`img.tif::z=0,t=0` for a + 2D image). It is the row key. +- **Rows** are sorted by path, then z, t. +- **Earlier versions:** the reader refuses schema 1 and 2 (`CsvFormatError`); a run + recomputes them completely (manifest `schema_version` differs). Each channel's + histogram equals the version-2 row's, whole and in blocks + (`tests/test_schema_v2_equivalence.py` with the committed fixture in `tests/data`, + written by `tests/data/make_v2_fixture.py` with the pre-P3b code). +- **Step 4** pools every channel of a group together, as with version 2 + (`report.channel_histograms`), so its table and plot are unchanged. - **`position_id`** is filled in every row (D02, plan §12 question 20). Earlier tables with it blank are filled on the next run without reading pixels. - **Reading:** labels are always read back as text (`NA`, `001`, `TRUE` stay as typed). @@ -280,9 +303,15 @@ functions the tests use. - **Unit of reading:** read one TIFF page at a time and yield the planes inside it. Never load a full Z/C/T volume unless the file stores it as a single page. +- **Channels of a row (question 27, the owner's choice):** `iter_xyc_planes` reads and bins + them one at a time (`XycPlane.histograms`), keeping only the finished histograms; the + last decoded page is kept, so a page holding several channels (RGB) is decoded once. + When C is a page axis and pages hold several Z/T planes (volumetric), pages are decoded + again per XYC plane: time, never memory. - **Estimate per plane, from the header:** decoded page + encoded bytes + plane copy + - histogram temporaries (`histogram.estimate_histogram_memory`). It is deliberately an - upper estimate. + histogram temporaries (`histogram.estimate_histogram_memory`) + the finished bin counts + of every channel of the XYC plane (`8 × bins × channel_count`). It is deliberately an + upper estimate, and one whole/blocks decision covers every channel. - **Limit:** default 1 GiB. A plane over the limit is read in blocks (question 15): - Blocks are whole TIFF strips or tiles, or OME-Zarr chunks, in full-width bands first, as many as fit the limit (`planes._choose_block`, estimate `planes._block_memory`). @@ -315,7 +344,8 @@ functions the tests use. - **Pixel sizes:** `different_pixel_sizes` (X by Y; z-step only for images with Z > 1; micrometres, 4 significant digits) and `missing_pixel_size` (known for some only). OME-Zarr sizes come from the level-0 scale with axis units; no unit means unknown. -- **`different_plane_counts`:** C (or S without C), Z, T or any other non-XY dimension; +- **`different_plane_counts`:** Z, T or any other dimension except X, Y, C and S; + channels are the error `different_channel_counts` (question 27), also in this panel; `different_sizes` is width/height only, so no dimension is reported twice. - **Notebook:** the consistency codes (`dataset.CONSISTENCY_CODES`) appear only in step 1's "Dataset consistency" panel with the balance table; the Messages card keeps the rest. @@ -358,13 +388,14 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p | Topic | Decision | |---|---| | Binning | Per-plane min–max, same bin count per experiment, pooled counts for groups (binning study) | -| Core dependencies | `numpy`, `tifffile` and `morphobricks-normalization` (git dependency pinned to a release tag; `[tool.hatch.metadata] allow-direct-references`). `ipywidgets` and `matplotlib` in the `notebook` extra; `zarr` in the `zarr` extra; `pytest` and `ruff` in `dev` | +| Core dependencies | `numpy`, `tifffile` and `morphobricks-normalization` (git dependency pinned by commit until its v0.2.0, the owner wants no tags for now; `[tool.hatch.metadata] allow-direct-references`). `ipywidgets` and `matplotlib` in the `notebook` extra; `zarr` in the `zarr` extra; `pytest` and `ruff` in `dev` | | OME-Zarr (Q12) | Input may be TIFF and/or OME-Zarr (folders ending in `.zarr`), mixed. OME-Zarr axes come from their metadata and are matched by role; level 0 of a single image only, several images per store is `multiple_series`. TIFF choice once per experiment (`convert_tiffs`): read directly (default) or convert to `/ome_zarr` (OME-NGFF 0.4) and compute rows from the copies; switching recomputes TIFF rows and removes unused copies. Reads OME-NGFF 0.4 and 0.5 (0.5 needs zarr 3). In the notebook: its own step-3 card "(Optional) Image conversion" after Settings (Q13): radio buttons (read directly = recommended), a disk-space warning (amber, red when the estimate exceeds free space, never blocking), and conversion settings (chunk size, compression + level, pyramid levels) shown only when converting. CLI `--convert-tiff` with `--chunk-size`, `--compression`, `--compression-level`, `--pyramid-levels`. No overlap setting: chunks never overlap | | System files (Q5) | Ignored silently; `single` check and sibling uniqueness are case-insensitive | | Too-deep folders (Q7) | Error | | Folders named `single` (Q23) | Accepted as the level's value `single` when lower case and the only folder in its place (MorphoBricks one-member level, MorphoRelate M3); another case or other folders beside it stay `reserved_folder_name` | | Channel names (Q1) | Manifest only (from OME), warning `channel_names_differ`; CSV stays at schema 1 | -| Plane coordinates (Q17) | `z`, `c`, `t` always written (0 for absent axes), `plane_id` always `::z=…,c=…,t=…`; CSV schema 2, version-1 tables recomputed on the next run | +| Plane coordinates (Q17) | `z`, `c`, `t` always written (0 for absent axes), `plane_id` always `::z=…,c=…,t=…`; CSV schema 2, version-1 tables recomputed on the next run. Superseded for the CSV by Q27: rows have `z`, `t` and channels in columns | +| Channels across columns (Q27, P3b) | CSV schema 3 with the brick's writer: one row per XYC plane, per-channel CellProfiler-style columns, `plane_id` `::z=…,t=…`. Channels read one at a time (owner's choice); memory estimate adds the plane's finished bin counts. `different_channel_counts` error before any read (headers in step 1, confirmed roles in step 2); `different_plane_counts` without channels. Schema 1/2 tables recomputed; manifest version 1 records schema 3; step 4 output unchanged. Brick pinned by commit `0e53e67` | | OME-NGFF labels (Q18) | Read-only helpers `ngff_label_names`/`open_ngff_labels` for later bricks (CellPatcher masks); label images are never analysed and the outputs do not change | | Multi-series / pyramids (Q2) | Level 0 only; more than one series is an error | | RGB `S` axis (Q3) | Prefilled as C, confirmed like any role | @@ -435,12 +466,13 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p `write_plate`) writes stores with zarr directly, independent of `zarr_export`; 0.5 tests skip with zarr 2.18. - **Proving "one page at a time":** spy on `planes._read_page`. To prove "only changed - files are read", spy on `run.iter_planes`. For planes in blocks, spy on + files are read", spy on `run.iter_xyc_planes` (`run.iter_planes` only converts TIFF + images to OME-Zarr). For planes in blocks, spy on `planes._read_segment` and make `asarray` of every tifffile class fail. - **Planes in blocks with small test images:** the memory limit `plane_layout(path, mapping, memory_limit=1).block_memory_estimate` gives the smallest - blocks a file can be read in. Consume each lazy plane before the next one (the file - closes when `iter_planes` ends). The histogram tests, with an independent copy of the + blocks a file can be read in. Consume each lazy plane (and each `XycPlane`'s channels) + before the next one (the file closes when `iter_planes`/`iter_xyc_planes` ends). The histogram tests, with an independent copy of the original whole-plane formulas (`reference_histogram`), are in the normalization brick. - **Speed:** rerun tests use 8-bit images (256 bins). Wide 65,536-bin CSVs make every run about 1 s. @@ -653,7 +685,9 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb v0.1.0 with deprecated re-exports, its version in the manifest, Step 0 checking for it, and contract §3–§7 moved to the normalization contract (PRs #16 and #17, question 19), and `position_id` filled per cross-brick decision D02 (question 20), and follow-ups - P2 (dataset consistency warnings, question 21) and P3 (experiment design, question 22). + P2 (dataset consistency warnings, question 21) and P3 (experiment design, question 22), + and P3b (`plane_histograms.csv` schema version 3 with the channels in columns, question + 27). - **Unreleased (for MorphoRelate milestone M3):** folders named `single` read as a one-member level (question 23), a unit per axis in `PhysicalSizes` and physical sizes in `inspect_image` (question 24), OME-Zarr physical sizes from the OME-NGFF scale (question diff --git a/README.md b/README.md index 2a157025..9b254e72 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,9 @@ [![CI](https://github.com/CellMigrationLab/morphobricks-preprocessing/actions/workflows/ci.yml/badge.svg)](https://github.com/CellMigrationLab/morphobricks-preprocessing/actions/workflows/ci.yml) The preprocessing brick of MorphoBricks. It reads an experiment folder of -microscopy TIFFs, computes one intensity histogram per 2D plane, and writes them, -with each plane's experimental labels, to `plane_histograms.csv`. Later bricks +microscopy TIFFs, computes one intensity histogram per channel of every plane, and writes +them, with each plane's experimental labels, to `plane_histograms.csv`: one row per plane +(an image at one depth slice and time point), each channel in its own columns. Later bricks (for example patch extraction) use that table to calculate normalization parameters for any group of images without rereading the pixels. @@ -21,12 +22,14 @@ Pre-alpha. Implemented and tested: - Folder discovery and validation (`discovery`, `validation`) - TIFF header inspection without reading pixels (`metadata`) - Dataset consistency warnings in the folder check: images per condition and biological - repeat, repeats missing from some conditions, differing plane counts and pixel sizes - (`dataset`, `physical_sizes`) + repeat, repeats missing from some conditions, differing plane counts and pixel sizes, + and an error for differing numbers of channels (`dataset`, `metadata`, + `physical_sizes`) - Axis mapping: prefill from metadata, validation, applying one mapping to every file (`axes`) -- Reading TIFFs one 2D plane at a time, with a memory limit and a preview plane; - planes too large for the limit are read in blocks with identical results (`planes`) +- Reading TIFFs one 2D plane at a time, with a memory limit and a preview plane, and the + channels of a table row one after another; planes too large for the limit are read in + blocks with identical results (`planes`) - The experiment design: pairing biological repeats across conditions, confirming they are comparable, and marking controls, in `experiment_design.csv` (`design`) - Running an experiment end to end and updating earlier output incrementally, with a @@ -34,9 +37,10 @@ Pre-alpha. Implemented and tested: - A step-by-step notebook ([notebooks/MorphoBricks_Preprocessing.ipynb](notebooks/MorphoBricks_Preprocessing.ipynb)) and a command line (`morphobricks-preprocessing check` / `run`) -From the normalization brick (`morphobricks_normalization`, a required dependency): -per-plane histograms (`histogram`), the histogram table (`csv_io`), pooled group -statistics (`pooling`) and normalization parameters (`normalization`). Importing these +From the normalization brick (`morphobricks_normalization`, a required dependency, pinned +to the commit of its Milestone 2b until its v0.2.0 release): per-plane histograms +(`histogram`), the histogram table (`csv_io`, schema version 3), pooled group statistics +(`pooling`) and normalization parameters (`normalization`). Importing these names from `morphobricks_preprocessing` still works for one version, with a `DeprecationWarning`. @@ -110,22 +114,22 @@ record = PlaneRecord( condition="control", biological_repeat="bio_rep1", technical_repeat="tech_rep1", - histogram=compute_plane_histogram(plane), + histograms=[compute_plane_histogram(plane)], # one per channel ) write_plane_histograms([record], "plane_histograms.csv") -# In a later brick: pool any group and derive normalization parameters +# In a later brick: pool one channel of any group and derive normalization parameters records = read_plane_histograms("plane_histograms.csv") for key, members in group_records(records, "condition").items(): - pooled = PooledHistogram(members) + pooled = PooledHistogram(members, channel=0) params = percentile_range(pooled, low=1, high=99.8) normalized = params.apply(plane, clip=(0, 1)) ``` ## How the statistics work -- Every plane is binned over **its own minimum to maximum**, with the same number - of bins for every plane. Adding images never changes existing rows. +- Every channel of every plane is binned over **its own minimum to maximum**, with + the same number of bins for every plane. Adding images never changes existing rows. - Integer bins have **whole-number limits**. When a plane's range fits within the bin count (always the case for 8- and 16-bit images with the default 65,536 bins), the histogram is exact. diff --git a/docs/contract.md b/docs/contract.md index e76b05a9..0a469a1d 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -34,9 +34,11 @@ whole experiment; any change to them goes through the normalization brick. ## 2. Axes Roles X, Y, Z, C, T. Exactly one X and one Y; each role at most once. Non-singleton axes -must have a supported role. One confirmed mapping per experiment. Plane coordinates `z`, -`c`, `t` are zero-based and always written: an axis the image does not have, or has with -length 1 and no role, has coordinate 0. +must have a supported role. One confirmed mapping per experiment. A row of the table is +an XYC plane: every channel of an image at one Z and T position (§5). Plane coordinates +`z` and `t` are zero-based and always written: an axis the image does not have, or has +with length 1 and no role, has coordinate 0. Channels are numbered from 0 in the image's +channel order; an image without a C axis has one channel. | Rule | Detail | |---|---| @@ -52,7 +54,8 @@ length 1 and no role, has coordinate 0. | Experiment mapping | Defined on the TIFF arrangement when the experiment has TIFF files; in an experiment of OME-Zarr images only, on their arrangement with the most axes. | | Channel names, significant bits | Recorded in the run manifest when metadata provides them; not part of the CSV. Channel names that differ at the same channel index are a warning. | | Physical sizes | Read with the header, not part of the CSV or the manifest, and compared between images in the folder check (§11, in micrometres with each axis's own unit; a size without a known unit cannot be compared): `PhysicalSizes` with the size of one step along X, Y, Z and T and each axis's unit (OME-NGFF unit names). TIFF: OME `PhysicalSize{X,Y,Z}` with their own units (micrometres when left out) and `TimeIncrement`, or ImageJ resolution, `spacing` and `finterval` with `unit`, `yunit`, `zunit`. OME-Zarr (0.4 and 0.5): the `scale` of level 0's `coordinateTransformations`, times the multiscales entry's own `scale` when given, with each axis's `unit` (common spellings such as `µm`, `um`, `micron` and `s` given as UDUNITS-2 names, others as written). Ångström (`Å`) and ImageJ's escaped micro sign (`\u00B5m`) are recognised. Nothing is guessed: what the metadata does not give is None; a scale in a separate file (`path`) or not one number per axis gives no sizes; an OME-Zarr scale without a unit is kept with the unit None. | -| Plane counts | Images may have different numbers of channels, depth slices, time points or planes along an unassigned dimension; this is a warning (§11). | +| Plane counts | Images may have different numbers of depth slices, time points or planes along an unassigned dimension; this is a warning (§11). | +| Channel counts | Every image of an experiment has the same number of channels (normalization plan N43): along the C axis, 1 without one (also when a missing C axis is confirmed as length 1). A different number is the error `different_channel_counts`, found before any image is read: in the folder check from the headers (the `C` dimension, or the RGB samples `S` when there is no `C`), and again with the confirmed axis roles, which decide (§11). | ## 3. Pixel types @@ -68,17 +71,27 @@ is binned over its own exact range; integer bins have whole-number limits, float equal widths; every pixel is within half a bin of its representative; a plane read in blocks gives exactly the row of the whole plane. -## 5. CSV schema (`plane_histograms.csv`, schema version 2) +## 5. CSV schema (`plane_histograms.csv`, schema version 3) Defined by the [normalization brick's contract][normalization contract] §6.2 (columns) and -§6.1 (shared columns). This brick writes it next to `run_manifest.json` (§9), with plane -coordinates and labels as in §1–§2. +§6.1 (shared and per-channel columns). This brick writes it with that brick's writer next +to `run_manifest.json` (§9), with plane coordinates and labels as in §1–§2: one row per +XYC plane (an image at one Z and T position), each channel's extrema, pixel counts and +bins in its own columns (`Intensity_MinIntensity_C0`, `Histogram_Count_C1_00042`, …), +`plane_id` = `::z=…,t=…`, rows sorted by path, then `z`, then `t` +(follow-up P3b, plan §12 question 27). + +| Rule | Detail | +|---|---| +| Reading | The channels of a plane are read and binned one at a time; only one channel's pixels are in memory, next to the finished histograms of the plane's other channels. The memory estimate of a plane (plan §5) includes those histograms' bin counts, and one decision, whole or in blocks, covers every channel of every plane of an image. | +| Histograms | Each channel's histogram is the one schema version 2 held in that channel's row, whole or in blocks (tested against a table written by version 2). | +| Earlier versions | A table of schema version 1 or 2 (one row per plane and channel, with a `c` column) is recomputed completely on the next run, like any change of version (§9). | This brick fills `position_id` in every row, with the image's position identifier (§1). The file-name fallback follows cross-brick decision D02 (one file is exactly one position here); the normalization contract's note that `position_id` is never inferred from file -names predates D02. The columns and `schema_version` are unchanged. A table written before -this rule, with `position_id` blank, is filled on the next run without reading pixels. +names predates D02. A table written before this rule, with `position_id` blank, is filled +on the next run without reading pixels. ## 6. Group statistics @@ -94,8 +107,8 @@ method is an offset and a scale, `(pixels − offset) / scale`. | Severity | Situations | Effect | |---|---|---| -| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); mixed dtypes; unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; output folder inside the experiment; no usable planes | Processing is blocked | -| Warning | Files that are not images; empty folders; images outside leaf folders; images excluded by the user, or excluded paths that match no image; channel names differing between files; non-finite pixels; different image widths or heights; different numbers of channels, depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; planes over the memory limit that are read in blocks | Processing continues; the item is reported | +| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; output folder inside the experiment; no usable planes | Processing is blocked | +| Warning | Files that are not images; empty folders; images outside leaf folders; images excluded by the user, or excluded paths that match no image; channel names differing between files; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; planes over the memory limit that are read in blocks | Processing continues; the item is reported | Every message states what happened in plain language, lists the affected paths, and suggests an action. Technical details go to the diagnostic log. @@ -127,14 +140,14 @@ file's schema or statistical interpretation. | Key | Meaning | |---|---| -| `manifest_version`, `schema_version` | Manifest format (`1`) and CSV format (`2`). | +| `manifest_version`, `schema_version` | Manifest format (`1`) and CSV format (`3`). A manifest whose `schema_version` is not the current one makes the next run recompute every row. | | `written_at` | UTC time of writing, ISO 8601. | | `experiment_root` | Absolute path of the experiment folder, for information. | | `dtype` | Pixel type of the experiment. | | `settings` | `bin_count`, `memory_limit` (bytes), `missing_as_singleton`, `exclude` (sorted relative paths), `convert_tiffs` (whether TIFF images are converted to OME-Zarr and processed from the copies; false when absent), `conversion` (`chunk_size`, `compression`, `compression_level`, `pyramid_levels` — null for automatic; defaults when absent). | | `axis_mapping` | `axes` (file axis labels) and `roles` (`X`, `Y`, `Z`, `C`, `T` or null per axis). | | `labels` | One entry per leaf folder: `folder` (relative path, `.` for the root) and its three labels. | -| `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count`, `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `ome_zarr_path` (for a TIFF converted to OME-Zarr, the copy relative to the output folder; null otherwise or when the key is absent), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent), `position_id` (the image's position identifier, §1; null when absent, in manifests written before it). For an OME-Zarr image, `size` is the total size of its files and `mtime_ns` the time of its newest file. | +| `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count` (its rows in the table: XYC planes), `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `ome_zarr_path` (for a TIFF converted to OME-Zarr, the copy relative to the output folder; null otherwise or when the key is absent), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent), `position_id` (the image's position identifier, §1; null when absent, in manifests written before it). For an OME-Zarr image, `size` is the total size of its files and `mtime_ns` the time of its newest file. | | `csv_sha256` | SHA-256 of the table written with this manifest. | | `experiment_design_sha256` | SHA-256 of the `experiment_design.csv` written with this run (§12), so later bricks can check that the design they read is the one used; null when the run had no design, or absent in manifests written before it. Optional key: the manifest stays version 1. | | `versions` | Versions of `morphobricks-preprocessing`, `morphobricks-normalization` (which computes the histograms and writes the CSV; absent in manifests written before it), `numpy`, `tifffile` and Python. Information only: a different version never causes a rebuild. | @@ -176,6 +189,8 @@ presentation does not change the conversion rules below. The folder check reads only folder names and image headers. It gathers every consistency issue of an experiment in one place (normalization plan N37). The new checks are warnings: they never block processing, and each names the affected images or folders and what to check. +The exception is the number of channels, an error since the channels are columns of one +row (follow-up P3b, normalization plan N43). | Code | Severity | Rule | |---|---|---| @@ -183,7 +198,8 @@ they never block processing, and each names the affected images or folders and w | `repeat_missing_conditions` | Warning | With more than one condition, a biological repeat not found in every condition is reported, with its folders. Repeats are matched across conditions by their repeat group (§12; by default the folder name). It is still used. | | `different_pixel_sizes` | Warning | The physical pixel size (X by Y) differs between images, or the z-step differs between images with more than one depth slice. Sizes are compared in micrometres, rounded to four significant digits. | | `missing_pixel_size` | Warning | The pixel size (or the z-step of images with more than one depth slice) is known for some images and not for others. Nothing is reported when no image has it. | -| `different_plane_counts` | Warning | The number of channels (the `C` dimension, or the RGB samples `S` when there is no `C`), depth slices (`Z`), time points (`T`) or planes along any other dimension except X and Y differs between images. A missing dimension counts as 1. | +| `different_channel_counts` | Error | The number of channels differs between images: the `C` dimension, or the RGB samples `S` when there is no `C`; once the axis roles are confirmed, the length of the C axis (1 without one), which replaces the header's count. The images not having the most common number are listed, with each number's image count. Every image of an experiment must have the same channels: split the others into another experiment or exclude them. | +| `different_plane_counts` | Warning | The number of depth slices (`Z`), time points (`T`) or planes along any other dimension except X, Y, C and S differs between images. A missing dimension counts as 1. Channels are compared by `different_channel_counts`. | | `different_sizes` | Warning | The image width or height differs between images (TIFF and OME-Zarr). The other dimensions are compared by `different_plane_counts`. | | Rule | Detail | @@ -191,7 +207,7 @@ they never block processing, and each names the affected images or folders and w | Counts | Only images to analyse count: excluded images are left out, and a folder whose images are all excluded is not a biological repeat. A biological repeat with images directly in its folder has no technical-repeat folders (count 0). | | Balance table | Conditions as columns, repeat groups as rows (§12; by default the biological-repeat folder name, `single` when there are no biological-repeat folders); each cell holds the number of images and of technical-repeat folders. | | Pixel sizes | TIFF: OME `PhysicalSizeX`, `PhysicalSizeY`, `PhysicalSizeZ` with their units (micrometres when not given), or ImageJ resolution tags and `spacing` with the ImageJ unit (unknown without a unit). OME-Zarr: the level-0 scale (`coordinateTransformations` of the first dataset, times the image's own scale when given) with the axis units; a scale without a unit is unknown. Sizes are never guessed. | -| Panel | The notebook shows the balance table with the issues `mixed_dtypes`, `inconsistent_dimensions`, `missing_axes`, `inconsistent_mapping`, `different_sizes`, `different_plane_counts`, `channel_names_differ`, `different_pixel_sizes`, `missing_pixel_size`, `unbalanced_images` and `repeat_missing_conditions`. The command line `check` prints the balance table and every issue; `run` reports the same warnings. | +| Panel | The notebook shows the balance table with the issues `mixed_dtypes`, `inconsistent_dimensions`, `missing_axes`, `inconsistent_mapping`, `different_sizes`, `different_channel_counts`, `different_plane_counts`, `channel_names_differ`, `different_pixel_sizes`, `missing_pixel_size`, `unbalanced_images` and `repeat_missing_conditions`. The command line `check` prints the balance table and every issue; `run` reports the same warnings, and stops on `different_channel_counts`. | | Outputs | Unchanged: the checks write nothing, and `plane_histograms.csv` and `run_manifest.json` do not change. | ## 12. Experiment design (`experiment_design.csv`, schema version 1) diff --git a/docs/plan.md b/docs/plan.md index c237e00e..61e7b9f1 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -144,10 +144,11 @@ read every pixel (the axis preview reads one sample plane per distinct layout). **Pass 2, histograms:** after confirmation, read one 2D plane at a time; compute its exact finite extrema, pixel counts and histogram in a single read. Never load a full Z/C/T volume -when only a plane is needed. +when only a plane is needed. A row of the table is an XYC plane: its channels are read one +after another, and their histograms make the row (question 27). -Estimate each plane's working memory before loading it, including the decoded I/O unit and -temporary arrays. Use a configurable limit (default 1 GiB). A plane whose estimate is over +Estimate each plane's working memory before loading it, including the decoded I/O unit, +temporary arrays and the finished histograms of the other channels of its XYC plane. Use a configurable limit (default 1 GiB). A plane whose estimate is over the limit is read in blocks instead (question 15): rectangles of the pieces the image is stored in (TIFF strips or tiles, OME-Zarr chunks), as large as the limit allows, so the memory limit keeps its meaning as the most one step may use. Uncompressed TIFF pieces and @@ -163,15 +164,19 @@ convert source values. ## 6. Output CSV contract -Write one row per 2D plane to `plane_histograms.csv`. The default output folder is a +Write one row per XYC plane (every channel of an image at one Z and T position) to +`plane_histograms.csv`, schema version 3 (question 27). The default output folder is a sibling of the experiment folder named `_MorphoBricks_Preprocessing`; the user may choose another, but never one inside the experiment folder or a raw image folder. -Columns, in order: `position_id, image_file, z, c, t, condition, biological_repeat, -technical_repeat, source_relative_path, plane_id, dtype, bit_depth, pixel_min, pixel_max, -total_pixel_count, valid_pixel_count, binning_min, binning_max, bin_count, -schema_version`, then `histogram_bin_00000` onwards. Details in contract §5. -`position_id` is the image name without its extension (question 20). +Columns, in order: `position_id, image_file, z, t, condition, biological_repeat, +technical_repeat, source_relative_path, plane_id, dtype, bit_depth, channel_count, +total_pixel_count, bin_count, schema_version`, then for each channel `i` +`Intensity_MinIntensity_C, Intensity_MaxIntensity_C, +Histogram_ValidPixelCount_C, Histogram_BinningMin_C, Histogram_BinningMax_C` +and `Histogram_Count_C_00000` onwards. Details in contract §5 and the normalization +brick's contract §6.2. `position_id` is the image name without its extension (question +20). The wide CSV is for computational use; spreadsheet programs may not open all columns. The package provides `read_plane_histograms()`, which keeps labels as text. Alongside the CSV, @@ -329,6 +334,12 @@ API, CLI). The notebook and CLI only call the API. (TIFF and OME-Zarr, from the OME-NGFF scale), and steps 1 and 2 of the notebook embeddable by another brick. Outputs are unchanged. *Implemented, see questions 23 to 26.* +- **Channels across columns (follow-up P3b of the normalization plan):** + `plane_histograms.csv` schema version 3, written with the normalization brick's + Milestone 2b (pinned by commit): one row per XYC plane with each channel in its own + columns; its channels read one at a time (`planes.iter_xyc_planes`); images with + different numbers of channels are an error before any image is read; earlier tables are + recomputed on the next run. *Implemented, see question 27.* ## 11. Tests and acceptance criteria @@ -350,6 +361,9 @@ Acceptance checks: - Reruns are idempotent and only new or changed files are read. - Planes read in blocks give the same rows (and OME-Zarr copies) as planes read whole, for every pixel type and for constant, non-finite and empty planes. +- Each channel of a schema-version-3 row has the histogram that version 2 wrote in that + channel's row, read whole and in blocks (committed version-2 fixture); step 4's + statistics are those of the version-2 rows. - The CSV round-trips through `read_plane_histograms()` with labels unchanged, and the notebook shows understandable summaries and validation messages. @@ -643,3 +657,31 @@ Raised during review; to be decided before the milestones that need them: and the card is shown again when the check finds that folder inside the experiment, so the box to fix it is never hidden). The defaults give the pages as before, and this notebook calls them without arguments; the widgets, the checks and the gating are the same either way.* +27. How does this brick follow the channels moving into columns (normalization plan + N41–N44, its Milestone 2b, follow-up P3b)? + *Decided (October 2026): `plane_histograms.csv` is written in schema version 3 with + the normalization brick's writer and `PlaneRecord`: one row per XYC plane (an image at + one Z and T position), each channel's extrema, pixel counts and bins in its own + CellProfiler-style columns (`Intensity_MinIntensity_C0`, `Histogram_Count_C1_00042`, + …), `plane_id` `::z=…,t=…`, rows sorted by path, z and t. The brick is pinned by + commit (`0e53e67`, the merge of its Milestone 2b), not by tag, because the owner wants + no tags before its v0.2.0. Asked of the owner: the channels of a plane are read and + binned one at a time (`iter_xyc_planes`), so only one channel's pixels are in memory + next to the finished histograms of the others; the memory estimate adds those bin + counts (8 bytes per bin and channel), and one decision, whole or in blocks, covers + every channel of every plane of an image. Each channel's histogram is the one version + 2 wrote, whole and in blocks, which a committed version-2 fixture checks. Images with + different numbers of channels are the error `different_channel_counts` (N43), before + any image is read: in step 1 from the headers (the C axis, or the RGB samples without + one) and in the axis review with the confirmed roles, which replace the header's count + (a confirmed missing C axis is one channel); the notebook shows it in step 1's + "Dataset consistency" panel and step 2's messages, the command line in `check` and + `run`. `different_plane_counts` compares depth, time and other planes only. A table of + schema version 1 or 2 is recomputed completely on the next run; the manifest stays + version 1, records schema version 3, and a file's `plane_count` counts its XYC planes. + Step 4 reads the new layout and, as before, pools every channel of a group together, + so its table (whose "planes" still counts channel planes) and plot do not change. The + warning `no_finite_pixels` counts the planes with a channel without finite pixels. The + deprecated re-exports of the moved names (question 19) keep pointing at the brick's + current objects, now with its Milestone 2b signatures; names it added later are not + re-exported. `experiment_design.csv` and the OME-Zarr copies do not change.* diff --git a/docs/user_guide.md b/docs/user_guide.md index 93e1b56b..360d5f91 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -102,11 +102,14 @@ To change a label, rename the folder and check again. separately or exclude it. - **Pixel types:** 8-, 16-, 32- and 64-bit signed or unsigned integers, 32- and 64-bit floats. All images of an experiment must have the same pixel type. -- **Dimensions:** every image is split into 2D planes along Z (depth), C (channel) and T - (time). All TIFF images must have their dimensions in the same order; the lengths - (number of slices, time points, size) may differ. OME-Zarr images may store them in - another order: they are matched by role, so z = 2, c = 1 means the same plane in every - image. +- **Dimensions:** every image is split into planes along Z (depth) and T (time); a plane + holds every channel (C) of the image at that Z and T, one row of the table. All TIFF + images must have their dimensions in the same order; the number of slices, time points + and the size may differ. OME-Zarr images may store them in another order: they are + matched by role, so z = 2, c = 1 means the same slice and channel in every image. +- **Channels:** all images of an experiment must have the same number of channels (an + image without a channel dimension has one). Split images with other channels into a + separate experiment, or exclude them. - **RGB images:** the three colour samples are treated as channels (C). - **Pyramidal images:** only the full-resolution level is read. - **Not supported yet:** files with several positions or scenes (several image series), @@ -165,7 +168,8 @@ results or comparing another grouping. warnings** to see the checks that compare the images with each other: pixel type, dimensions, width and height, numbers of channels, slices and time points, channel names and physical pixel sizes (see §8). Warnings never stop you; they say what to - check. + check. Images with different numbers of channels, pixel types or dimension orders are + errors: fix them before going on. - **Messages** lists the other problems (folders, files that are not images, unreadable images), with the warning count visible in orange and the error count in red. Expand **Show warnings and errors** to see each problem, the affected files and what @@ -177,7 +181,8 @@ results or comparing another grouping. - A line under each dropdown says what the choice means, for example *3 depth slices, numbered z = 0 to 2*. A blue line says how many planes each image will be split into. - Click **Show preview** and move the sliders to look at different slices, channels and - time points of any image. Each picture should be a normal image of your sample; + time points of any image. If the roles you choose give the images different numbers of + channels, a message says so and step 3 stays locked. Each picture should be a normal image of your sample; stripes or a smear mean X or Y is on the wrong dimension. Brightness is stretched for display only. - If the preview looks right, press **Yes, the preview looks right**, then click @@ -240,7 +245,8 @@ results or comparing another grouping. are calculated and the plot is drawn. The button says **Calculating…** and stays disabled until the calculation finishes or an error is shown; you can then try again. - The table lists pooled statistics per group: minimum, maximum, mean, standard - deviation, median and percentiles. + deviation, median and percentiles. Every channel of a group is pooled together, as + before; its **planes** column counts each channel of a plane as one plane. - The plot compares the groups' intensity distributions: curves further right are brighter. @@ -298,13 +304,18 @@ Both commands exit with status 1 when an error blocks processing. The default output folder is next to the experiment folder: `/data/exp1` → `/data/exp1_MorphoBricks_Preprocessing`. -- **`plane_histograms.csv`:** one row per plane with the position (`position_id`, the file - name without its extension), the file name, the zero-based plane - coordinates `z`, `c`, `t` (0 when the image has no such dimension), the three - labels, the pixel type, the exact minimum and maximum, pixel counts and the histogram. - The table is very wide (one column per bin), so spreadsheet programs may not open it - completely; read it in Python with `read_plane_histograms()`. The columns are defined in - the [normalization brick's contract](https://github.com/CellMigrationLab/morphobricks-normalization/blob/main/docs/contract.md) §6.2. +- **`plane_histograms.csv`:** one row per plane (every channel of an image at one depth + slice and time point) with the position (`position_id`, the file name without its + extension), the file name, the zero-based plane coordinates `z` and `t` (0 when the + image has no such dimension), the three labels, the pixel type and the number of + channels; then, for each channel, its exact minimum and maximum, pixel counts and + histogram, in columns ending in the channel number (`Intensity_MinIntensity_C0`, + `Histogram_Count_C1_00042`, …). The table is very wide (one column per bin and channel), + so spreadsheet programs may not open it completely; read it in Python with + `read_plane_histograms()`. The columns are defined in the [normalization brick's + contract](https://github.com/CellMigrationLab/morphobricks-normalization/blob/main/docs/contract.md) + §6.2 (schema version 3). A table written by an earlier version, with one row per channel, + is recomputed on the next run; later bricks ask you to run this brick again on it. - **`experiment_design.csv`:** one row per biological-repeat folder of each condition: its repeat group, whether its condition is a control, its image and technical-repeat counts, and whether and when the repeat group was confirmed. Written in step 1 and by every run. @@ -344,7 +355,8 @@ Run the notebook or `run` again after adding, changing or removing images: - rows of deleted or renamed images are removed (a renamed image counts as removed and added); - everything is recomputed when the bin count, the axis roles or the pixel type changed, - when the table was edited by hand, or when you ask for it. + when the table was edited by hand or written by an earlier version of the brick, or when + you ask for it. To leave out images, for example a damaged file, list them under *Exclude images* (one per line) or pass `--exclude`. Excluded images are not read, are listed in the messages and the @@ -372,6 +384,7 @@ continues. | `multiple_series` | Error | A file or OME-Zarr folder holds several images (positions, scenes, series, wells or fields). Export them separately, or exclude it. | | `unsupported_dtype` | Error | Unsupported pixel type (e.g. float16). Convert, or exclude it. | | `mixed_dtypes` | Error | Images have different pixel types. Process them separately or convert. | +| `different_channel_counts` | Error | Images have different numbers of channels (an image without a channel dimension has one). Every image of an experiment must have the same channels: split the others into a separate experiment, or exclude them. If a dimension got the channel role by mistake, correct the axis roles. | | `inconsistent_dimensions`, `inconsistent_mapping` | Error | Images have their dimensions in a different order. Export them the same way or process separately. | | `missing_axes` | Error | Some images lack a dimension. Confirm they have one plane along it (step 2), or export them again. | | `ambiguous_missing_axes` | Error | It is unclear which dimension an image lacks. Export it again. | @@ -386,7 +399,7 @@ continues. | `non_tiff_file`, `empty_folder`, `tiff_outside_leaf` | Warning | Files or folders that are skipped. | | `planes_read_in_blocks` | Warning | Planes too large for the memory limit are read in blocks. The results are the same; nothing to do. Raise the limit to read them whole, which is faster. | | `different_sizes` | Warning | Images have different widths or heights. Usually expected; otherwise check the camera settings. | -| `different_plane_counts` | Warning | Images have different numbers of channels, depth slices, time points or other planes. Images with more planes weigh more in pooled statistics. Check the acquisition if this is not expected. | +| `different_plane_counts` | Warning | Images have different numbers of depth slices, time points or other planes. Images with more planes weigh more in pooled statistics. Check the acquisition if this is not expected. | | `different_pixel_sizes` | Warning | The physical pixel size or z-step in the metadata differs between images (for example different objectives). Check the microscope settings if this is not intended. | | `missing_pixel_size` | Warning | Some images give their pixel size in the metadata and others do not, so it cannot be checked. Export them again with metadata if needed. | | `unbalanced_images` | Warning | A condition and biological repeat has fewer than half or more than twice the median number of images. Check for missing, misplaced or duplicated images. | @@ -395,7 +408,7 @@ continues. | `repeat_group_conflict` | Error | A repeat group holds two folders of the same condition. Move one into another group. | | `channel_names_differ` | Warning | Channel names differ between images at the same position. Check the channel order. | | `non_finite_pixels` | Warning | NaN or infinite pixels are left out of the histograms and counted. | -| `no_finite_pixels` | Warning | Some planes have no finite pixels; they are left out of group statistics. | +| `no_finite_pixels` | Warning | Some planes have a channel without finite pixels; those channels are left out of group statistics. | | `excluded_images`, `excluded_path_not_found` | Warning | Images excluded as requested, or an excluded path that matches no image. | | `zarr_not_installed` | Error | OME-Zarr images, or converting TIFF images, need the `zarr` extra. Install it, exclude the OME-Zarr images, or read the TIFF images directly. | diff --git a/notebooks/MorphoBricks_Preprocessing.ipynb b/notebooks/MorphoBricks_Preprocessing.ipynb index 1e7fa315..98b21eae 100644 --- a/notebooks/MorphoBricks_Preprocessing.ipynb +++ b/notebooks/MorphoBricks_Preprocessing.ipynb @@ -10,7 +10,8 @@ "\n", "This notebook prepares an experiment for the other MorphoBricks bricks. It measures the\n", "brightness of every 2D plane of your images (TIFF or OME-Zarr) and saves one **intensity histogram per\n", - "plane** in a table, **plane_histograms.csv**. Later bricks use this table to normalise images per\n", + "channel** of every plane in a table, **plane_histograms.csv**: one row per plane, the channels side by\n", + "side. Later bricks use this table to normalise images per\n", "condition, repeat or image without reading the pixels again.\n", "\n", "### What the notebook does\n", @@ -233,7 +234,7 @@ " biological repeat that a condition does not have. Below it, **Show consistency warnings**\n", " lists differences between the images: pixel type, dimensions, sizes, numbers of channels,\n", " slices or time points, channel names and pixel sizes. Warnings never stop you; each says\n", - " what to check.\n", + " what to check. Different numbers of channels are an error: every image needs the same channels.\n", "\n", "> 💡 **Tip:** to change a label, rename the folder and click **Check folder** again.\n", ">\n", @@ -432,6 +433,10 @@ "\n", "**\"No role is chosen for the axis\"**: a dimension with several planes needs Z, C or T in step 2.\n", "\n", + "**\"The images have different numbers of channels\"**: each row of the table holds every channel of\n", + "a plane, so all images need the same channels. Process the others as a separate experiment, or\n", + "exclude them; if a dimension got the role C by mistake, correct it in step 2.\n", + "\n", "**\"… cannot be read within the memory limit, not even in blocks\"**: untick *Use the\n", "recommended settings* in step 3 and raise the memory limit, if the computer has enough\n", "memory, or save the images again as tiled TIFF files or OME-Zarr. Planes that are only too\n", diff --git a/pyproject.toml b/pyproject.toml index 0d381d5d..11ac2fac 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -23,7 +23,10 @@ dependencies = [ "numpy>=1.24", "tifffile>=2023.7.10", # Per-plane histograms, the CSV table, pooling and normalization (moved there in P1). - "morphobricks-normalization @ git+https://github.com/CellMigrationLab/morphobricks-normalization.git@v0.1.0", + # Pinned by commit, not by tag: the owner wants no tags until the brick's v0.2.0 + # release. 0e53e67 is the merge of its Milestone 2b (PR #8), which writes + # plane_histograms.csv schema version 3 with the channels in columns (follow-up P3b). + "morphobricks-normalization @ git+https://github.com/CellMigrationLab/morphobricks-normalization.git@0e53e6762617e4297df0593c6977fdfb7b405522", ] [project.optional-dependencies] diff --git a/src/morphobricks_preprocessing/__init__.py b/src/morphobricks_preprocessing/__init__.py index 50d5dda4..5fe27b65 100644 --- a/src/morphobricks_preprocessing/__init__.py +++ b/src/morphobricks_preprocessing/__init__.py @@ -1,7 +1,8 @@ """MorphoBricks preprocessing brick. -Builds one intensity histogram per 2D plane and stores them, with each plane's -experimental labels, in a CSV table. Later bricks pool these histograms to get +Builds one intensity histogram per channel of every plane and stores them, with +each plane's experimental labels, in a CSV table: one row per plane (an image at +one Z and T position), the channels in columns. Later bricks pool these histograms to get group statistics and normalization parameters without rereading the images. """ @@ -85,8 +86,10 @@ PlaneLayout, PlaneMemoryError, PlaneReadError, + XycPlane, check_planes, iter_planes, + iter_xyc_planes, plane_layout, read_plane, read_sample_plane, @@ -184,6 +187,7 @@ def __dir__() -> list[str]: "TiffMetadata", "UnsupportedDtypeError", "ValidationIssue", + "XycPlane", "ZarrUnavailableError", "axis_length_ranges", "balance_table", @@ -214,6 +218,7 @@ def __dir__() -> list[str]: "inspect_tiff", "is_zarr_store", "iter_planes", + "iter_xyc_planes", "load_design", "make_plane_id", "min_max", diff --git a/src/morphobricks_preprocessing/axes.py b/src/morphobricks_preprocessing/axes.py index 0b82b123..61bd028f 100644 --- a/src/morphobricks_preprocessing/axes.py +++ b/src/morphobricks_preprocessing/axes.py @@ -18,7 +18,7 @@ from dataclasses import dataclass, field from types import MappingProxyType -from .metadata import MetadataReport, TiffMetadata +from .metadata import MetadataReport, TiffMetadata, channel_count_issues from .validation import ValidationIssue, error, has_errors, plural ROLES = ("X", "Y", "Z", "C", "T") @@ -54,7 +54,9 @@ } # Metadata issues that the axis review checks again against the confirmed mapping. -_SUPERSEDED_METADATA_CODES = frozenset({"missing_axes", "inconsistent_dimensions"}) +_SUPERSEDED_METADATA_CODES = frozenset( + {"missing_axes", "inconsistent_dimensions", "different_channel_counts"} +) @dataclass(frozen=True) @@ -204,9 +206,10 @@ def ok(self) -> bool: def combined_issues(self, metadata: MetadataReport) -> tuple[ValidationIssue, ...]: """Merge the metadata issues with this review's, errors first. - The review checks the arrangement of dimensions again against the - confirmed mapping, so the metadata's ``missing_axes`` and - ``inconsistent_dimensions`` issues are replaced by the review's own. + The review checks the arrangement of dimensions and the numbers of + channels again against the confirmed mapping, so the metadata's + ``missing_axes``, ``inconsistent_dimensions`` and + ``different_channel_counts`` issues are replaced by the review's own. """ kept = [i for i in metadata.issues if i.code not in _SUPERSEDED_METADATA_CODES] merged = [*kept, *self.issues] @@ -476,8 +479,9 @@ def review_axes( Per-file mappings for the files that fit and the issues found: those from :func:`check_mapping`, files that lack dimensions without confirmation (``missing_axes``) or in an ambiguous way - (``ambiguous_missing_axes``), and files the mapping does not fit - (``inconsistent_mapping``). + (``ambiguous_missing_axes``), files the mapping does not fit + (``inconsistent_mapping``), and files with different numbers of channels + along the C axis (``different_channel_counts``; 1 without one). """ if isinstance(files, MetadataReport): files = files.files @@ -567,9 +571,24 @@ def review_axes( "differently arranged images as a separate experiment.", ) ) + issues.extend( + channel_count_issues( + { + f.relative_path: _channel_count(f, file_mappings[f.relative_path]) + for f in readable + if f.relative_path in file_mappings + } + ) + ) return AxisReview(mapping=mapping, file_mappings=file_mappings, issues=tuple(issues)) +def _channel_count(f: TiffMetadata, mapping: AxisMapping) -> int: + """Return the number of channels of ``f`` under its own mapping; 1 without a C axis.""" + c = mapping.index_of("C") + return 1 if c is None else f.shape[c] + + def _embeddings(short: str, long: str) -> list[tuple[int, ...]]: """Every way ``short`` is ``long`` with some characters left out (as kept indices).""" results: list[tuple[int, ...]] = [] diff --git a/src/morphobricks_preprocessing/dataset.py b/src/morphobricks_preprocessing/dataset.py index 68292e89..d03d2a91 100644 --- a/src/morphobricks_preprocessing/dataset.py +++ b/src/morphobricks_preprocessing/dataset.py @@ -2,8 +2,8 @@ The folder check (pass 1) gathers every consistency warning in one place (``docs/contract.md`` §11, normalization plan N37). The image headers are -compared in :mod:`.metadata` (pixel types, axes, sizes, plane counts, channel -names, pixel sizes); this module looks at the folders: how many images each +compared in :mod:`.metadata` (pixel types, axes, sizes, channel and plane +counts, channel names, pixel sizes); this module looks at the folders: how many images each condition and biological repeat has, and whether every biological repeat is found in every condition. Biological repeats are matched across conditions by their repeat group, by default their folder name. Warnings never block @@ -28,6 +28,7 @@ "missing_axes", "inconsistent_mapping", "different_sizes", + "different_channel_counts", "different_plane_counts", "channel_names_differ", "different_pixel_sizes", diff --git a/src/morphobricks_preprocessing/metadata.py b/src/morphobricks_preprocessing/metadata.py index 8ae16c8a..f1d2d0d7 100644 --- a/src/morphobricks_preprocessing/metadata.py +++ b/src/morphobricks_preprocessing/metadata.py @@ -4,10 +4,10 @@ candidate image this records whether it opens, its shape, pixel type, axis labels and series/page counts. Across the experiment it checks the rules that must hold before any pixel is read (``docs/contract.md`` §3 and §8): one -supported pixel type, and the same arrangement of dimensions in every file. -Images that differ in width or height, in their numbers of channels, depth -slices or time points, or in their physical pixel sizes are allowed with a -warning (contract §11). +supported pixel type, the same arrangement of dimensions and the same number +of channels in every file. Images that differ in width or height, in their +numbers of depth slices or time points, or in their physical pixel sizes are +allowed with a warning (contract §11). Axis roles are not assigned here; the axis labels are the ones ``tifffile`` reports from the file's metadata, used to prefill the axis review. @@ -17,7 +17,7 @@ import logging import os -from collections.abc import Callable, Hashable, Iterable +from collections.abc import Callable, Hashable, Iterable, Mapping from dataclasses import dataclass from pathlib import Path from xml.etree import ElementTree @@ -277,14 +277,16 @@ def check_metadata(files: Iterable[TiffMetadata]) -> list[ValidationIssue]: Reports, as errors: images that do not open (``unreadable_tiff``, ``unreadable_ome_zarr``, or ``zarr_not_installed`` for OME-Zarr images without the ``zarr`` package), files or stores with more than one image, - unsupported pixel types, more than one pixel type, and TIFF files whose - dimensions are arranged differently. OME-Zarr images declare their axes, + unsupported pixel types, more than one pixel type, TIFF files whose + dimensions are arranged differently, and images with different numbers of + channels (``different_channel_counts``; the C axis, or the RGB samples + without one, until the axis roles are confirmed). OME-Zarr images declare their axes, so their arrangement is checked by role during the axis review instead. Reports, as warnings: images of different widths or heights (``different_sizes``), channel names that differ at the same channel - (``channel_names_differ``), different numbers of channels, depth slices, - time points or planes along another dimension (``different_plane_counts``), + (``channel_names_differ``), different numbers of depth slices, time points + or planes along another dimension (``different_plane_counts``), and physical pixel sizes that differ (``different_pixel_sizes``) or are known for only some images (``missing_pixel_size``). @@ -396,6 +398,7 @@ def check_metadata(files: Iterable[TiffMetadata]) -> list[ValidationIssue]: issues.extend(_check_dimensions([f for f in readable if not f.is_ome_zarr])) issues.extend(_check_image_sizes(readable)) issues.extend(_check_channel_names(readable)) + issues.extend(_check_channel_counts(readable)) issues.extend(_check_plane_counts(readable)) issues.extend(_check_pixel_sizes(readable)) return issues @@ -508,21 +511,70 @@ def _check_image_sizes(files: list[TiffMetadata]) -> list[ValidationIssue]: # What the planes along each dimension are called in messages. -_PLANE_NAMES = {"C": "channels", "Z": "depth slices (Z)", "T": "time points"} -# Dimensions that do not hold planes: the image itself, and RGB samples, which are -# counted as channels when the image has no channel dimension. +_PLANE_NAMES = {"Z": "depth slices (Z)", "T": "time points"} +# Dimensions that do not hold planes: the image itself, and channels (or RGB samples), +# which are compared by _check_channel_counts. _NOT_PLANES = frozenset("XYSC") +def header_channel_count(f: TiffMetadata) -> int: + """Return the channel count the header suggests: the C axis, else the RGB samples.""" + return f.length("C") if "C" in f.axes else f.length("S") + + +def channel_count_issues(counts: Mapping[str, int]) -> list[ValidationIssue]: + """Report images with different numbers of channels (normalization plan N43). + + Each row of ``plane_histograms.csv`` holds every channel of a plane, so all + images of an experiment must have the same number of channels. + + Parameters + ---------- + counts + Number of channels of each image, by relative path. + + Returns + ------- + list of ValidationIssue + The error ``different_channel_counts`` naming the images not having the + most common number, or nothing. + """ + groups: dict[int, list[str]] = {} + for path, count in sorted(counts.items()): + groups.setdefault(count, []).append(path) + if len(groups) < 2: + return [] + ordered = sorted(groups.items(), key=lambda item: (-len(item[1]), item[0])) + summary = ", ".join( + f"{plural(count, 'channel')} ({plural(len(paths), 'image')})" for count, paths in ordered + ) + return [ + error( + "different_channel_counts", + f"The images have different numbers of channels: {summary}. Each row of the " + "histogram table holds every channel of a plane, so every image of an experiment " + "must have the same channels. The images listed are those without the most " + "common number of channels.", + sorted(path for _, paths in ordered[1:] for path in paths), + "Split the images with other channels into a separate experiment, or exclude " + "them. If a channel axis was assigned the wrong role, correct the axis roles.", + ) + ] + + +def _check_channel_counts(files: list[TiffMetadata]) -> list[ValidationIssue]: + return channel_count_issues({f.relative_path: header_channel_count(f) for f in files}) + + def _plane_counts(f: TiffMetadata) -> dict[str, int]: - counts = {"C": f.length("C") if "C" in f.axes else f.length("S"), "Z": 1, "T": 1} + counts = {"Z": 1, "T": 1} counts.update({axis: f.length(axis) for axis in f.axes if axis not in _NOT_PLANES}) return counts def _check_plane_counts(files: list[TiffMetadata]) -> list[ValidationIssue]: counts = {f.relative_path: _plane_counts(f) for f in files} - axes = sorted({axis for c in counts.values() for axis in c}, key=lambda a: ("CZT".find(a), a)) + axes = sorted({axis for c in counts.values() for axis in c}, key=lambda a: ("ZT".find(a), a)) descriptions: list[str] = [] differing: list[str] = [] for axis in axes: @@ -545,7 +597,7 @@ def _check_plane_counts(files: list[TiffMetadata]) -> list[ValidationIssue]: sorted(set(differing)), "No action needed if this is expected, for example time series of different " "lengths. Otherwise check that these images were acquired with the same " - "channels, depth range and time points.", + "depth range and time points.", ) ] diff --git a/src/morphobricks_preprocessing/notebook_ui.py b/src/morphobricks_preprocessing/notebook_ui.py index 25d95744..937a9c92 100644 --- a/src/morphobricks_preprocessing/notebook_ui.py +++ b/src/morphobricks_preprocessing/notebook_ui.py @@ -83,7 +83,13 @@ plane_layout, read_plane, ) -from .report import LEVEL_NAMES, group_summaries, group_table, issue_counts +from .report import ( + LEVEL_NAMES, + channel_histograms, + group_summaries, + group_table, + issue_counts, +) from .run import Preflight, RunResult, preflight, run_experiment from .validation import ValidationIssue, has_errors, plural from .zarr_export import ( @@ -1852,7 +1858,8 @@ def _plan_html(self, review: AxisReview | None = None) -> str: f" ({len(self.preflight.excluded):,} excluded)." if self.preflight.excluded else "." ), f"Output folder: {html.escape(str(output))}", - f"Files written: {OUTPUT_FILENAME} (one row per plane) and " + f"Files written: {OUTPUT_FILENAME} (one row per plane, its channels in " + "columns) and " f"{MANIFEST_FILENAME} (what was read, with which settings).", ] files = self.preflight.metadata.files @@ -2181,7 +2188,7 @@ def _calculate_statistics(self, path: Path, percentiles: Sequence[float]) -> Non summaries = group_summaries(records, level, percentiles) groups = group_records(records, level) images = {r.source_relative_path for r in records} - first = records[0].histogram if records else None + first = records[0].channel_histogram(0) if records else None self.stats_tiles.value = ui.tiles( [ ("🧪", f"{len(summaries):,}", f"{LEVEL_NAMES[level]} groups"), @@ -2439,7 +2446,7 @@ def distribution_png(groups: dict[tuple[str, ...], list]) -> bytes | None: curves = [] for key, members in list(groups.items())[:_MAX_PLOTTED_GROUPS]: try: - pooled = PooledHistogram(members) + pooled = PooledHistogram(channel_histograms(members)) except NoValidPixelsError: continue cumulative = np.cumsum(pooled.counts) / pooled.pixel_count diff --git a/src/morphobricks_preprocessing/planes.py b/src/morphobricks_preprocessing/planes.py index 5076da39..614d9eac 100644 --- a/src/morphobricks_preprocessing/planes.py +++ b/src/morphobricks_preprocessing/planes.py @@ -13,9 +13,19 @@ decodes only the chunks the plane touches. A chunk that spans several planes is decoded again for each of them, which costs time but never memory. +A row of the histogram table is an XYC plane: every channel of an image at one +Z and T position (normalization contract §6.2). :func:`iter_xyc_planes` yields +them; their channels are read and binned one after another, so only one +channel's pixels are in memory at a time, next to the histograms of the channels +already done. A page that holds several channels is decoded once for all of +them; a page that holds planes of several Z or T positions is decoded again for +each XYC plane when the channels are stored in different pages. + Before anything is decoded, each plane's working memory is estimated from the header: the decoded page (for OME-Zarr, every chunk the plane touches), its -encoded bytes, a copy of the plane and the temporary arrays of the histogram. +encoded bytes, a copy of the plane, the temporary arrays of the histogram and +the finished histograms of every channel of the XYC plane. One decision, whole +or in blocks, applies to every channel of every plane of a file. Planes above the memory limit (default 1 GiB) are read in blocks instead (``docs/plan.md`` §5): rectangles made of the pieces the image is stored in @@ -43,6 +53,7 @@ import numpy as np import tifffile +from morphobricks_normalization.dtypes import default_bin_count from morphobricks_normalization.histogram import ( PlaneHistogram, compute_block_histogram, @@ -191,6 +202,42 @@ def histogram(self, bin_count: int | None = None) -> PlaneHistogram: ) +@dataclass(frozen=True) +class XycPlane: + """Every channel of one image at one Z and T position: one row of the table. + + The channels are read on request, one at a time, from the open file. Use + the plane before asking :func:`iter_xyc_planes` for the next one. + + Attributes + ---------- + z, t + Zero-based coordinates; None when the experiment mapping has no such axis. + channel_count + Number of channels; 1 for an image without a channel axis. + read + Called as ``read(c)`` to read channel ``c`` as a :class:`Plane`. + """ + + z: int | None + t: int | None + channel_count: int + read: Callable[[int], Plane] = field(repr=False, compare=False) + + def channel(self, c: int) -> Plane: + """Read channel ``c`` (zero-based).""" + if not 0 <= c < self.channel_count: + raise IndexError(f"The plane has {self.channel_count} channel(s); got c={c}.") + return self.read(c) + + def histograms(self, bin_count: int | None = None) -> tuple[PlaneHistogram, ...]: + """Compute the histogram of every channel, reading one channel at a time. + + Each histogram is identical to that of the channel's plane read whole. + """ + return tuple(self.channel(c).histogram(bin_count) for c in range(self.channel_count)) + + @dataclass(frozen=True) class _Blocking: """How a plane can be split into blocks, in the order of its stored axes. @@ -226,7 +273,9 @@ class PlaneLayout: plane_shape (Y, X) shape of each plane. plane_count - Number of planes in the file. + Number of 2D planes in the file. + channel_count + Number of channels of each XYC plane; 1 without a channel axis. page_shape Shape of the unit decoded at once: one TIFF page, as a part of ``shape``, or the chunk shape of an OME-Zarr array. @@ -238,7 +287,8 @@ class PlaneLayout: for OME-Zarr, whose compressed chunk sizes are not known in advance. memory_estimate Estimated working memory to read one whole plane and compute its - histogram, in bytes. + histogram, holding the finished histograms of every channel of its XYC + plane, in bytes. memory_limit The memory limit the layout was worked out for, in bytes. block_shape @@ -259,6 +309,7 @@ class PlaneLayout: encoded_nbytes: int memory_estimate: int memory_limit: int = DEFAULT_MEMORY_LIMIT + channel_count: int = 1 block_shape: tuple[int, int] | None = None block_memory_estimate: int | None = None blocking: _Blocking | None = field(default=None, repr=False, compare=False) @@ -367,18 +418,62 @@ def iter_planes( series = tif.series[0] layout = _layout(series, mapping, path, bin_count, memory_limit) _check_limit(layout, path) - if layout.in_blocks: - for position in _positions(layout, mapping): - yield _tiff_block_plane(series, layout, mapping, position, path) - return - current, page = -1, None + read = _tiff_reader(series, layout, mapping, path) for position in _positions(layout, mapping): - page_index = _page_index(layout, position) - if page_index != current: - page = None # let the previous page go before decoding the next - page = _read_page(series, page_index, layout, path) - current = page_index - yield _plane(page, layout, mapping, position) + yield read(position) + + +def iter_xyc_planes( + path: str | os.PathLike, + mapping: AxisMapping, + *, + memory_limit: int = DEFAULT_MEMORY_LIMIT, + bin_count: int | None = None, +) -> Iterator[XycPlane]: + """Yield every XYC plane of a file: all channels at each Z and T position. + + Planes come in the file's storage order of the axes other than C. Their + channels are read on request, one at a time (see :meth:`XycPlane.channel`), + with the same memory check and block reading as :func:`iter_planes`; use + each plane before asking for the next one, while the file is still open. + + Parameters + ---------- + path + The TIFF file or OME-Zarr folder. + mapping + The axis mapping for this file (see :attr:`AxisReview.file_mappings`). + memory_limit + Largest working memory allowed for one plane or block, in bytes. + bin_count + Histogram bin count, for the memory estimate; the dtype's default if None. + + Yields + ------ + XycPlane + + Raises + ------ + PlaneMemoryError + If a plane cannot be read within ``memory_limit``, not even in blocks. + PlaneReadError + If the file cannot be read plane by plane under ``mapping``. + """ + if is_zarr_store(path): + image = _open_zarr(path) + layout = _zarr_layout(image, mapping, path, bin_count, memory_limit) + _check_limit(layout, path) + yield from _xyc_planes( + layout, + mapping, + lambda position: _zarr_plane(image, layout, mapping, position, path), + ) + return + with _open(path) as tif: + series = tif.series[0] + layout = _layout(series, mapping, path, bin_count, memory_limit) + _check_limit(layout, path) + yield from _xyc_planes(layout, mapping, _tiff_reader(series, layout, mapping, path)) def read_plane( @@ -619,11 +714,13 @@ def _layout( plane_pixels = shape[y] * shape[x] page_nbytes = page_size * dtype.itemsize encoded_nbytes = int(sum(keyframe.databytecounts)) + channels = _channel_count(shape, mapping) memory = ( page_nbytes + encoded_nbytes + plane_pixels * dtype.itemsize + estimate_histogram_memory(plane_pixels, dtype, bin_count) + + _finished_histograms_nbytes(dtype, bin_count, channels) ) layout = PlaneLayout( shape=shape, @@ -635,12 +732,23 @@ def _layout( encoded_nbytes=encoded_nbytes, memory_estimate=memory, memory_limit=memory_limit, + channel_count=channels, ) if memory <= memory_limit: return layout return _with_blocks(layout, _tiff_blocking(keyframe, shape, split, y, x), bin_count) +def _channel_count(shape: tuple[int, ...], mapping: AxisMapping) -> int: + c = mapping.index_of("C") + return 1 if c is None else shape[c] + + +def _finished_histograms_nbytes(dtype: np.dtype, bin_count: int | None, channels: int) -> int: + """Memory of the bin counts of every channel of an XYC plane, kept until its row is made.""" + return channels * 8 * (default_bin_count(dtype) if bin_count is None else bin_count) + + def _check_limit(layout: PlaneLayout, path) -> None: if not layout.fits: raise PlaneMemoryError( @@ -652,14 +760,58 @@ def _check_limit(layout: PlaneLayout, path) -> None: ) -def _positions(layout: PlaneLayout, mapping: AxisMapping) -> Iterator[dict[int, int]]: - """Index per non-XY dimension of every plane, in storage order.""" - y, x = mapping.index_of("Y"), mapping.index_of("X") - others = [d for d in range(len(layout.shape)) if d not in (y, x)] +def _positions( + layout: PlaneLayout, mapping: AxisMapping, roles: str = "XY" +) -> Iterator[dict[int, int]]: + """Index per dimension other than those of ``roles``, for every plane in storage order.""" + skipped = {mapping.index_of(role) for role in roles} + others = [d for d in range(len(layout.shape)) if d not in skipped] for index in np.ndindex(*(layout.shape[d] for d in others)): yield dict(zip(others, index, strict=True)) +def _xyc_planes( + layout: PlaneLayout, mapping: AxisMapping, read: Callable[[dict[int, int]], Plane] +) -> Iterator[XycPlane]: + """Yield the XYC planes of a file, each reading its channels with ``read(position)``.""" + c = mapping.index_of("C") + + def at(position: dict[int, int], channel: int) -> dict[int, int]: + return position if c is None else {**position, c: channel} + + for position in _positions(layout, mapping, "XYC"): + coordinates = _coordinates(mapping, at(position, 0)) + yield XycPlane( + coordinates["z"], + coordinates["t"], + layout.channel_count, + lambda channel, p=position: read(at(p, channel)), + ) + + +def _tiff_reader( + series: tifffile.TiffPageSeries, layout: PlaneLayout, mapping: AxisMapping, path +) -> Callable[[dict[int, int]], Plane]: + """Return a function reading the plane at a position, keeping the last page decoded. + + Consecutive planes of one page decode it once; the previous page is let go + before the next is decoded, so at most one page is in memory. + """ + if layout.in_blocks: + return lambda position: _tiff_block_plane(series, layout, mapping, position, path) + current: dict[str, object] = {"index": -1, "page": None} + + def read(position: dict[int, int]) -> Plane: + index = _page_index(layout, position) + if index != current["index"]: + current["page"] = None # let the previous page go before decoding the next + current["page"] = _read_page(series, index, layout, path) + current["index"] = index + return _plane(current["page"], layout, mapping, position) + + return read + + def _read_one( series: tifffile.TiffPageSeries, layout: PlaneLayout, @@ -823,8 +975,10 @@ def _block_memory( ) -> int: """Estimated working memory to read one block and add it to the histogram.""" pixels = rows * cols - memory = pixels * layout.dtype.itemsize + estimate_histogram_memory( - pixels, layout.dtype, bin_count + memory = ( + pixels * layout.dtype.itemsize + + estimate_histogram_memory(pixels, layout.dtype, bin_count) + + _finished_histograms_nbytes(layout.dtype, bin_count, layout.channel_count) ) unit_rows, unit_cols = blocking.unit if blocking.kind == "zarr": @@ -1109,10 +1263,12 @@ def _zarr_layout( chunk_nbytes = math.prod(chunks) * dtype.itemsize page_nbytes = chunk_nbytes * chunks_per_plane plane_pixels = shape[y] * shape[x] + channels = _channel_count(shape, mapping) memory = ( page_nbytes + plane_pixels * dtype.itemsize + estimate_histogram_memory(plane_pixels, dtype, bin_count) + + _finished_histograms_nbytes(dtype, bin_count, channels) ) layout = PlaneLayout( shape=shape, @@ -1124,6 +1280,7 @@ def _zarr_layout( encoded_nbytes=0, memory_estimate=memory, memory_limit=memory_limit, + channel_count=channels, ) if memory <= memory_limit: return layout diff --git a/src/morphobricks_preprocessing/report.py b/src/morphobricks_preprocessing/report.py index a0bb3e4e..1c342e83 100644 --- a/src/morphobricks_preprocessing/report.py +++ b/src/morphobricks_preprocessing/report.py @@ -10,6 +10,7 @@ from collections.abc import Iterable, Sequence from dataclasses import dataclass +from morphobricks_normalization.histogram import PlaneHistogram from morphobricks_normalization.pooling import NoValidPixelsError, PooledHistogram from morphobricks_normalization.records import LEVELS, PlaneRecord, group_records @@ -216,7 +217,8 @@ class GroupSummary: labels The group's labels, from condition down to the chosen level. plane_count - Planes in the group, including planes without finite pixels. + Planes in the group, each channel of an XYC plane counted as one plane, + including planes without finite pixels. pixel_count Finite pixels pooled. minimum, maximum, mean, std, median @@ -255,15 +257,16 @@ def group_summaries( """ summaries = [] for key, members in group_records(list(records), level).items(): + histograms = channel_histograms(members) try: - pooled = PooledHistogram(members) + pooled = PooledHistogram(histograms) except NoValidPixelsError: - summaries.append(GroupSummary(key, len(members), 0)) + summaries.append(GroupSummary(key, len(histograms), 0)) continue summaries.append( GroupSummary( labels=key, - plane_count=len(members), + plane_count=len(histograms), pixel_count=pooled.pixel_count, minimum=pooled.minimum, maximum=pooled.maximum, @@ -276,6 +279,16 @@ def group_summaries( return summaries +def channel_histograms(records: Iterable[PlaneRecord]) -> list[PlaneHistogram]: + """Return the histogram of every channel of every row, rows first. + + The group statistics pool every channel of a group together, as they did + when the table had one row per channel (schema version 2), so the table and + plot of step 4 are unchanged. + """ + return [histogram for record in records for histogram in record.histograms] + + def group_table(summaries: Sequence[GroupSummary], level: str) -> list[list[str]]: """Turn the summaries into rows of text, with a header row, for printing or HTML.""" label_columns = [LEVEL_NAMES[name] for name in (*LEVELS, "image")] diff --git a/src/morphobricks_preprocessing/run.py b/src/morphobricks_preprocessing/run.py index 4df35ffb..1d5ada94 100644 --- a/src/morphobricks_preprocessing/run.py +++ b/src/morphobricks_preprocessing/run.py @@ -66,6 +66,7 @@ PlaneReadError, check_planes, iter_planes, + iter_xyc_planes, plane_layout, read_sample_plane, ) @@ -681,18 +682,20 @@ def _read_file( settings: RunSettings, position: str | None, ) -> list[PlaneRecord]: - """Histogram every plane of ``path``, recording the rows under the source ``rel``.""" + """Histogram every XYC plane of ``path``, recording the rows under the source ``rel``. + + One row per plane holds the histograms of all its channels (schema version 3). + """ records = [] - for plane in iter_planes( + for plane in iter_xyc_planes( path, mapping, memory_limit=settings.memory_limit, bin_count=settings.bin_count ): records.append( PlaneRecord( rel, *labels, - histogram=plane.histogram(settings.bin_count), + histograms=plane.histograms(settings.bin_count), z=plane.z, - c=plane.c, t=plane.t, position_id=position, ) @@ -817,12 +820,12 @@ def _bin_count(dtype, bin_count: int | None) -> tuple[int | None, list[Validatio def _pixel_issues(records: list[PlaneRecord]) -> list[ValidationIssue]: + # Every channel of every row is checked. + channels = [(r.source_relative_path, h) for r in records for h in r.histograms] issues = [] - with_non_finite = sorted( - {r.source_relative_path for r in records if r.histogram.non_finite_pixel_count} - ) + with_non_finite = sorted({rel for rel, h in channels if h.non_finite_pixel_count}) if with_non_finite: - total = sum(r.histogram.non_finite_pixel_count for r in records) + total = sum(h.non_finite_pixel_count for _, h in channels) issues.append( warning( "non_finite_pixels", @@ -833,23 +836,25 @@ def _pixel_issues(records: list[PlaneRecord]) -> list[ValidationIssue]: "No action needed if these values mark missing data.", ) ) - empty = [r for r in records if r.histogram.is_empty] - if empty and len(empty) == len(records): + empty = [rel for rel, h in channels if h.is_empty] + if empty and len(empty) == len(channels): issues.append( error( "no_usable_planes", "No plane has any finite pixel value, so no statistics can be calculated.", - sorted({r.source_relative_path for r in empty}), + sorted(set(empty)), "Check that these are the right images and that they were exported correctly.", ) ) elif empty: + planes = {r.plane_id for r in records if any(h.is_empty for h in r.histograms)} issues.append( warning( "no_finite_pixels", - f"{plural(len(empty), 'plane')} have no finite pixel values. They are kept in " - "the table with zero counts and left out of group statistics.", - sorted({r.source_relative_path for r in empty}), + f"{plural(len(planes), 'plane')} have no finite pixel values in at least one " + "channel. These channels are kept in the table with zero counts and left out " + "of group statistics.", + sorted(set(empty)), "No action needed if these planes are expected to be empty.", ) ) @@ -901,7 +906,7 @@ def _normalise(path: str) -> str: def _row_order(record: PlaneRecord) -> tuple: - return (record.source_relative_path, record.z, record.c, record.t) + return (record.source_relative_path, record.z, record.t) def _sorted(issues: list[ValidationIssue]) -> tuple[ValidationIssue, ...]: diff --git a/tests/data/make_v2_fixture.py b/tests/data/make_v2_fixture.py new file mode 100644 index 00000000..131eb24e --- /dev/null +++ b/tests/data/make_v2_fixture.py @@ -0,0 +1,60 @@ +"""Write the schema-version-2 fixture of ``test_schema_v2_equivalence.py``. + +Run once, from the repository root, with the code of version 2: this repository at +commit ``55f1074`` (before follow-up P3b) and morphobricks-normalization v0.1.0, +in an environment of their own: + + git worktree add ../v2 55f1074 + git clone -b v0.1.0 https://github.com/CellMigrationLab/morphobricks-normalization ../n1 + pip install ../n1 tifffile zarr pytest + PYTHONPATH=../v2/src python tests/data/make_v2_fixture.py + +It writes, next to this script: + +- ``v2_pixels.npz``: the pixels of every image of the experiments in + ``tests/v2_fixture.py``; +- ``v2_.csv``: the ``plane_histograms.csv`` that version's + ``run_experiment`` wrote for them (schema version 2, one row per plane and + channel). + +The fixture is committed, so the test compares the current code with the output +of the earlier code rather than with itself. +""" + +from __future__ import annotations + +import shutil +import sys +import tempfile +from pathlib import Path + +import numpy as np + +HERE = Path(__file__).parent +sys.path.insert(0, str(HERE.parent)) + +from morphobricks_normalization.csv_io import SCHEMA_VERSION # noqa: E402 + +from morphobricks_preprocessing import prefill_mapping, run_experiment # noqa: E402 +from v2_fixture import EXPERIMENTS, PIXELS, make_pixels, write_experiment # noqa: E402 + + +def main() -> None: + """Write the pixels and the version-2 tables.""" + if SCHEMA_VERSION != "2": + sys.exit(f"Run this with the code of schema version 2, not {SCHEMA_VERSION}.") + pixels = make_pixels() + np.savez_compressed(PIXELS, **pixels) + for dtype in EXPERIMENTS: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) / "exp" + bins, axes = write_experiment(root, dtype, pixels) + result = run_experiment(root, prefill_mapping(axes), bin_count=bins) + if not result.written: + sys.exit(f"{dtype}: {[issue.message for issue in result.issues]}") + shutil.copyfile(result.csv_path, HERE / f"v2_{dtype}.csv") + print(f"{dtype}: {result.plane_count} rows") + + +if __name__ == "__main__": + main() diff --git a/tests/data/v2_float32.csv b/tests/data/v2_float32.csv new file mode 100644 index 00000000..2506abb2 --- /dev/null +++ b/tests/data/v2_float32.csv @@ -0,0 +1,9 @@ +position_id,image_file,z,c,t,condition,biological_repeat,technical_repeat,source_relative_path,plane_id,dtype,bit_depth,pixel_min,pixel_max,total_pixel_count,valid_pixel_count,binning_min,binning_max,bin_count,schema_version,histogram_bin_00000,histogram_bin_00001,histogram_bin_00002,histogram_bin_00003,histogram_bin_00004,histogram_bin_00005,histogram_bin_00006,histogram_bin_00007,histogram_bin_00008,histogram_bin_00009,histogram_bin_00010,histogram_bin_00011,histogram_bin_00012,histogram_bin_00013,histogram_bin_00014,histogram_bin_00015,histogram_bin_00016,histogram_bin_00017,histogram_bin_00018,histogram_bin_00019,histogram_bin_00020,histogram_bin_00021,histogram_bin_00022,histogram_bin_00023,histogram_bin_00024,histogram_bin_00025,histogram_bin_00026,histogram_bin_00027,histogram_bin_00028,histogram_bin_00029,histogram_bin_00030,histogram_bin_00031,histogram_bin_00032,histogram_bin_00033,histogram_bin_00034,histogram_bin_00035,histogram_bin_00036,histogram_bin_00037,histogram_bin_00038,histogram_bin_00039,histogram_bin_00040,histogram_bin_00041,histogram_bin_00042,histogram_bin_00043,histogram_bin_00044,histogram_bin_00045,histogram_bin_00046,histogram_bin_00047,histogram_bin_00048,histogram_bin_00049,histogram_bin_00050,histogram_bin_00051,histogram_bin_00052,histogram_bin_00053,histogram_bin_00054,histogram_bin_00055,histogram_bin_00056,histogram_bin_00057,histogram_bin_00058,histogram_bin_00059,histogram_bin_00060,histogram_bin_00061,histogram_bin_00062,histogram_bin_00063 +a,a.tif,0,0,0,ctrl,rep1,single,ctrl/rep1/a.tif,"ctrl/rep1/a.tif::z=0,c=0,t=0",float32,32,-2.2212765216827393,110.27455139160156,195,192,-2.2212765216827393,110.27455139160156,64,2,1,0,1,1,2,0,0,0,2,0,3,0,2,2,4,4,3,2,2,5,8,6,5,5,8,6,7,7,7,3,9,5,6,6,8,4,3,6,4,1,8,4,3,3,5,5,4,1,2,0,1,1,1,1,2,1,0,1,0,0,0,0,0,1 +a,a.tif,0,0,1,ctrl,rep1,single,ctrl/rep1/a.tif,"ctrl/rep1/a.tif::z=0,c=0,t=1",float32,32,7.42095947265625,104.61268615722656,195,195,7.42095947265625,104.61268615722656,64,2,2,0,0,2,1,4,1,2,1,0,2,3,2,2,2,3,1,3,4,3,5,6,5,10,8,5,8,8,8,4,3,5,6,6,8,5,7,6,3,5,4,1,1,4,6,4,2,2,3,0,0,1,2,2,0,1,0,1,1,0,0,0,0,1 +a,a.tif,0,1,0,ctrl,rep1,single,ctrl/rep1/a.tif,"ctrl/rep1/a.tif::z=0,c=1,t=0",float32,32,-3.4378063678741455,102.93948364257812,195,195,-3.4378063678741455,102.93948364257812,64,2,2,0,0,1,0,0,1,1,2,2,1,1,1,1,4,1,4,2,4,3,6,5,2,5,6,4,3,8,2,5,12,4,4,4,4,7,8,6,8,6,4,7,6,4,6,5,3,2,4,3,4,1,1,0,0,2,2,0,0,0,0,0,0,1 +a,a.tif,0,1,1,ctrl,rep1,single,ctrl/rep1/a.tif,"ctrl/rep1/a.tif::z=0,c=1,t=1",float32,32,-7.232322692871094,95.71029663085938,195,195,-7.232322692871094,95.71029663085938,64,2,1,1,0,1,0,0,0,0,0,1,0,0,1,1,1,2,1,0,3,3,4,0,6,6,5,4,9,7,4,8,6,8,4,7,7,4,9,8,9,2,2,6,5,3,5,3,6,3,5,5,0,4,2,1,0,0,2,1,3,2,1,0,1,2 +a,a.tif,0,0,0,drug,rep1,single,drug/rep1/a.tif,"drug/rep1/a.tif::z=0,c=0,t=0",float32,32,6.057465553283691,107.17195892333984,195,192,6.057465553283691,107.17195892333984,64,2,1,3,0,1,2,2,2,0,4,0,4,2,1,7,3,4,3,7,5,2,7,4,6,3,7,0,4,7,2,3,5,10,7,12,5,7,5,0,5,3,3,3,2,3,2,6,1,2,3,3,2,0,2,2,1,0,0,0,1,0,0,0,0,1 +a,a.tif,0,0,1,drug,rep1,single,drug/rep1/a.tif,"drug/rep1/a.tif::z=0,c=0,t=1",float32,32,-0.30146220326423645,112.90068817138672,195,195,-0.30146220326423645,112.90068817138672,64,2,1,1,0,0,1,1,1,0,3,0,0,1,4,4,3,5,2,1,7,8,4,5,7,2,6,12,12,6,7,7,11,7,2,5,9,3,3,5,2,5,5,4,3,4,4,1,0,3,3,0,1,1,0,0,0,0,0,0,0,0,1,0,0,2 +a,a.tif,0,1,0,drug,rep1,single,drug/rep1/a.tif,"drug/rep1/a.tif::z=0,c=1,t=0",float32,32,7.5,7.5,195,195,7.5,7.5,64,2,195,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +a,a.tif,0,1,1,drug,rep1,single,drug/rep1/a.tif,"drug/rep1/a.tif::z=0,c=1,t=1",float32,32,,,195,0,,,64,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 diff --git a/tests/data/v2_pixels.npz b/tests/data/v2_pixels.npz new file mode 100644 index 00000000..d4fdc1e5 Binary files /dev/null and b/tests/data/v2_pixels.npz differ diff --git a/tests/data/v2_uint16.csv b/tests/data/v2_uint16.csv new file mode 100644 index 00000000..724c14de --- /dev/null +++ b/tests/data/v2_uint16.csv @@ -0,0 +1,25 @@ +position_id,image_file,z,c,t,condition,biological_repeat,technical_repeat,source_relative_path,plane_id,dtype,bit_depth,pixel_min,pixel_max,total_pixel_count,valid_pixel_count,binning_min,binning_max,bin_count,schema_version,histogram_bin_00000,histogram_bin_00001,histogram_bin_00002,histogram_bin_00003,histogram_bin_00004,histogram_bin_00005,histogram_bin_00006,histogram_bin_00007,histogram_bin_00008,histogram_bin_00009,histogram_bin_00010,histogram_bin_00011,histogram_bin_00012,histogram_bin_00013,histogram_bin_00014,histogram_bin_00015,histogram_bin_00016,histogram_bin_00017,histogram_bin_00018,histogram_bin_00019,histogram_bin_00020,histogram_bin_00021,histogram_bin_00022,histogram_bin_00023,histogram_bin_00024,histogram_bin_00025,histogram_bin_00026,histogram_bin_00027,histogram_bin_00028,histogram_bin_00029,histogram_bin_00030,histogram_bin_00031,histogram_bin_00032,histogram_bin_00033,histogram_bin_00034,histogram_bin_00035,histogram_bin_00036,histogram_bin_00037,histogram_bin_00038,histogram_bin_00039,histogram_bin_00040,histogram_bin_00041,histogram_bin_00042,histogram_bin_00043,histogram_bin_00044,histogram_bin_00045,histogram_bin_00046,histogram_bin_00047,histogram_bin_00048,histogram_bin_00049,histogram_bin_00050,histogram_bin_00051,histogram_bin_00052,histogram_bin_00053,histogram_bin_00054,histogram_bin_00055,histogram_bin_00056,histogram_bin_00057,histogram_bin_00058,histogram_bin_00059,histogram_bin_00060,histogram_bin_00061,histogram_bin_00062,histogram_bin_00063,histogram_bin_00064,histogram_bin_00065,histogram_bin_00066,histogram_bin_00067,histogram_bin_00068,histogram_bin_00069,histogram_bin_00070,histogram_bin_00071,histogram_bin_00072,histogram_bin_00073,histogram_bin_00074,histogram_bin_00075,histogram_bin_00076,histogram_bin_00077,histogram_bin_00078,histogram_bin_00079,histogram_bin_00080,histogram_bin_00081,histogram_bin_00082,histogram_bin_00083,histogram_bin_00084,histogram_bin_00085,histogram_bin_00086,histogram_bin_00087,histogram_bin_00088,histogram_bin_00089,histogram_bin_00090,histogram_bin_00091,histogram_bin_00092,histogram_bin_00093,histogram_bin_00094,histogram_bin_00095,histogram_bin_00096,histogram_bin_00097,histogram_bin_00098,histogram_bin_00099,histogram_bin_00100,histogram_bin_00101,histogram_bin_00102,histogram_bin_00103,histogram_bin_00104,histogram_bin_00105,histogram_bin_00106,histogram_bin_00107,histogram_bin_00108,histogram_bin_00109,histogram_bin_00110,histogram_bin_00111,histogram_bin_00112,histogram_bin_00113,histogram_bin_00114,histogram_bin_00115,histogram_bin_00116,histogram_bin_00117,histogram_bin_00118,histogram_bin_00119,histogram_bin_00120,histogram_bin_00121,histogram_bin_00122,histogram_bin_00123,histogram_bin_00124,histogram_bin_00125,histogram_bin_00126,histogram_bin_00127,histogram_bin_00128,histogram_bin_00129,histogram_bin_00130,histogram_bin_00131,histogram_bin_00132,histogram_bin_00133,histogram_bin_00134,histogram_bin_00135,histogram_bin_00136,histogram_bin_00137,histogram_bin_00138,histogram_bin_00139,histogram_bin_00140,histogram_bin_00141,histogram_bin_00142,histogram_bin_00143,histogram_bin_00144,histogram_bin_00145,histogram_bin_00146,histogram_bin_00147,histogram_bin_00148,histogram_bin_00149,histogram_bin_00150,histogram_bin_00151,histogram_bin_00152,histogram_bin_00153,histogram_bin_00154,histogram_bin_00155,histogram_bin_00156,histogram_bin_00157,histogram_bin_00158,histogram_bin_00159,histogram_bin_00160,histogram_bin_00161,histogram_bin_00162,histogram_bin_00163,histogram_bin_00164,histogram_bin_00165,histogram_bin_00166,histogram_bin_00167,histogram_bin_00168,histogram_bin_00169,histogram_bin_00170,histogram_bin_00171,histogram_bin_00172,histogram_bin_00173,histogram_bin_00174,histogram_bin_00175,histogram_bin_00176,histogram_bin_00177,histogram_bin_00178,histogram_bin_00179,histogram_bin_00180,histogram_bin_00181,histogram_bin_00182,histogram_bin_00183,histogram_bin_00184,histogram_bin_00185,histogram_bin_00186,histogram_bin_00187,histogram_bin_00188,histogram_bin_00189,histogram_bin_00190,histogram_bin_00191,histogram_bin_00192,histogram_bin_00193,histogram_bin_00194,histogram_bin_00195,histogram_bin_00196,histogram_bin_00197,histogram_bin_00198,histogram_bin_00199,histogram_bin_00200,histogram_bin_00201,histogram_bin_00202,histogram_bin_00203,histogram_bin_00204,histogram_bin_00205,histogram_bin_00206,histogram_bin_00207,histogram_bin_00208,histogram_bin_00209,histogram_bin_00210,histogram_bin_00211,histogram_bin_00212,histogram_bin_00213,histogram_bin_00214,histogram_bin_00215,histogram_bin_00216,histogram_bin_00217,histogram_bin_00218,histogram_bin_00219,histogram_bin_00220,histogram_bin_00221,histogram_bin_00222,histogram_bin_00223,histogram_bin_00224,histogram_bin_00225,histogram_bin_00226,histogram_bin_00227,histogram_bin_00228,histogram_bin_00229,histogram_bin_00230,histogram_bin_00231,histogram_bin_00232,histogram_bin_00233,histogram_bin_00234,histogram_bin_00235,histogram_bin_00236,histogram_bin_00237,histogram_bin_00238,histogram_bin_00239,histogram_bin_00240,histogram_bin_00241,histogram_bin_00242,histogram_bin_00243,histogram_bin_00244,histogram_bin_00245,histogram_bin_00246,histogram_bin_00247,histogram_bin_00248,histogram_bin_00249,histogram_bin_00250,histogram_bin_00251,histogram_bin_00252,histogram_bin_00253,histogram_bin_00254,histogram_bin_00255,histogram_bin_00256,histogram_bin_00257,histogram_bin_00258,histogram_bin_00259,histogram_bin_00260,histogram_bin_00261,histogram_bin_00262,histogram_bin_00263,histogram_bin_00264,histogram_bin_00265,histogram_bin_00266,histogram_bin_00267,histogram_bin_00268,histogram_bin_00269,histogram_bin_00270,histogram_bin_00271,histogram_bin_00272,histogram_bin_00273,histogram_bin_00274,histogram_bin_00275,histogram_bin_00276,histogram_bin_00277,histogram_bin_00278,histogram_bin_00279,histogram_bin_00280,histogram_bin_00281,histogram_bin_00282,histogram_bin_00283,histogram_bin_00284,histogram_bin_00285,histogram_bin_00286,histogram_bin_00287,histogram_bin_00288,histogram_bin_00289,histogram_bin_00290,histogram_bin_00291,histogram_bin_00292,histogram_bin_00293,histogram_bin_00294,histogram_bin_00295,histogram_bin_00296,histogram_bin_00297,histogram_bin_00298,histogram_bin_00299,histogram_bin_00300,histogram_bin_00301,histogram_bin_00302,histogram_bin_00303,histogram_bin_00304,histogram_bin_00305,histogram_bin_00306,histogram_bin_00307,histogram_bin_00308,histogram_bin_00309,histogram_bin_00310,histogram_bin_00311,histogram_bin_00312,histogram_bin_00313,histogram_bin_00314,histogram_bin_00315,histogram_bin_00316,histogram_bin_00317,histogram_bin_00318,histogram_bin_00319,histogram_bin_00320,histogram_bin_00321,histogram_bin_00322,histogram_bin_00323,histogram_bin_00324,histogram_bin_00325,histogram_bin_00326,histogram_bin_00327,histogram_bin_00328,histogram_bin_00329,histogram_bin_00330,histogram_bin_00331,histogram_bin_00332,histogram_bin_00333,histogram_bin_00334,histogram_bin_00335,histogram_bin_00336,histogram_bin_00337,histogram_bin_00338,histogram_bin_00339,histogram_bin_00340,histogram_bin_00341,histogram_bin_00342,histogram_bin_00343,histogram_bin_00344,histogram_bin_00345,histogram_bin_00346,histogram_bin_00347,histogram_bin_00348,histogram_bin_00349,histogram_bin_00350,histogram_bin_00351,histogram_bin_00352,histogram_bin_00353,histogram_bin_00354,histogram_bin_00355,histogram_bin_00356,histogram_bin_00357,histogram_bin_00358,histogram_bin_00359,histogram_bin_00360,histogram_bin_00361,histogram_bin_00362,histogram_bin_00363,histogram_bin_00364,histogram_bin_00365,histogram_bin_00366,histogram_bin_00367,histogram_bin_00368,histogram_bin_00369,histogram_bin_00370,histogram_bin_00371,histogram_bin_00372,histogram_bin_00373,histogram_bin_00374,histogram_bin_00375,histogram_bin_00376,histogram_bin_00377,histogram_bin_00378,histogram_bin_00379,histogram_bin_00380,histogram_bin_00381,histogram_bin_00382,histogram_bin_00383,histogram_bin_00384,histogram_bin_00385,histogram_bin_00386,histogram_bin_00387,histogram_bin_00388,histogram_bin_00389,histogram_bin_00390,histogram_bin_00391,histogram_bin_00392,histogram_bin_00393,histogram_bin_00394,histogram_bin_00395,histogram_bin_00396,histogram_bin_00397,histogram_bin_00398,histogram_bin_00399,histogram_bin_00400,histogram_bin_00401,histogram_bin_00402,histogram_bin_00403,histogram_bin_00404,histogram_bin_00405,histogram_bin_00406,histogram_bin_00407,histogram_bin_00408,histogram_bin_00409,histogram_bin_00410,histogram_bin_00411,histogram_bin_00412,histogram_bin_00413,histogram_bin_00414,histogram_bin_00415,histogram_bin_00416,histogram_bin_00417,histogram_bin_00418,histogram_bin_00419,histogram_bin_00420,histogram_bin_00421,histogram_bin_00422,histogram_bin_00423,histogram_bin_00424,histogram_bin_00425,histogram_bin_00426,histogram_bin_00427,histogram_bin_00428,histogram_bin_00429,histogram_bin_00430,histogram_bin_00431,histogram_bin_00432,histogram_bin_00433,histogram_bin_00434,histogram_bin_00435,histogram_bin_00436,histogram_bin_00437,histogram_bin_00438,histogram_bin_00439,histogram_bin_00440,histogram_bin_00441,histogram_bin_00442,histogram_bin_00443,histogram_bin_00444,histogram_bin_00445,histogram_bin_00446,histogram_bin_00447,histogram_bin_00448,histogram_bin_00449,histogram_bin_00450,histogram_bin_00451,histogram_bin_00452,histogram_bin_00453,histogram_bin_00454,histogram_bin_00455,histogram_bin_00456,histogram_bin_00457,histogram_bin_00458,histogram_bin_00459,histogram_bin_00460,histogram_bin_00461,histogram_bin_00462,histogram_bin_00463,histogram_bin_00464,histogram_bin_00465,histogram_bin_00466,histogram_bin_00467,histogram_bin_00468,histogram_bin_00469,histogram_bin_00470,histogram_bin_00471,histogram_bin_00472,histogram_bin_00473,histogram_bin_00474,histogram_bin_00475,histogram_bin_00476,histogram_bin_00477,histogram_bin_00478,histogram_bin_00479,histogram_bin_00480,histogram_bin_00481,histogram_bin_00482,histogram_bin_00483,histogram_bin_00484,histogram_bin_00485,histogram_bin_00486,histogram_bin_00487,histogram_bin_00488,histogram_bin_00489,histogram_bin_00490,histogram_bin_00491,histogram_bin_00492,histogram_bin_00493,histogram_bin_00494,histogram_bin_00495,histogram_bin_00496,histogram_bin_00497,histogram_bin_00498,histogram_bin_00499 +a,a.tif,0,0,0,ctrl,rep1,single,ctrl/rep1/a.tif,"ctrl/rep1/a.tif::z=0,c=0,t=0",uint16,16,100,399,360,360,100,399,500,2,1,1,3,0,4,0,1,1,2,0,1,1,4,0,3,1,0,2,1,2,1,1,1,3,3,1,1,2,0,0,1,1,1,0,2,1,0,3,1,1,1,1,1,1,3,1,0,0,0,1,0,0,2,0,2,1,0,0,1,2,0,0,1,0,0,1,0,2,2,4,0,1,1,0,0,0,1,3,0,2,3,3,2,1,1,0,2,0,0,2,0,0,2,2,3,0,1,0,2,0,1,1,0,2,1,0,0,1,1,2,2,0,1,2,1,2,1,1,1,0,2,2,3,1,2,0,3,2,1,2,0,0,2,0,0,2,1,0,2,0,0,0,2,1,1,1,1,2,1,2,1,2,1,4,0,1,0,3,2,1,0,1,1,1,0,1,2,1,2,1,0,2,2,1,1,1,0,0,3,0,0,2,0,0,2,0,3,2,1,1,3,3,0,0,2,3,0,1,0,0,2,5,0,0,0,1,1,2,0,0,1,2,0,2,1,3,1,0,3,1,2,0,0,0,2,0,0,3,2,4,1,0,1,1,0,2,2,1,1,0,2,1,1,2,2,2,2,1,2,1,1,0,2,2,1,1,0,3,3,2,0,2,2,0,2,0,3,3,1,2,1,0,1,1,0,0,0,3,4,2,2,0,2,4,1,2,1,3,1,1,1,4,1,0,2,1,0,0,3,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +a,a.tif,0,1,0,ctrl,rep1,single,ctrl/rep1/a.tif,"ctrl/rep1/a.tif::z=0,c=1,t=0",uint16,16,124,5097,360,360,124,5097,500,2,2,1,0,1,1,0,1,0,0,0,0,2,0,0,0,0,0,1,1,2,0,1,0,0,1,2,0,1,0,0,1,1,0,0,2,1,0,0,2,1,0,2,0,1,0,2,0,0,0,1,0,1,1,1,0,0,1,1,0,0,0,0,0,0,0,1,0,0,1,1,1,1,0,0,1,1,0,1,0,1,1,1,0,2,1,3,0,4,0,2,1,0,2,4,0,3,0,2,0,1,1,0,1,1,1,1,2,0,0,0,0,2,1,0,1,0,0,2,0,0,0,0,1,0,1,0,1,2,0,1,1,0,1,1,0,0,1,1,0,0,0,1,0,0,1,4,2,0,0,1,1,0,0,0,1,1,1,2,0,1,2,1,0,0,0,0,0,1,2,1,0,0,0,1,1,0,2,0,1,0,0,0,1,1,0,2,0,0,0,2,0,0,2,0,2,2,1,1,0,1,0,0,1,0,0,2,0,0,0,0,1,1,3,0,1,1,0,0,1,1,1,0,1,1,2,2,0,0,1,0,1,1,1,0,0,0,2,2,0,0,1,2,1,0,0,0,1,0,2,1,1,2,1,0,1,1,1,0,2,0,0,0,1,1,2,0,1,1,0,0,0,1,2,0,1,1,1,2,2,0,1,1,0,0,1,0,3,2,2,2,0,3,1,1,1,1,0,0,1,0,0,2,1,0,2,3,2,1,0,0,0,2,1,0,3,0,1,2,0,1,0,2,0,0,0,1,1,0,0,0,0,1,0,1,2,1,0,1,1,1,0,0,0,0,1,0,1,2,0,0,1,0,0,0,2,1,0,1,1,0,0,1,2,0,0,1,0,1,0,1,1,1,0,2,0,0,1,0,1,1,1,0,0,0,0,0,2,2,2,3,2,0,1,1,1,0,0,0,1,0,2,0,0,0,0,0,0,1,0,0,3,2,0,0,0,2,3,0,3,1,2,2,0,0,0,2,0,1,0,2,0,0,1,0,0,0,1,1,1,1,1,1,0,1,1,0,0,1,1,0,0,1,0,0,0,1,2,0,0,1,0,0,0,1,1,0,0,2,2,0,2,0,1,0,0,2,0,0,0,1,1,0,0,0,0,1,1,1,0,0,2,2,2,1,1,1,0,0,1,1 +a,a.tif,0,2,0,ctrl,rep1,single,ctrl/rep1/a.tif,"ctrl/rep1/a.tif::z=0,c=2,t=0",uint16,16,111,60071,360,360,111,60071,500,2,2,2,0,0,0,1,2,3,0,1,1,0,1,0,0,0,1,2,1,0,0,0,1,2,1,0,0,1,0,0,0,0,0,2,0,0,0,1,0,0,0,1,1,1,1,0,1,0,0,0,0,1,0,1,0,1,0,1,0,0,0,1,3,0,1,1,1,1,1,1,0,1,1,1,1,0,0,1,1,0,1,0,0,0,2,0,1,1,0,2,2,1,0,2,0,1,1,0,2,2,1,0,0,2,1,1,1,0,1,0,0,0,1,2,1,1,0,0,1,1,0,2,0,1,0,2,2,0,0,1,0,0,0,1,1,3,0,0,0,0,2,1,0,0,1,1,1,1,0,1,1,0,0,1,0,0,0,0,0,1,0,1,1,2,0,0,2,2,0,0,0,1,1,0,0,1,1,1,0,2,0,0,1,3,0,1,1,1,1,1,1,1,1,1,0,0,0,2,0,1,2,1,1,0,1,1,0,1,0,1,0,0,1,0,1,2,1,1,0,4,1,0,0,1,1,1,0,0,1,3,0,0,1,0,2,0,0,1,0,1,1,0,0,0,0,0,2,2,1,2,0,1,1,0,3,0,0,1,7,0,0,1,0,0,1,1,2,0,0,0,0,1,0,0,2,0,0,0,0,0,0,0,3,1,1,1,0,1,1,0,0,1,0,1,1,0,2,1,0,1,1,1,0,1,0,1,1,0,1,0,1,1,1,1,2,1,2,1,0,0,0,0,1,1,1,1,1,0,0,2,1,1,2,2,0,0,0,1,0,2,0,1,1,0,2,0,1,0,0,1,0,1,0,1,1,0,2,0,1,0,0,1,0,0,1,1,0,1,1,1,1,0,0,1,0,0,2,1,1,1,0,2,1,2,1,1,2,3,0,0,0,2,0,0,0,3,0,0,0,0,1,0,0,1,3,1,1,1,1,0,1,1,0,0,0,1,1,1,1,0,0,3,2,1,1,0,2,1,3,1,1,0,1,0,2,0,0,0,0,1,2,3,0,1,1,1,0,2,1,1,0,2,0,1,2,0,1,0,0,2,0,0,0,1,0,2,0,0,0,0,1,0,0,0,0,0,1,0,1,0,0,0,0,0,0,1,0,0,2,1,0,0,0,2,1,0,2,1,0,4 +a,a.tif,1,0,0,ctrl,rep1,single,ctrl/rep1/a.tif,"ctrl/rep1/a.tif::z=1,c=0,t=0",uint16,16,100,399,360,360,100,399,500,2,1,3,0,1,2,0,1,2,0,0,2,1,2,0,2,0,1,0,1,1,1,2,3,0,0,2,4,1,1,0,0,1,0,2,0,2,3,0,3,0,3,2,1,0,0,1,1,0,0,2,2,2,2,2,1,0,1,2,3,3,1,1,0,1,0,0,1,2,1,1,0,1,0,1,2,0,2,0,1,1,0,2,0,2,2,3,0,0,3,2,2,2,1,1,1,4,2,2,2,0,0,0,1,0,0,2,0,2,3,2,0,0,0,0,1,0,1,3,3,0,0,0,2,2,1,3,1,2,1,0,1,2,0,0,1,1,0,0,1,0,0,1,2,0,1,3,2,0,1,1,2,2,3,1,0,0,2,3,1,1,0,0,1,1,1,0,2,2,2,1,1,2,1,6,1,3,3,1,1,1,2,1,1,3,1,0,1,0,3,0,1,2,1,1,3,0,2,1,0,0,1,3,2,0,2,1,1,1,5,0,1,1,2,4,3,1,0,3,0,0,1,0,2,4,0,2,1,1,0,2,1,2,0,0,3,0,4,0,2,2,1,1,1,1,1,1,2,2,0,0,2,0,0,1,0,1,0,0,2,1,1,0,1,1,1,3,0,0,1,2,1,1,0,3,3,1,1,3,1,1,1,0,0,1,0,0,1,0,4,1,1,0,1,4,3,1,1,1,3,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +a,a.tif,1,1,0,ctrl,rep1,single,ctrl/rep1/a.tif,"ctrl/rep1/a.tif::z=1,c=1,t=0",uint16,16,113,5089,360,360,113,5089,500,2,1,3,0,1,0,1,0,0,0,0,1,1,1,1,0,0,0,0,0,1,0,0,1,1,0,0,0,1,0,0,0,2,2,0,1,0,1,1,0,0,1,1,2,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,2,1,1,2,1,2,2,0,0,1,1,0,1,0,2,2,0,1,0,0,0,0,1,1,1,1,1,1,0,0,0,0,1,1,1,0,1,0,0,1,2,1,1,1,0,0,0,1,0,2,0,1,2,0,1,0,2,0,0,0,0,0,3,1,2,1,1,0,2,1,0,0,0,1,1,0,4,0,2,0,0,1,4,0,0,0,0,2,1,2,0,1,1,1,2,0,0,0,0,3,1,0,1,0,0,0,0,1,0,1,1,0,0,0,0,1,0,0,0,3,0,0,2,0,5,0,0,0,1,1,3,3,0,0,1,1,1,0,1,0,1,1,0,0,2,2,0,1,1,0,3,2,1,1,1,2,1,0,0,2,1,0,0,0,0,1,2,2,2,2,0,3,0,1,0,1,0,1,2,2,0,0,0,0,1,0,0,0,4,0,0,1,1,0,0,2,1,0,1,0,1,0,0,1,1,0,0,0,0,1,1,1,0,0,0,0,1,0,1,0,0,1,2,1,0,0,3,0,2,0,1,1,1,1,0,1,1,1,0,1,1,0,1,0,0,0,0,1,0,0,0,0,2,1,0,0,2,1,0,0,0,1,0,0,0,0,0,2,0,1,1,0,1,1,2,1,2,1,1,0,1,0,0,0,0,1,2,0,1,0,0,0,1,0,0,0,2,0,1,0,2,2,2,1,1,2,1,1,1,1,2,0,0,2,1,0,1,1,1,1,2,1,0,0,0,0,0,1,1,1,0,0,1,1,0,0,0,1,0,0,2,0,0,2,0,2,1,0,0,0,0,0,0,2,0,2,0,1,0,0,1,1,1,2,1,0,0,0,2,1,2,1,1,1,0,0,3,0,2,0,0,1,0,1,1,0,1,3,0,0,2,0,1,0,0,0,1,0,2,0,2,0,0,1,1,0,2,1,1,1,0,2,0,1,0,1,1,2,1,0,0,2,0,1,2,0,1,0,2,1,1,0,0,2,2,0,1,2,0,1,1,2 +a,a.tif,1,2,0,ctrl,rep1,single,ctrl/rep1/a.tif,"ctrl/rep1/a.tif::z=1,c=2,t=0",uint16,16,113,59733,360,360,113,59733,500,2,1,1,1,1,1,0,2,0,0,0,3,0,2,0,1,0,1,0,0,1,2,0,0,1,1,0,0,0,0,0,1,0,0,1,2,1,2,1,1,2,0,0,1,1,0,1,1,2,0,1,0,0,0,2,0,1,3,0,0,0,1,2,0,1,0,0,2,1,2,2,0,0,0,1,1,0,0,0,1,0,0,1,1,0,1,1,2,1,0,1,0,0,2,0,3,0,0,1,0,1,0,0,0,0,0,1,0,0,1,0,1,0,1,1,0,0,1,1,1,0,1,1,1,0,0,1,0,0,2,1,2,3,0,2,2,1,0,1,2,2,1,0,2,0,0,0,3,1,2,0,0,1,0,0,1,0,0,0,0,0,1,0,1,1,0,0,1,1,0,1,1,0,2,1,0,1,1,0,2,1,0,0,0,0,1,0,2,0,0,0,2,1,1,1,2,0,0,1,0,1,0,0,1,0,0,0,0,0,0,1,4,0,0,0,0,2,0,1,0,3,0,2,0,1,0,1,1,1,1,2,0,1,0,1,1,0,1,0,0,0,0,0,1,0,0,0,0,1,0,2,2,0,1,2,0,1,1,0,0,2,1,2,4,0,0,0,1,1,0,0,0,1,1,1,1,0,1,1,1,1,2,2,2,4,0,2,2,0,0,1,0,0,0,0,0,2,0,2,0,1,1,1,1,1,1,1,0,2,1,0,0,0,0,1,0,1,2,1,0,0,0,0,2,0,1,1,1,0,1,1,1,0,0,2,1,1,0,1,0,0,0,1,4,1,2,0,0,0,1,0,0,1,1,0,1,2,0,1,1,0,0,1,0,0,0,1,0,1,0,1,0,0,0,3,1,1,0,0,1,0,2,0,0,0,2,0,3,2,1,1,0,0,0,0,0,0,0,1,0,1,2,0,2,0,2,1,0,0,0,2,1,1,1,1,2,1,2,0,0,1,0,0,2,2,0,3,1,1,1,0,2,2,0,0,0,1,0,2,0,0,1,3,0,1,0,0,1,0,0,0,0,0,1,1,0,1,1,1,2,1,0,1,0,3,0,0,1,1,0,1,1,0,1,3,1,2,0,0,0,1,0,0,1,0,2,1,1,2,2,0,1,0,0,0,0,2,0,0,0,1 +b,b.tif,0,0,0,ctrl,rep1,single,ctrl/rep1/b.tif,"ctrl/rep1/b.tif::z=0,c=0,t=0",uint16,16,100,399,360,360,100,399,500,2,1,0,1,2,2,3,1,2,1,1,3,0,2,1,1,3,1,0,2,0,1,2,1,0,1,2,1,2,1,1,1,2,3,0,0,1,1,1,2,2,1,0,2,1,1,1,1,1,2,3,0,1,0,2,1,0,1,0,1,0,1,0,1,0,1,1,1,1,1,2,0,2,1,2,1,0,0,2,1,1,5,1,0,0,0,1,1,1,2,2,1,1,3,1,0,4,4,1,1,1,1,3,2,3,4,1,2,1,0,2,2,1,1,0,0,2,1,0,1,2,0,0,0,0,0,1,1,1,0,1,0,1,0,1,0,1,3,0,0,1,3,1,0,1,3,2,1,1,1,1,0,0,0,0,1,2,1,0,2,2,1,3,4,2,1,0,2,2,1,0,2,1,0,0,1,1,0,1,3,0,6,5,2,1,1,0,1,2,0,1,1,1,1,2,0,3,1,3,0,2,2,0,0,0,0,1,2,2,2,0,4,3,0,1,1,0,0,0,1,1,0,0,0,1,1,0,3,0,0,0,2,1,1,1,0,4,2,1,2,1,2,3,2,0,1,2,0,3,0,3,2,0,2,0,2,0,0,2,1,1,0,4,1,1,1,0,2,0,1,0,3,2,1,2,2,0,2,1,1,2,3,0,3,0,1,1,0,2,3,3,1,0,3,0,2,0,4,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +b,b.tif,0,1,0,ctrl,rep1,single,ctrl/rep1/b.tif,"ctrl/rep1/b.tif::z=0,c=1,t=0",uint16,16,109,5082,360,360,109,5082,500,2,3,1,2,2,0,5,1,0,0,1,2,1,0,1,1,2,1,0,1,0,1,0,1,1,0,2,1,0,0,0,0,0,2,0,1,0,2,0,2,1,0,1,0,0,0,1,1,0,2,0,0,0,0,0,0,0,3,1,2,0,0,0,0,0,0,1,0,0,0,1,2,1,1,0,1,1,2,0,0,0,0,2,1,0,0,0,1,0,1,1,0,0,1,2,0,1,2,0,1,1,1,0,0,1,1,0,0,0,2,0,1,0,0,0,0,1,0,1,0,1,0,0,1,1,1,0,0,0,0,0,1,0,0,1,0,0,1,1,0,2,0,0,1,1,1,1,0,2,0,2,0,1,0,1,0,0,1,0,0,3,1,1,1,1,1,1,1,1,2,1,0,0,0,2,1,1,0,0,1,0,2,0,2,2,0,1,0,0,1,2,0,0,0,1,2,0,2,0,2,0,0,1,1,3,1,0,0,1,0,0,0,0,2,2,1,0,0,1,0,0,0,2,0,0,0,1,0,2,1,1,2,1,2,2,1,2,1,1,0,0,0,0,1,2,2,1,2,0,0,1,0,0,0,0,1,0,3,0,1,2,1,0,0,1,0,0,2,0,1,0,1,1,0,0,0,0,1,1,1,1,0,0,0,0,0,1,2,1,2,0,1,1,0,0,0,0,1,1,0,3,1,0,1,1,0,0,2,0,1,1,0,1,0,0,0,0,2,1,2,0,0,2,0,2,0,0,2,0,1,1,2,0,1,0,0,0,1,1,0,0,2,2,0,0,3,1,0,1,1,0,1,0,0,0,1,1,1,1,0,0,0,0,0,1,1,0,2,1,1,0,0,1,0,1,0,0,0,1,0,0,0,0,1,2,1,1,0,0,2,2,1,0,1,1,1,0,1,0,0,0,0,3,0,1,0,0,1,1,1,1,0,0,0,0,1,2,0,1,0,2,2,2,1,1,0,0,1,0,1,0,1,3,0,1,0,0,0,0,1,0,0,0,0,0,0,0,1,2,1,0,0,1,0,2,1,1,0,1,2,1,0,1,1,2,0,1,1,0,2,1,1,1,1,0,0,2,0,1,1,3,2,1,2,1,1,1,1,1,3,2,1,2,1,3,1,0,1,0,1,1 +b,b.tif,0,2,0,ctrl,rep1,single,ctrl/rep1/b.tif,"ctrl/rep1/b.tif::z=0,c=2,t=0",uint16,16,347,59604,360,360,347,59604,500,2,1,2,3,1,0,0,1,1,1,2,2,0,0,3,0,1,0,1,2,0,2,0,1,1,0,0,0,1,1,2,0,0,0,0,0,0,0,2,1,2,0,1,1,1,0,2,3,3,0,2,0,2,1,2,0,0,1,1,0,1,0,2,1,1,0,2,2,1,1,0,0,0,0,0,0,0,0,4,0,1,0,0,3,0,0,1,1,0,1,3,1,2,2,0,1,1,0,1,1,2,0,0,1,2,0,1,0,0,2,2,0,0,1,2,1,0,1,0,0,1,2,1,1,0,0,0,1,0,1,1,2,1,1,1,1,2,0,1,0,0,0,0,0,2,3,0,1,1,3,0,0,2,1,1,1,1,2,0,0,1,0,2,1,0,1,2,0,0,1,0,1,0,0,0,0,1,1,1,1,0,2,0,0,0,1,0,0,2,0,2,0,1,1,1,1,1,0,2,2,0,0,1,1,1,0,0,0,0,1,0,0,1,1,0,1,2,2,6,1,1,1,0,1,1,1,2,0,0,0,1,1,1,0,0,1,1,0,1,1,1,1,0,0,1,0,0,3,1,1,1,1,1,0,0,0,0,0,1,0,0,0,0,1,0,0,2,1,0,0,1,0,3,1,0,1,0,2,0,0,0,2,1,0,0,0,1,1,3,0,4,4,1,0,0,0,1,0,0,0,0,1,0,0,0,1,1,0,0,0,0,0,2,0,0,0,2,0,0,1,1,1,0,0,0,1,0,0,2,0,0,0,1,1,1,2,0,0,0,1,0,2,0,0,1,2,0,0,1,1,0,0,0,0,1,1,1,1,0,0,1,0,2,2,0,1,0,0,0,1,1,2,2,0,0,2,0,0,1,0,1,0,0,1,0,0,0,0,0,0,0,0,3,0,0,2,0,0,0,0,1,0,0,0,1,0,2,1,0,0,1,0,0,1,1,2,1,0,1,0,1,0,1,1,2,1,0,1,1,0,2,0,1,0,1,0,1,0,1,3,0,1,1,0,0,0,0,0,0,0,2,0,1,0,0,0,0,0,3,0,2,1,1,1,0,0,1,0,0,1,0,0,1,0,3,1,0,1,0,1,1,1,0,2,1,1,1,0,1,0,0,1,1,1,0,0,0,2,0,0,3 +b,b.tif,1,0,0,ctrl,rep1,single,ctrl/rep1/b.tif,"ctrl/rep1/b.tif::z=1,c=0,t=0",uint16,16,100,399,360,360,100,399,500,2,1,2,2,1,1,0,1,2,4,2,4,1,2,0,3,1,2,1,1,1,0,0,0,1,5,0,2,2,0,1,1,1,1,1,3,1,0,2,2,0,1,2,3,0,2,2,1,2,0,0,1,0,2,0,1,0,0,3,2,2,1,1,1,0,0,0,2,0,2,0,3,4,2,3,1,1,1,0,2,1,0,0,0,3,0,1,0,1,0,1,0,0,0,1,0,1,1,2,1,2,4,3,0,0,3,1,1,1,2,1,3,2,1,0,0,0,2,0,1,0,0,2,0,1,1,1,0,0,2,0,0,0,1,0,1,1,2,3,1,1,1,1,0,1,1,2,0,2,1,0,0,2,3,0,2,1,0,1,1,0,2,0,0,1,1,2,1,1,1,2,1,1,0,0,2,3,3,4,1,0,0,0,0,0,1,1,2,2,0,0,1,2,0,0,2,1,2,1,2,2,1,2,1,3,1,0,0,2,1,2,1,1,3,1,2,2,3,0,2,1,2,1,2,0,0,1,2,4,2,1,1,1,1,0,2,1,0,3,2,1,3,1,1,2,1,2,3,0,2,0,1,1,1,1,2,1,1,1,3,0,2,4,1,0,1,0,2,0,1,1,3,1,0,0,0,4,1,2,1,0,1,1,1,1,2,0,3,0,1,3,4,2,2,1,0,0,0,2,4,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +b,b.tif,1,1,0,ctrl,rep1,single,ctrl/rep1/b.tif,"ctrl/rep1/b.tif::z=1,c=1,t=0",uint16,16,115,5076,360,360,115,5076,500,2,3,0,2,1,0,1,1,1,0,0,1,0,1,3,2,1,1,1,2,0,1,1,0,0,1,0,0,1,1,1,1,0,2,0,0,0,0,0,1,1,1,2,0,0,1,0,0,0,1,1,0,2,0,1,4,0,2,1,1,1,1,1,0,0,0,1,0,0,0,3,0,0,1,1,0,0,1,1,0,1,0,1,0,2,1,2,1,1,2,0,0,1,0,0,0,1,0,1,0,1,1,1,0,0,1,2,0,0,0,0,2,1,0,2,1,0,0,0,3,0,0,1,2,3,1,0,0,0,0,1,0,1,0,0,1,0,1,3,2,1,0,0,0,0,0,0,1,2,0,0,2,0,0,0,1,0,0,0,1,1,2,0,1,2,1,0,1,2,0,1,0,0,1,0,2,0,1,0,1,0,4,2,0,0,0,0,1,1,0,2,2,0,0,1,2,0,0,0,1,0,1,0,1,1,0,1,1,0,1,2,2,0,1,1,1,1,1,0,0,0,1,1,1,0,3,0,1,0,1,2,0,0,1,0,1,0,0,0,1,1,0,0,0,0,0,2,0,1,0,1,1,1,1,1,0,2,0,1,1,1,0,1,1,0,1,1,2,0,0,0,0,0,1,0,3,1,1,0,0,0,2,0,1,0,1,0,1,1,0,2,0,1,1,0,0,4,0,1,0,0,1,2,1,2,1,1,0,1,0,0,1,1,1,0,2,0,0,0,1,1,1,1,1,1,2,1,1,0,1,0,1,1,0,0,0,1,4,1,1,1,0,1,0,0,1,0,2,1,0,0,1,0,0,1,0,0,0,1,1,0,0,1,0,1,0,0,1,0,1,0,2,1,0,2,0,0,0,1,0,1,1,0,2,0,2,3,1,0,1,1,1,3,1,0,1,0,0,2,1,0,1,1,1,0,2,0,1,0,1,1,0,0,0,0,1,0,0,1,1,0,0,0,1,2,1,0,0,1,0,1,0,1,1,0,0,2,0,2,0,1,1,0,0,0,3,0,1,1,0,1,1,0,1,0,1,1,3,1,0,3,0,0,0,0,1,2,1,2,0,1,0,1,1,0,0,1,4,0,1,1,1,1,0,0,0,3,2,1,0,0,0,0,0,1,1,2,0,0,0,1 +b,b.tif,1,2,0,ctrl,rep1,single,ctrl/rep1/b.tif,"ctrl/rep1/b.tif::z=1,c=2,t=0",uint16,16,135,60091,360,360,135,60091,500,2,2,0,1,0,0,0,1,1,0,1,0,3,0,0,0,1,1,3,0,1,0,1,0,1,0,3,1,1,1,1,0,1,0,2,0,1,0,2,3,0,1,2,0,0,1,1,0,1,3,1,0,0,2,2,1,0,1,0,1,0,1,0,0,1,2,1,2,2,0,1,2,0,0,0,0,0,0,0,2,0,2,2,2,0,0,0,0,2,2,0,1,1,0,1,0,2,1,0,1,1,0,1,1,0,0,1,0,2,1,0,1,1,1,0,1,0,0,0,0,1,0,1,1,0,0,0,0,0,1,0,3,0,0,0,0,0,1,0,3,6,2,0,0,0,0,0,1,1,0,0,1,0,2,1,0,1,1,0,1,0,1,0,0,0,1,0,0,1,2,0,2,4,0,2,2,1,0,0,0,0,1,0,0,1,0,0,1,0,2,2,1,1,1,2,0,0,3,1,0,0,0,2,0,0,2,1,0,0,2,1,0,0,1,3,0,2,0,1,1,0,2,2,2,0,2,0,1,0,3,0,0,0,1,2,0,1,1,0,0,2,2,1,4,1,0,0,0,0,1,0,1,0,3,0,0,0,1,1,1,0,1,0,1,1,0,0,0,1,0,0,1,2,1,0,1,1,0,0,2,0,2,1,2,1,2,1,1,2,0,1,0,1,0,0,1,0,0,0,1,1,1,0,0,1,0,0,1,0,0,0,1,2,1,0,2,0,0,3,0,1,1,0,1,1,1,0,0,2,1,1,1,1,0,1,0,0,1,0,0,0,1,0,0,0,1,0,0,0,0,0,0,1,0,0,1,2,0,1,1,0,0,1,0,1,1,0,1,0,0,0,1,0,3,2,1,1,0,0,1,0,0,1,0,0,0,0,0,0,1,0,1,1,1,1,1,0,1,0,0,0,0,0,1,2,0,0,0,1,1,0,2,1,1,0,1,0,0,0,1,1,1,1,2,1,0,1,3,0,0,0,4,0,0,3,1,0,2,0,0,0,0,2,2,0,2,2,1,0,1,0,1,0,1,0,0,1,2,1,1,0,2,0,1,0,0,0,0,1,0,0,3,0,0,2,0,0,1,0,0,0,0,0,0,1,1,0,0,0,2,0,1,1,0,2,2,1,1,1,1,2 +a,a.tif,0,0,0,drug,rep1,single,drug/rep1/a.tif,"drug/rep1/a.tif::z=0,c=0,t=0",uint16,16,100,399,360,360,100,399,500,2,1,0,4,2,1,1,0,2,1,4,0,1,3,2,3,2,1,1,0,3,2,2,3,2,1,0,2,2,1,2,1,0,1,0,1,1,2,0,1,2,1,0,1,1,2,2,1,0,2,3,1,2,0,1,1,0,0,3,1,3,2,2,2,1,0,0,4,2,2,1,1,1,1,0,1,0,1,0,2,1,1,0,0,0,0,1,1,2,2,0,1,0,2,2,2,2,0,0,1,1,2,0,0,0,1,0,2,2,2,0,2,4,1,0,3,1,1,0,0,1,2,2,2,1,1,0,4,1,1,0,1,3,1,1,2,1,2,2,3,1,1,2,1,1,0,0,3,0,2,0,2,0,1,1,1,2,1,1,2,1,1,0,1,1,0,3,1,1,1,4,0,2,1,2,0,4,1,1,1,2,0,1,3,0,2,1,1,1,0,1,1,2,2,2,1,2,0,1,0,1,1,3,2,0,2,0,3,1,2,1,0,3,3,2,0,0,2,1,0,0,1,0,1,1,0,1,0,2,1,1,0,1,1,0,2,1,2,1,1,1,1,0,1,0,3,2,0,0,3,1,1,2,1,1,3,1,0,1,1,1,1,1,1,2,0,0,1,2,1,1,2,2,0,0,1,0,1,1,0,0,2,1,2,2,2,1,1,1,1,2,2,0,1,0,1,2,0,1,3,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +a,a.tif,0,1,0,drug,rep1,single,drug/rep1/a.tif,"drug/rep1/a.tif::z=0,c=1,t=0",uint16,16,101,5098,360,360,101,5098,500,2,2,0,0,1,0,0,1,3,1,1,2,0,0,1,0,2,2,1,0,1,0,0,1,1,0,0,1,1,0,0,2,0,1,2,2,1,1,2,0,0,1,1,2,0,0,0,0,0,1,0,1,0,0,0,0,1,0,0,1,4,1,0,1,2,0,1,0,0,1,1,0,1,0,1,2,1,0,0,0,3,2,1,0,0,0,2,2,0,1,1,0,0,1,1,1,0,0,0,0,1,0,2,0,1,0,1,0,0,2,1,0,0,1,1,3,1,0,0,2,0,0,0,0,1,2,0,1,1,2,0,1,0,0,1,0,0,2,1,1,1,2,1,0,2,1,0,0,2,1,1,0,0,0,0,1,2,0,1,0,0,0,2,0,1,0,1,0,0,1,2,2,2,0,0,0,2,2,0,0,0,0,1,0,1,0,2,0,0,0,1,0,0,1,1,1,2,1,1,1,1,0,1,1,2,1,1,0,0,0,0,1,2,0,0,0,1,5,3,0,2,0,0,0,3,1,0,0,0,1,0,1,1,2,2,1,0,0,1,2,1,2,1,0,1,1,0,2,0,0,1,1,1,0,1,2,0,0,1,1,2,1,0,0,0,0,0,1,3,0,3,1,0,1,0,2,0,3,0,1,0,3,1,0,3,0,0,2,1,1,0,1,2,0,1,2,0,2,0,0,1,0,1,1,1,0,0,0,1,1,2,0,1,1,1,0,2,1,1,1,0,0,0,0,0,1,0,2,1,1,1,0,0,0,0,1,2,1,1,0,0,1,1,2,2,0,3,1,0,1,1,0,2,0,1,0,1,3,1,0,0,1,2,2,0,1,0,0,0,1,1,0,0,0,1,0,1,2,1,0,2,0,3,0,0,0,0,0,0,0,0,1,0,1,1,1,0,0,3,2,0,1,0,0,0,2,2,0,0,1,2,1,1,0,0,1,2,0,0,0,0,1,1,0,0,0,0,0,1,0,0,0,3,1,0,1,1,0,0,0,0,2,0,1,0,1,0,1,0,0,0,0,0,1,1,0,0,2,0,0,0,0,1,1,5,0,0,0,1,0,0,0,0,1,0,0,0,1,1,0,0,1,0,0,0,1,0,0,0,2,1,2,0,0,0,0,0,1,0,1,2 +a,a.tif,0,2,0,drug,rep1,single,drug/rep1/a.tif,"drug/rep1/a.tif::z=0,c=2,t=0",uint16,16,480,59728,360,360,480,59728,500,2,1,1,1,0,0,0,0,1,0,1,0,0,0,1,2,2,3,0,0,1,0,1,1,0,1,2,0,1,1,0,0,1,0,0,2,0,1,0,0,0,0,1,0,0,2,0,0,1,1,1,0,0,1,0,1,0,3,0,0,1,0,2,1,1,1,2,1,1,1,2,3,3,0,2,0,1,0,2,0,1,1,0,0,1,1,0,0,1,1,0,0,1,3,2,0,0,1,1,2,0,0,1,2,2,0,1,1,0,0,2,0,1,0,0,0,2,0,0,2,0,1,0,3,2,1,1,0,1,0,0,1,0,0,1,0,0,0,0,3,1,0,0,0,2,0,1,1,1,2,1,2,0,0,1,0,0,2,0,0,1,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,1,1,0,0,0,1,1,1,0,0,0,0,0,0,2,1,1,2,1,1,2,1,0,0,1,1,0,0,0,0,0,1,0,1,2,2,0,0,1,0,0,1,0,3,1,0,0,1,0,1,1,3,0,0,0,2,0,2,0,1,2,0,0,2,0,0,0,2,1,0,0,0,1,0,1,1,1,0,1,0,0,0,0,1,0,0,0,1,1,1,1,2,2,1,0,0,1,0,2,2,1,1,0,1,1,0,0,2,0,0,0,2,0,1,0,2,1,0,0,0,1,2,0,1,0,0,0,0,0,3,1,0,2,1,0,1,0,0,1,0,1,0,2,1,0,0,1,0,0,1,3,3,1,0,2,0,1,0,1,0,2,2,0,1,0,0,1,1,1,0,1,1,0,1,0,1,0,0,0,0,2,1,0,2,1,1,0,0,0,1,0,0,4,0,0,0,0,1,2,2,0,1,2,0,0,0,1,1,2,1,2,1,0,1,3,1,3,0,3,0,1,4,1,0,1,0,0,0,0,1,2,0,0,2,0,0,0,1,2,0,0,0,1,0,0,2,2,0,0,0,0,1,0,3,0,0,0,1,4,0,0,1,1,0,1,0,0,1,0,1,1,1,0,1,1,1,1,0,0,0,0,1,0,1,0,0,0,2,0,2,0,0,1,0,0,1,0,1,0,1,0,1,1,1,0,0,0,1,3,1,2,2,1,0,2,0,1,1,0,0,0,2,0,0,3 +a,a.tif,1,0,0,drug,rep1,single,drug/rep1/a.tif,"drug/rep1/a.tif::z=1,c=0,t=0",uint16,16,102,399,360,360,102,399,500,2,1,1,2,1,0,2,1,1,1,1,2,3,2,0,1,2,0,1,2,0,2,0,3,2,3,2,1,0,2,0,2,1,1,1,1,0,3,1,2,3,2,4,0,1,0,0,2,1,0,1,1,3,3,1,0,0,0,1,0,0,1,2,0,0,0,1,1,0,0,2,2,2,1,1,2,1,0,0,1,1,2,3,2,3,0,2,2,4,1,1,1,1,1,1,0,0,1,1,2,3,2,1,1,2,2,2,1,3,1,0,0,2,1,1,1,1,2,4,1,0,1,0,1,1,3,0,0,0,1,1,1,0,1,0,2,4,1,0,1,0,0,1,0,3,2,0,1,1,1,1,1,2,2,2,2,2,1,2,4,1,2,1,0,0,0,0,1,2,1,0,0,0,0,1,0,0,1,0,0,2,2,1,2,2,0,1,0,0,0,1,2,3,2,0,1,0,1,2,2,1,4,2,0,2,1,0,1,1,2,0,1,1,2,2,1,1,2,0,1,2,1,1,0,1,0,1,2,2,2,1,1,2,0,0,2,2,2,0,0,0,1,0,1,2,0,3,3,0,5,1,1,0,1,3,0,2,2,0,0,1,2,3,2,2,1,1,3,1,2,6,2,1,1,2,1,2,3,0,1,0,3,1,2,1,1,0,1,0,0,0,3,1,3,2,1,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +a,a.tif,1,1,0,drug,rep1,single,drug/rep1/a.tif,"drug/rep1/a.tif::z=1,c=1,t=0",uint16,16,102,5055,360,360,102,5055,500,2,3,1,3,0,0,0,0,2,0,2,3,1,1,0,1,0,0,0,0,1,1,1,1,0,1,1,0,0,2,0,0,2,1,1,2,0,0,0,0,2,2,0,1,0,2,1,2,0,2,0,0,1,0,2,0,0,1,0,0,0,1,0,0,1,0,1,0,1,1,1,0,1,0,0,1,1,0,0,0,1,0,0,1,0,1,1,0,0,0,0,1,2,1,1,1,1,0,5,0,2,0,2,2,0,2,1,0,1,1,0,1,0,1,0,1,0,0,0,1,0,1,0,0,0,0,2,0,1,1,1,0,0,0,1,0,3,1,0,2,1,0,3,0,3,0,1,0,1,1,1,0,2,0,1,2,0,0,1,1,0,0,2,0,0,0,2,0,2,1,1,0,2,0,1,1,0,0,0,2,0,2,0,1,1,2,0,1,0,0,0,0,0,3,0,0,1,1,1,1,2,1,0,0,0,1,4,0,0,0,1,0,0,0,0,0,1,3,1,2,1,0,2,2,0,0,0,1,0,1,1,0,0,0,1,1,3,0,2,3,0,0,0,0,0,0,0,1,0,0,2,0,0,3,2,0,0,0,0,1,0,1,0,0,1,2,2,0,0,1,2,0,1,1,0,1,0,0,3,1,0,1,1,0,1,1,0,0,0,1,0,1,3,2,1,0,1,0,2,1,1,1,1,1,2,0,0,1,0,1,0,1,0,0,1,2,0,0,2,0,2,0,1,1,0,0,0,0,1,3,0,0,0,2,0,0,0,1,0,0,1,2,0,1,0,1,0,0,0,0,3,0,2,2,0,1,2,1,0,1,0,2,1,1,0,0,0,0,0,1,1,1,1,0,0,1,0,0,0,2,0,0,2,0,0,0,0,0,2,0,0,1,1,3,0,1,0,2,0,0,1,1,1,0,1,0,0,0,0,0,0,0,3,2,0,1,1,1,2,0,0,0,2,2,1,1,1,1,1,1,0,1,1,0,1,0,2,0,0,1,0,2,1,0,1,0,0,1,0,0,1,1,0,1,2,0,0,1,1,0,1,3,3,1,0,0,0,1,0,1,1,1,1,0,0,1,2,0,0,0,0,1,0,2,0,0,1,0,0,0,1,1,2,1,0,2,0,1,1,2,1 +a,a.tif,1,2,0,drug,rep1,single,drug/rep1/a.tif,"drug/rep1/a.tif::z=1,c=2,t=0",uint16,16,255,60008,360,360,255,60008,500,2,2,0,0,0,0,2,2,1,0,1,0,1,0,0,0,0,0,2,1,0,0,1,0,1,0,2,0,0,0,0,2,1,1,0,0,0,1,0,1,0,1,0,0,0,1,3,2,1,0,1,0,1,1,0,0,1,2,0,1,4,1,0,0,0,1,1,0,0,0,0,0,1,1,1,1,1,1,0,0,0,0,1,2,1,0,2,2,2,1,1,0,0,0,2,1,0,0,1,0,1,0,0,0,2,0,1,1,1,0,1,1,2,1,1,0,1,0,0,2,3,2,1,1,0,1,0,4,0,1,0,0,1,1,1,0,0,0,0,0,0,2,0,2,1,1,1,0,1,0,0,0,1,1,0,1,2,0,1,1,1,1,1,0,1,1,2,3,1,2,0,1,0,0,2,2,0,0,3,1,1,0,0,2,1,0,0,0,1,0,1,0,1,1,1,1,0,1,0,1,1,1,2,0,1,1,1,0,0,3,0,1,1,0,2,1,1,1,1,0,0,0,0,1,3,0,1,1,2,1,0,1,1,3,0,0,4,1,2,1,0,0,1,0,0,1,0,0,2,1,1,1,0,1,0,1,1,0,1,0,1,0,0,0,0,3,2,0,1,0,0,0,0,1,0,1,0,0,1,1,1,2,2,0,1,0,2,2,0,1,0,2,0,1,2,3,1,1,1,1,1,1,0,1,2,2,0,0,0,0,2,2,1,0,1,2,0,0,0,0,0,0,2,2,0,1,0,0,0,2,0,1,0,0,1,1,1,1,0,0,0,1,0,0,1,0,1,3,0,0,1,0,0,2,1,0,0,0,1,1,0,1,0,1,0,1,0,0,0,0,0,0,0,1,0,0,1,0,0,1,1,0,0,2,0,4,3,0,3,0,0,1,0,1,0,0,0,0,1,0,0,1,1,0,3,1,1,1,1,1,0,4,1,2,0,0,0,1,2,1,0,0,1,1,2,0,0,1,0,1,1,1,0,1,0,1,2,0,0,1,0,0,0,0,1,1,0,0,0,0,0,0,1,0,1,0,1,1,0,1,1,0,2,1,1,0,0,1,1,0,0,0,0,0,0,0,0,1,1,0,0,0,0,1,1,0,0,2,0,0,2,1,0,1,0,0,4,3,1,1,1 +c,c.ome.zarr,0,0,0,drug,rep1,single,drug/rep1/c.ome.zarr,"drug/rep1/c.ome.zarr::z=0,c=0,t=0",uint16,16,100,399,360,360,100,399,500,2,1,1,3,5,1,1,3,2,2,4,3,0,1,0,3,0,2,3,0,1,1,1,0,0,2,0,4,1,2,0,1,0,3,0,2,1,2,0,1,2,3,2,1,1,3,0,2,1,2,1,0,1,2,0,0,1,0,3,2,0,0,2,0,1,2,1,1,1,1,0,0,0,0,3,3,0,1,2,0,3,0,1,1,2,1,1,3,1,1,1,1,0,1,3,2,1,0,1,2,1,3,1,0,1,1,1,3,1,2,1,0,1,2,1,0,0,0,1,1,1,2,0,1,1,0,5,2,2,3,3,2,2,1,3,3,1,2,1,1,2,0,0,0,0,0,1,1,2,0,0,0,2,3,1,0,2,2,1,2,0,1,1,2,1,1,3,2,1,1,2,1,1,0,1,0,1,2,3,1,3,2,1,0,1,0,0,1,1,1,1,0,1,3,0,1,1,0,2,1,0,3,0,1,0,4,1,0,1,1,0,5,3,0,2,3,1,2,1,0,0,1,1,0,0,0,2,1,1,2,1,0,0,0,1,1,1,1,0,0,2,2,2,0,0,1,1,0,0,1,3,1,0,0,1,1,0,2,2,1,0,1,1,1,0,1,0,2,2,1,2,0,1,3,0,0,4,1,2,2,1,2,1,1,1,0,1,2,0,0,1,3,3,1,0,2,2,1,0,1,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +c,c.ome.zarr,0,1,0,drug,rep1,single,drug/rep1/c.ome.zarr,"drug/rep1/c.ome.zarr::z=0,c=1,t=0",uint16,16,101,5093,360,360,101,5093,500,2,3,2,2,0,0,1,1,0,2,1,0,0,0,0,2,2,0,1,1,1,1,2,1,0,2,2,0,1,0,3,1,1,0,1,1,1,1,0,0,1,0,1,0,1,1,1,1,0,0,0,0,1,0,0,2,0,0,0,0,1,1,1,1,2,1,0,0,0,0,2,1,0,2,0,1,1,0,1,1,0,1,0,2,0,1,1,0,1,1,0,0,0,1,1,1,2,0,4,0,0,1,0,0,0,2,0,0,0,1,0,0,0,1,1,2,0,1,0,1,4,0,0,0,1,1,2,1,0,1,0,0,1,0,0,1,1,1,0,1,2,1,0,0,1,0,0,1,0,1,1,0,0,4,0,0,0,0,0,3,1,0,0,0,2,2,2,1,1,1,3,1,0,1,2,0,1,2,0,1,2,2,1,0,0,1,0,2,0,0,0,0,1,0,0,0,0,1,0,1,0,1,2,1,0,1,2,0,2,0,3,0,2,1,0,0,1,3,0,0,3,1,1,0,0,2,0,1,1,0,0,0,0,0,2,1,0,0,1,1,0,0,0,0,0,0,3,1,0,2,0,0,0,1,0,0,1,0,0,0,1,0,0,0,1,0,2,1,0,2,0,1,0,0,0,0,0,0,0,2,2,0,0,0,0,1,1,2,0,1,1,1,1,2,0,0,0,0,2,2,0,1,2,0,1,1,2,1,0,0,2,0,0,1,0,1,0,0,1,0,3,0,0,0,0,0,0,0,0,1,1,1,0,2,0,1,1,0,2,0,1,2,1,0,0,1,3,2,0,0,0,0,0,0,1,1,3,1,1,2,1,0,2,0,1,0,1,2,1,1,1,2,3,3,1,0,2,0,2,0,1,1,1,0,0,1,0,0,3,0,0,0,2,2,0,0,3,1,0,1,4,0,0,1,0,3,0,2,0,3,1,0,0,1,1,0,2,0,0,2,0,0,0,0,0,0,0,0,0,0,3,0,1,1,0,1,2,0,0,0,2,3,2,1,0,0,1,0,0,0,1,0,0,0,1,0,1,1,0,0,0,1,0,1,0,0,0,0,0,1,2,0,0,0,0,1,0,0,2,1,1,0,0,2,1,0,2,0,0,0,0,0,0,0,1,1,0,0,0,2,1 +c,c.ome.zarr,0,2,0,drug,rep1,single,drug/rep1/c.ome.zarr,"drug/rep1/c.ome.zarr::z=0,c=2,t=0",uint16,16,116,59958,360,360,116,59958,500,2,3,0,0,0,1,0,0,0,0,0,2,1,0,1,1,1,1,0,1,0,0,2,0,0,0,3,0,0,3,0,1,1,0,2,0,2,1,1,0,1,1,0,4,0,0,1,0,0,0,0,0,1,0,1,0,0,1,0,0,0,1,1,0,0,0,0,0,0,1,3,1,1,0,0,0,0,1,1,0,1,2,1,0,1,1,1,2,1,1,2,1,1,0,0,1,2,1,0,0,0,1,1,2,0,0,0,0,1,1,1,1,1,1,1,0,1,1,1,0,0,1,1,1,0,1,1,0,0,1,0,0,2,1,1,0,0,1,0,0,0,1,0,0,2,0,0,1,0,2,1,0,0,0,0,0,0,1,0,0,1,0,1,2,1,2,1,1,2,4,4,0,1,2,0,1,0,1,1,3,1,1,1,0,0,0,1,0,1,1,0,0,0,1,2,1,0,2,0,1,0,0,0,0,1,1,1,1,0,0,0,0,0,0,1,1,0,0,1,3,0,0,0,0,1,1,0,2,1,1,2,0,0,1,2,0,0,0,0,1,0,1,0,1,0,0,0,2,0,1,2,1,2,2,2,1,1,4,2,2,1,0,0,0,1,2,2,0,0,0,1,1,0,1,2,1,1,0,0,1,1,2,1,1,0,0,0,0,1,3,1,1,1,1,1,0,0,0,0,2,1,1,1,2,0,1,0,1,0,1,0,0,0,1,0,0,1,1,0,0,0,1,1,2,1,0,2,2,0,2,0,1,1,0,1,1,1,1,0,1,0,1,1,1,0,0,0,0,1,1,0,0,1,1,0,0,1,4,0,1,1,0,2,1,0,2,0,0,2,0,0,2,1,1,1,0,0,0,0,0,0,0,0,0,1,1,0,1,1,2,1,0,0,0,0,1,0,1,0,1,1,1,1,3,0,1,0,0,0,1,0,4,3,0,1,1,2,0,2,1,0,0,0,1,1,1,1,2,0,1,1,0,1,3,0,0,2,1,1,1,1,0,1,0,0,0,0,0,2,1,2,0,1,1,2,2,1,2,0,1,1,1,0,0,0,0,2,0,1,0,0,0,1,2,1,1,0,0,0,0,1,1,0,0,1,0,1,1,1,0,0,1,1,0,2,0,0,1,0,0,1 +c,c.ome.zarr,1,0,0,drug,rep1,single,drug/rep1/c.ome.zarr,"drug/rep1/c.ome.zarr::z=1,c=0,t=0",uint16,16,100,399,360,360,100,399,500,2,1,0,3,1,2,0,2,1,3,1,0,1,0,1,3,1,0,2,4,1,1,4,0,1,1,2,0,1,2,0,1,1,2,0,0,2,3,1,2,2,1,0,2,1,0,0,1,0,1,2,1,1,2,0,2,0,1,0,1,1,0,4,1,1,1,1,1,2,1,1,1,1,0,3,1,3,1,1,2,1,0,1,0,0,0,1,0,4,1,1,0,1,0,1,1,0,2,1,1,0,0,2,0,1,0,1,3,2,2,1,1,0,1,1,2,3,1,0,0,2,1,4,2,0,2,1,1,3,2,1,2,1,2,1,2,0,2,1,2,0,1,0,1,2,0,1,2,0,1,1,0,1,0,0,3,0,0,2,1,0,3,4,2,1,3,2,0,0,0,1,0,1,1,1,1,0,1,2,1,1,0,3,1,1,0,0,1,1,2,0,0,1,2,0,0,4,2,1,2,1,3,4,2,2,2,2,2,0,2,2,2,1,0,2,0,1,0,0,0,2,1,1,2,0,1,0,0,0,0,1,3,5,2,1,1,0,0,1,2,0,1,3,1,1,1,0,0,2,1,0,3,1,4,2,1,1,1,3,0,3,2,2,1,2,0,4,0,2,1,0,2,1,0,0,1,0,1,0,2,0,4,0,0,2,1,3,4,0,1,2,1,2,3,0,1,2,2,1,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 +c,c.ome.zarr,1,1,0,drug,rep1,single,drug/rep1/c.ome.zarr,"drug/rep1/c.ome.zarr::z=1,c=1,t=0",uint16,16,107,5094,360,360,107,5094,500,2,1,1,1,0,2,0,1,0,0,1,1,0,0,1,2,0,1,0,0,1,1,2,0,0,0,0,0,1,1,1,2,0,0,1,0,0,0,1,0,1,1,1,1,0,1,1,2,0,0,0,0,0,2,0,1,1,0,1,1,0,1,0,1,0,0,0,0,1,1,2,2,0,1,2,0,1,2,3,0,0,0,3,1,0,1,2,0,0,2,2,1,0,1,0,1,1,1,0,1,0,3,1,0,0,0,1,1,1,0,0,1,0,0,2,2,1,1,0,1,1,0,1,0,1,2,1,0,0,1,0,2,1,0,2,1,1,1,1,1,0,0,1,0,0,3,0,1,0,0,2,0,0,1,0,0,2,2,3,0,0,0,1,0,0,0,0,0,1,2,1,0,1,1,3,1,0,1,0,0,0,0,0,1,0,1,0,0,0,0,0,1,0,1,0,1,1,0,0,0,0,0,0,3,0,0,3,1,0,1,1,0,1,0,0,0,2,2,1,1,0,3,2,1,0,0,1,0,1,1,0,0,1,0,0,1,1,0,0,0,0,2,0,0,1,0,1,0,0,1,0,0,0,2,1,0,0,1,0,0,1,1,1,1,0,0,0,0,1,1,0,0,0,0,1,0,2,2,1,0,0,1,1,2,0,1,2,1,1,2,2,1,1,1,1,2,0,0,0,3,0,0,0,2,0,1,2,0,0,0,0,1,0,2,0,2,0,0,2,0,1,1,3,2,2,1,2,1,1,0,2,0,2,1,0,2,2,0,1,0,1,0,1,1,0,1,0,1,0,2,0,0,0,1,1,2,0,2,2,0,1,2,2,0,0,0,0,1,1,0,0,1,1,1,1,0,1,1,1,0,0,1,1,2,0,0,1,2,1,1,0,1,2,0,0,0,2,3,0,0,1,4,1,1,0,0,1,2,2,1,0,0,1,1,2,0,2,0,2,1,0,0,0,1,0,1,0,1,2,1,1,2,0,0,0,0,0,1,0,1,0,0,1,0,0,0,1,3,0,0,0,1,0,0,0,2,1,0,0,1,2,0,0,1,0,0,0,2,1,2,0,0,3,2,0,1,0,0,0,1,1,1,0,0,1,0,1,1,0,0,0,0,2,2,0,0,0,0,1,1,2 +c,c.ome.zarr,1,2,0,drug,rep1,single,drug/rep1/c.ome.zarr,"drug/rep1/c.ome.zarr::z=1,c=2,t=0",uint16,16,686,59962,360,360,686,59962,500,2,2,1,2,0,0,0,0,1,3,0,1,1,0,1,2,0,1,2,1,1,1,1,2,0,1,1,0,0,0,1,0,1,2,1,4,1,2,0,1,2,2,0,1,2,1,3,1,1,0,1,0,1,3,0,0,1,2,0,0,1,1,1,1,1,0,1,0,0,2,0,0,0,1,1,0,0,1,0,0,1,0,2,1,0,0,3,1,0,1,1,1,2,2,0,0,1,0,0,0,1,0,0,0,2,1,2,0,1,2,1,1,0,1,1,1,0,1,1,2,1,1,1,2,0,0,2,0,1,0,0,1,0,0,0,1,0,2,0,0,1,2,2,1,3,0,0,0,0,1,0,0,1,2,0,0,2,3,1,3,1,1,1,2,0,0,0,1,1,0,0,0,0,2,0,1,1,0,1,0,0,0,0,0,1,0,0,0,0,0,1,1,0,2,0,0,0,0,2,1,1,0,1,0,0,0,0,2,1,0,0,1,0,0,0,1,0,0,0,1,0,1,0,0,1,0,0,0,0,0,0,2,1,1,0,0,0,1,0,1,0,0,0,2,0,0,0,0,0,0,0,1,0,0,0,0,0,1,0,0,1,1,1,1,0,1,0,0,0,0,0,1,0,0,0,1,0,1,1,0,1,1,1,1,0,0,1,1,0,0,1,0,1,0,2,0,3,1,1,1,0,0,0,0,1,2,0,3,1,0,0,1,0,2,1,1,0,0,0,0,1,0,0,1,2,1,0,0,0,1,0,1,0,0,1,0,0,1,1,0,0,2,0,0,1,0,1,1,2,1,0,0,2,0,0,1,1,2,1,0,1,0,2,2,1,3,1,0,0,1,2,2,0,2,2,0,2,3,0,0,0,0,1,0,1,1,0,0,1,2,0,1,0,2,1,1,2,0,1,1,1,2,1,0,0,0,1,1,0,2,0,0,1,0,2,0,3,1,0,1,0,3,0,0,0,1,2,1,1,0,0,2,0,0,0,0,2,1,0,0,1,1,0,0,0,2,0,0,1,2,1,1,2,0,1,0,1,0,1,0,0,0,0,2,2,0,2,0,1,1,0,2,1,0,3,1,1,2,0,1,0,2,0,1,1,1,1,0,0,1,0,1,1,0,1,0,3,0,3,0,3 diff --git a/tests/simulation.py b/tests/simulation.py index f05e8c56..70309e89 100644 --- a/tests/simulation.py +++ b/tests/simulation.py @@ -54,7 +54,7 @@ def simulate_experiment( def make_records(planes, bin_count=None): return [ - PlaneRecord(path, *labels, histogram=compute_plane_histogram(pixels, bin_count)) + PlaneRecord(path, *labels, histograms=compute_plane_histogram(pixels, bin_count)) for labels, path, pixels in planes ] diff --git a/tests/test_axes.py b/tests/test_axes.py index 8230afdf..f7c0d9d5 100644 --- a/tests/test_axes.py +++ b/tests/test_axes.py @@ -3,9 +3,11 @@ from morphobricks_preprocessing import ( AxisMapping, + MetadataReport, Severity, TiffMetadata, check_mapping, + check_metadata, inspect_images, prefill_experiment_mapping, prefill_mapping, @@ -230,12 +232,36 @@ def test_missing_axis_blocks_until_confirmed(): def test_confirmed_missing_axis_is_read_as_singleton(): - files = [meta("a.tif", (2, 3, 8, 8), "ZCYX"), meta("b.tif", (2, 8, 8), "ZYX")] - review = review_axes(files, prefill_mapping("ZCYX"), missing_as_singleton=True) + files = [meta("a.tif", (2, 3, 8, 8), "ZTYX"), meta("b.tif", (2, 8, 8), "ZYX")] + review = review_axes(files, prefill_mapping("ZTYX"), missing_as_singleton=True) assert review.ok b = review.file_mappings["b.tif"] - assert b == AxisMapping("ZYX", "ZYX", singleton_roles=("C",)) - assert b.plane_roles == ("Z", "C") + assert b == AxisMapping("ZYX", "ZYX", singleton_roles=("T",)) + assert b.plane_roles == ("Z", "T") + + +def test_confirmed_missing_channel_axis_counts_as_one_channel(): + # Read as one channel, b.tif differs from the three channels of a.tif (N43). + files = [meta("a.tif", (2, 3, 8, 8), "ZCYX"), meta("b.tif", (2, 8, 8), "ZYX")] + review = review_axes(files, prefill_mapping("ZCYX"), missing_as_singleton=True) + assert review.file_mappings["b.tif"].singleton_roles == ("C",) + assert codes(review.issues) == ["different_channel_counts"] + found = issue(review.issues, "different_channel_counts") + assert found.severity is Severity.ERROR and found.paths == ("a.tif",) + + +def test_channel_count_follows_the_confirmed_channel_axis(): + # The header suggests one channel for an unlabelled axis; the confirmed role decides. + files = [meta("a.tif", (2, 8, 8), "IYX"), meta("b.tif", (3, 8, 8), "IYX")] + report = MetadataReport(tuple(files), tuple(check_metadata(files))) + assert "different_channel_counts" not in codes(report.issues) + as_time = review_axes(report, prefill_mapping("IYX").with_role(0, "T")) + assert as_time.ok + as_channel = review_axes(report, prefill_mapping("IYX").with_role(0, "C")) + assert codes(as_channel.combined_issues(report)) == [ + "different_channel_counts", + "different_plane_counts", + ] def test_several_missing_axes_are_resolved(): @@ -286,20 +312,24 @@ def test_review_from_metadata_report_replaces_missing_axes(tmp_path): write_tiff(tmp_path / "b.tif", (2, 6, 7), axes="ZYX") write_file(tmp_path / "bad.tif", b"garbage") report = inspect_images(tmp_path, ["a.tif", "b.tif", "bad.tif"]) - # One image has 3 channels and the other 1, which is also worth a warning. - assert codes(report.issues) == ["unreadable_tiff", "missing_axes", "different_plane_counts"] - mapping = prefill_experiment_mapping(report) - - unconfirmed = review_axes(report, mapping) - assert codes(unconfirmed.combined_issues(report)) == [ + # One image has 3 channels and the other 1: an error until the axes say otherwise. + assert codes(report.issues) == [ "unreadable_tiff", "missing_axes", - "different_plane_counts", + "different_channel_counts", ] + mapping = prefill_experiment_mapping(report) + + unconfirmed = review_axes(report, mapping) + assert codes(unconfirmed.combined_issues(report)) == ["unreadable_tiff", "missing_axes"] confirmed = review_axes(report, mapping, missing_as_singleton=True) - assert confirmed.ok - assert codes(confirmed.combined_issues(report)) == ["unreadable_tiff", "different_plane_counts"] + assert codes(confirmed.combined_issues(report)) == [ + "unreadable_tiff", + "different_channel_counts", + ] + as_time = review_axes(report, mapping.with_role(1, "T"), missing_as_singleton=True) + assert codes(as_time.combined_issues(report)) == ["unreadable_tiff"] def test_combined_issues_put_errors_first(tmp_path): diff --git a/tests/test_cli.py b/tests/test_cli.py index e1a46d6e..c5bf555b 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -71,10 +71,10 @@ def test_run_writes_the_table(experiment, capsys): assert main(["run", str(experiment), "--axes", "ZCYX"]) == 0 out = capsys.readouterr().out assert "[1/3] ctrl/rep1/a.tif" in out - assert "Wrote 18 planes" in out + assert "Wrote 6 planes" in out output = experiment.parent / "exp_MorphoBricks_Preprocessing" records = read_plane_histograms(output / "plane_histograms.csv") - assert {(r.z, r.c) for r in records} == {(z, c) for z in range(2) for c in range(3)} + assert {(r.z, r.channel_count) for r in records} == {(0, 3), (1, 3)} def test_run_options_reach_the_run(experiment, tmp_path, capsys): @@ -119,11 +119,26 @@ def test_run_without_readable_images(tmp_path, capsys): def test_missing_axes_option(tmp_path, capsys): root = tmp_path / "exp" - write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZCYX") + write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZTYX") write_data(root / "b.tif", np.zeros((2, 8, 8), np.uint8), axes="ZYX") confirm_design(root) + assert main(["run", str(root), "--axes", "ZTYX"]) == 1 + assert main(["run", str(root), "--axes", "ZTYX", "--missing-as-singleton"]) == 0 + + +def test_different_channel_counts_stop_check_and_run(tmp_path, capsys): + root = tmp_path / "exp" + write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZCYX") + write_data(root / "b.tif", np.zeros((2, 2, 8, 8), np.uint8), axes="ZCYX") + write_data(root / "c.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZCYX") + assert main(["check", str(root)]) == 1 + out = capsys.readouterr().out + assert "different numbers of channels: 3 channels (2 images), 2 channels (1 image)" in out + confirm_design(root) assert main(["run", str(root), "--axes", "ZCYX"]) == 1 - assert main(["run", str(root), "--axes", "ZCYX", "--missing-as-singleton"]) == 0 + out = capsys.readouterr().out + assert "different numbers of channels" in out and "b.tif" in out + assert "Nothing was written" in out def test_axes_are_required_for_run(experiment): diff --git a/tests/test_manifest.py b/tests/test_manifest.py index 32f64fbd..690f96b6 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -62,7 +62,7 @@ def test_written_json_is_readable_and_stable(tmp_path): raw = path.read_bytes() assert b"\r\n" not in raw and raw.endswith(b"\n") data = json.loads(raw) - assert data["manifest_version"] == "1" and data["schema_version"] == "2" + assert data["manifest_version"] == "1" and data["schema_version"] == "3" assert data["axis_mapping"] == {"axes": "ZYX", "roles": ["Z", "Y", "X"]} assert data["settings"]["exclude"] == ["a.tif", "b.tif"] assert data["labels"] == [ diff --git a/tests/test_metadata.py b/tests/test_metadata.py index 019adcf2..1fc5691d 100644 --- a/tests/test_metadata.py +++ b/tests/test_metadata.py @@ -314,7 +314,11 @@ def test_missing_axes_with_two_longest_arrangements_is_inconsistent(): meta("b.tif", (3, 2, 6, 7), axes="CZYX"), meta("c.tif", (6, 7), axes="YX"), ] - assert codes(check_metadata(files)) == ["inconsistent_dimensions", "different_plane_counts"] + assert codes(check_metadata(files)) == [ + "inconsistent_dimensions", + "different_channel_counts", + "different_plane_counts", + ] def test_different_sizes_are_a_warning(tmp_path): @@ -453,9 +457,13 @@ def test_different_plane_counts(tmp_path): assert "different_sizes" not in codes(report.issues) found = issue(report.issues, "different_plane_counts") assert found.severity is Severity.WARNING and found.paths == ("c.tif",) - assert "channels: 3 (2 images), 2 (1 image)" in found.message assert "time points: 2 (2 images), 5 (1 image)" in found.message - assert found.message.index("channels") < found.message.index("time points") + # Channels are compared on their own, as an error (N43). + assert "channels" not in found.message + channels = issue(report.issues, "different_channel_counts") + assert channels.severity is Severity.ERROR and channels.paths == ("c.tif",) + assert "3 channels (2 images), 2 channels (1 image)" in channels.message + assert "separate experiment" in channels.action def test_plane_counts_compare_rgb_samples_and_unlabelled_dimensions(): @@ -465,7 +473,10 @@ def test_plane_counts_compare_rgb_samples_and_unlabelled_dimensions(): meta("q3.tif", (3, 8, 8), axes="QYX"), ] found = issue(check_metadata(files), "different_plane_counts") - assert "channels: 1 (1 image), 2 (1 image), 3 (1 image)" in found.message assert "planes along Q: 1 (2 images), 3 (1 image)" in found.message + # RGB samples count as channels when the image has no channel axis. + channels = issue(check_metadata(files), "different_channel_counts") + assert "1 channel (1 image), 2 channels (1 image), 3 channels (1 image)" in channels.message + assert channels.paths == ("rgb.tif", "two.tif") same = [meta("a.tif", (3, 8, 8), axes="ZYX"), meta("b.tif", (3, 8, 8), axes="ZYX")] assert check_metadata(same) == [] diff --git a/tests/test_moved.py b/tests/test_moved.py index 69a683f4..e1f1a5c5 100644 --- a/tests/test_moved.py +++ b/tests/test_moved.py @@ -29,8 +29,10 @@ def new_object(module, name): @pytest.mark.parametrize("module", MOVED_MODULES) -def test_moved_modules_match_the_normalization_contract(module): - # Every public name of the new module is forwarded, and nothing else. +def test_moved_names_still_exist_in_the_normalization_brick(module): + # Every name that moved (normalization contract §1.1) is forwarded to the brick's + # current object. Names the brick added later (Milestone 2b: channel_column, + # WeightedHistogram, ...) were never here, so they are not forwarded. new = importlib.import_module(f"morphobricks_normalization.{module}") defined = { name @@ -39,7 +41,7 @@ def test_moved_modules_match_the_normalization_contract(module): and not isinstance(value, type(np)) and getattr(value, "__module__", new.__name__) == new.__name__ } - assert set(_moved.MOVED_NAMES[module]) == defined + assert set(_moved.MOVED_NAMES[module]) <= defined def test_top_level_names_are_the_ones_the_package_exported(): @@ -88,7 +90,7 @@ def test_from_imports_keep_working(): from morphobricks_preprocessing.csv_io import SCHEMA_VERSION with pytest.warns(DeprecationWarning, match="morphobricks_preprocessing.PlaneRecord"): from morphobricks_preprocessing import PlaneRecord - assert SCHEMA_VERSION == "2" + assert SCHEMA_VERSION == "3" # the brick's current value (Milestone 2b) from morphobricks_normalization import PlaneRecord as new_record assert PlaneRecord is new_record diff --git a/tests/test_notebook_ui.py b/tests/test_notebook_ui.py index f9502d98..a52d949f 100644 --- a/tests/test_notebook_ui.py +++ b/tests/test_notebook_ui.py @@ -191,6 +191,27 @@ def test_consistency_warnings_are_in_their_own_panel(experiment): assert not ui.is_visible(nb.consistency_card) +def test_different_channel_counts_are_an_error_in_the_consistency_panel(experiment): + rng = np.random.default_rng(1) + three = rng.integers(0, 9, (3, 3, 16, 16), dtype=np.uint8) + write_data(experiment / "drug/rep1/b.tif", three, "ZCYX") + nb = checked(experiment) + assert "mb-error'>1 error" in nb.consistency_counts.value + assert "different channel counts" in nb.consistency_issues.value + assert "2 channels (3 images), 3 channels (1 image)" in nb.consistency_issues.value + assert "drug/rep1/b.tif" in nb.consistency_issues.value + assert "different channel counts" not in nb.folder_issues.value + assert "Action required" in nb.folder_status.value + # Confirming the axis roles does not lift it: the run stays locked. + nb.labels_confirmed.button.click() + confirm_repeats(nb) + nb.preview_button.click() + nb.preview_checked.button.click() + nb.confirm_button.click() + assert nb.run_button.disabled and nb.run() is None + assert "fix any errors" in nb.readiness.value + + def test_next_steps_follow_confirmation_and_folder_changes(experiment): nb = checked(experiment) assert "Check the labels in Step 1" in nb.folder_next.value @@ -546,11 +567,11 @@ def test_preview_and_roles_with_an_ome_zarr_image(tmp_path): def test_missing_axes_card(tmp_path): root = tmp_path / "exp" - write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZCYX") + write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZTYX") write_data(root / "b.tif", np.zeros((2, 8, 8), np.uint8), axes="ZYX") nb = checked(root) assert ui.is_visible(nb.missing_card) - assert "1 image with axes ZYX lack C (channel)" in nb.missing_note.value + assert "1 image with axes ZYX lack T (time)" in nb.missing_note.value nb.preview_button.click() nb.preview_checked.button.click() assert nb.confirm_button.disabled @@ -561,7 +582,7 @@ def test_missing_axes_card(tmp_path): @pytest.mark.parametrize("blocking_file", [False, True]) def test_confirmed_missing_axes_allow_running_unless_other_errors_remain(tmp_path, blocking_file): root = tmp_path / "exp" - write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZCYX") + write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZTYX") write_data(root / "b.tif", np.zeros((2, 8, 8), np.uint8), axes="ZYX") if blocking_file: write_file(root / "bad.tif", b"garbage") @@ -582,12 +603,12 @@ def test_confirmed_missing_axes_allow_running_unless_other_errors_remain(tmp_pat result = nb.run() assert result.written rows = read_plane_histograms(result.csv_path) - assert [(r.z, r.c) for r in rows if r.source_relative_path == "b.tif"] == [(0, 0), (1, 0)] + assert [(r.z, r.t) for r in rows if r.source_relative_path == "b.tif"] == [(0, 0), (1, 0)] def test_undoing_missing_axes_confirmation_blocks_running_again(tmp_path): root = tmp_path / "exp" - write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZCYX") + write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZTYX") write_data(root / "b.tif", np.zeros((2, 8, 8), np.uint8), axes="ZYX") nb = checked(root) nb.labels_confirmed.button.click() @@ -1011,7 +1032,7 @@ def test_full_run_and_statistics(experiment, tmp_path): nb.bins.value = 64 nb.run_button.click() assert nb.result.written - assert "Done: 18 planes in the table" in nb.run_status.value + assert "Done: 9 planes in the table" in nb.run_status.value assert "Step 4 (optional)" in nb.run_next.value assert "What next?" in nb.run_next.value assert "Calculate statistics" in nb.run_next.value @@ -1019,7 +1040,7 @@ def test_full_run_and_statistics(experiment, tmp_path): assert nb.progress.value == nb.progress.max == 3 assert nb.progress.bar_style == "success" assert nb.run_button.description == " Compute histograms" and not nb.run_button.disabled - assert "Wrote 18 planes" in nb.details.value # the log reaches "Technical details" + assert "Wrote 9 planes" in nb.details.value # the log reaches "Technical details" assert read_manifest(tmp_path / "out" / "run_manifest.json").settings.bin_count == 64 # Step 4 is optional: the new table is selected but nothing is calculated yet. assert nb.csv_file.value == str(tmp_path / "out" / "plane_histograms.csv") @@ -1138,6 +1159,16 @@ def test_statistics_by_level_and_percentiles(experiment): assert "p50" in nb.statistics.value +def test_statistics_explain_a_table_of_an_earlier_version(experiment, tmp_path): + nb = checked(experiment) + nb.csv_file.value = str(Path(__file__).parent / "data" / "v2_float32.csv") + nb.statistics_button.click() + assert "The table cannot be read" in nb.stats_status.value + assert "schema version 2" in nb.stats_status.value + assert "Run the preprocessing brick again" in nb.stats_status.value + assert not ui.is_visible(nb.statistics_card) + + def test_statistics_need_a_table_and_valid_percentiles(experiment): nb = checked(experiment) nb.show_statistics() diff --git a/tests/test_planes.py b/tests/test_planes.py index ed606e5a..e33a4900 100644 --- a/tests/test_planes.py +++ b/tests/test_planes.py @@ -16,6 +16,7 @@ inspect_images, inspect_tiff, iter_planes, + iter_xyc_planes, plane_layout, prefill_mapping, read_plane, @@ -176,17 +177,29 @@ def test_rgb_samples_are_read_as_channels(tmp_path, page_reads, kwargs): def test_confirmed_missing_axis_gets_coordinate_zero(tmp_path): - full = write_data(tmp_path / "a.tif", random_data((2, 3, 6, 7), "uint8"), axes="ZCYX") + full = write_data(tmp_path / "a.tif", random_data((2, 3, 6, 7), "uint8"), axes="ZTYX") data = random_data((2, 6, 7), "uint8", seed=1) part = write_data(tmp_path / "b.tif", data, axes="ZYX") report = inspect_images(tmp_path, ["a.tif", "b.tif"]) - review = review_axes(report, prefill_mapping("ZCYX"), missing_as_singleton=True) + review = review_axes(report, prefill_mapping("ZTYX"), missing_as_singleton=True) assert review.ok planes = list(iter_planes(part, review.file_mappings["b.tif"])) - assert [p.coordinates for p in planes] == [{"z": z, "c": 0, "t": None} for z in range(2)] + assert [p.coordinates for p in planes] == [{"z": z, "c": None, "t": 0} for z in range(2)] np.testing.assert_array_equal(planes[1].pixels, data[1]) assert len(list(iter_planes(full, review.file_mappings["a.tif"]))) == 6 + rows = list(iter_xyc_planes(part, review.file_mappings["b.tif"])) + assert [(p.z, p.t, p.channel_count) for p in rows] == [(0, 0, 1), (1, 0, 1)] + + +def test_missing_channel_axis_gives_one_channel(tmp_path): + data = random_data((2, 6, 7), "uint8") + path = write_data(tmp_path / "b.tif", data, axes="ZYX") + mapping = AxisMapping("ZYX", "ZYX", singleton_roles=("C",)) + rows = list(iter_xyc_planes(path, mapping)) + assert [(p.z, p.t, p.channel_count) for p in rows] == [(0, None, 1), (1, None, 1)] + plane = rows[0].channel(0) + assert plane.coordinates == {"z": 0, "c": 0, "t": None} def test_pyramidal_file_is_read_at_full_resolution(tmp_path): @@ -306,11 +319,26 @@ def test_memory_estimate_covers_page_plane_and_histogram(tmp_path): assert layout.page_shape == (60, 70, 3) assert layout.page_nbytes == 60 * 70 * 3 histogram = estimate_histogram_memory(60 * 70, np.dtype("uint8")) + # The finished bin counts of the plane's three channels are kept for its row. + assert layout.channel_count == 3 + finished = 3 * 8 * 256 assert layout.memory_estimate == ( - layout.page_nbytes + layout.encoded_nbytes + 60 * 70 + histogram + layout.page_nbytes + layout.encoded_nbytes + 60 * 70 + histogram + finished ) +def test_memory_estimate_covers_every_channel_of_a_plane(tmp_path): + path = write_data(tmp_path / "a.tif", random_data((4, 400, 500), "uint16"), axes="CYX") + channels = plane_layout(path, AxisMapping("CYX", "CYX")) + times = plane_layout(path, AxisMapping("CYX", "TYX")) + assert (channels.channel_count, times.channel_count) == (4, 1) + assert channels.memory_estimate - times.memory_estimate == 3 * 8 * 65_536 + # One decision for every channel: a limit between the two reads the channels in blocks. + limit = times.memory_estimate + assert plane_layout(path, AxisMapping("CYX", "CYX"), memory_limit=limit).in_blocks + assert not plane_layout(path, AxisMapping("CYX", "TYX"), memory_limit=limit).in_blocks + + def test_memory_estimate_counts_the_whole_page_of_a_volume(tmp_path): data = random_data((4, 32, 32), "uint16") path = write_data(tmp_path / "a.tif", data, tile=(16, 16, 16), volumetric=True) @@ -364,7 +392,8 @@ def test_memory_estimate_depends_on_bin_count(tmp_path): mapping = prefill_mapping("YX") small = plane_layout(path, mapping, bin_count=2).memory_estimate large = plane_layout(path, mapping, bin_count=65_536).memory_estimate - assert large - small == 4 * 8 * (65_536 - 2) + # Temporary arrays of the histogram, and the finished counts of its one channel. + assert large - small == (4 * 8 + 8) * (65_536 - 2) def test_check_planes_reports_planes_over_the_limit(tmp_path): @@ -520,6 +549,78 @@ def test_planes_in_blocks_match_the_source(tmp_path, segment_reads, shape, axes, assert segment_reads # read piece by piece, never as whole pages +# XYC planes: one row of the table each (follow-up P3b) --------------------------------- + + +def expected_xyc(data, roles): + """Reference: the channel planes of every (z, t) position, in channel order.""" + planes = {} + for coords, pixels in expected_planes(data, roles): + planes.setdefault((coords["z"], coords["t"]), []).append(pixels) + return planes + + +def assert_xyc_planes_match(planes, expected, bins=None): + """Every XYC plane's histograms equal those of its channel planes read whole.""" + seen = [] + for plane in planes: + channels = expected[(plane.z, plane.t)] + assert plane.channel_count == len(channels) + histograms = plane.histograms(bins) + assert len(histograms) == len(channels) + for hist, pixels in zip(histograms, channels, strict=True): + whole = compute_plane_histogram(pixels.astype(pixels.dtype.newbyteorder("=")), bins) + assert (hist.pixel_min, hist.pixel_max) == (whole.pixel_min, whole.pixel_max) + assert hist.total_pixel_count == whole.total_pixel_count + np.testing.assert_array_equal(hist.counts, whole.counts) + seen.append((plane.z, plane.t)) + assert sorted(seen, key=str) == sorted(expected, key=str) + + +@pytest.mark.parametrize(("shape", "axes", "kwargs", "roles"), LAYOUTS) +def test_xyc_planes_hold_every_channel(tmp_path, page_reads, shape, axes, kwargs, roles): + data = random_data(shape, "uint16") + path = write_data(tmp_path / "a.tif", data, axes=axes, **kwargs) + info = inspect_tiff(path) + mapping = AxisMapping(info.axes, roles) if roles else prefill_mapping(info.axes) + assert_xyc_planes_match(iter_xyc_planes(path, mapping), expected_xyc(data, mapping.roles)) + # Every page is decoded once, also when the channels lie in different pages. + with tifffile.TiffFile(path) as tif: + page_count = len(tif.series[0].pages) + assert sorted(page_reads) == list(range(page_count)) + + +def test_rgb_page_is_decoded_once_for_its_three_channels(tmp_path, page_reads): + data = random_data((2, 6, 7, 3), "uint8") + path = write_data(tmp_path / "a.tif", data, axes="ZYXS", photometric="rgb") + mapping = prefill_mapping("ZYXS") + seen = [] + for plane in iter_xyc_planes(path, mapping): + seen.append((plane.z, plane.channel_count)) + for c in range(3): + np.testing.assert_array_equal(plane.channel(c).pixels, data[plane.z, :, :, c]) + with pytest.raises(IndexError, match="3 channel"): + plane.channel(3) + assert seen == [(0, 3), (1, 3)] + assert page_reads == [0, 1] + + +@pytest.mark.parametrize(("shape", "axes", "kwargs", "roles"), BLOCK_LAYOUTS) +def test_xyc_planes_in_blocks_give_the_histograms_read_whole( + tmp_path, segment_reads, shape, axes, kwargs, roles +): + dtype = "uint8" if "photometric" in kwargs else "uint16" + data = random_data(shape, dtype) + path = write_data(tmp_path / "a.tif", data, axes=axes, **kwargs) + info = inspect_tiff(path) + mapping = AxisMapping(info.axes, roles) if roles else prefill_mapping(info.axes) + limit = smallest_limit(path, mapping) + assert plane_layout(path, mapping, memory_limit=limit).in_blocks + planes = iter_xyc_planes(path, mapping, memory_limit=limit) + assert_xyc_planes_match(planes, expected_xyc(data, mapping.roles)) + assert segment_reads + + @pytest.mark.parametrize("dtype", DTYPES + [">u2", ">i4", ">f4", ">f8"]) @pytest.mark.parametrize("kwargs", [{}, {"compression": "zlib", "rowsperstrip": 3}]) def test_every_dtype_in_blocks_gives_the_same_histograms(tmp_path, segment_reads, dtype, kwargs): diff --git a/tests/test_report.py b/tests/test_report.py index 543d91cd..9b2072d2 100644 --- a/tests/test_report.py +++ b/tests/test_report.py @@ -101,7 +101,8 @@ def test_preflight_summary_with_conflicting_axes(tmp_path): write_tiff(tmp_path / "b.tif", (3, 6, 7), axes="CYX") text = preflight_summary(preflight(tmp_path)) assert "No axis mapping can be suggested" in text - assert "1 error" in text + # The arrangement differs, and so does the number of channels (3 against 1). + assert "2 errors" in text def test_run_summaries(experiment): diff --git a/tests/test_run.py b/tests/test_run.py index e18f52f5..f55cfadc 100644 --- a/tests/test_run.py +++ b/tests/test_run.py @@ -74,13 +74,13 @@ def relative(self, root): def reads(monkeypatch): """Record which files have their pixels read.""" paths = Reads() - original = run_module.iter_planes + original = run_module.iter_xyc_planes def spy(path, *args, **kwargs): paths.append(path) return original(path, *args, **kwargs) - monkeypatch.setattr(run_module, "iter_planes", spy) + monkeypatch.setattr(run_module, "iter_xyc_planes", spy) return paths @@ -125,7 +125,7 @@ def test_labels_and_coordinates_in_the_table(experiment): result = run_experiment(experiment, ZYX) records = read_plane_histograms(result.csv_path) assert [r.plane_id for r in records] == [ - f"{rel}::z={z},c=0,t=0" for rel in sorted(IMAGES) for z in (0, 1) + f"{rel}::z={z},t=0" for rel in sorted(IMAGES) for z in (0, 1) ] by_path = {r.source_relative_path: r for r in records} short = by_path["treated/bio_rep1/img_01.tif"] @@ -135,7 +135,7 @@ def test_labels_and_coordinates_in_the_table(experiment): SINGLE, ) assert {r.z for r in records} == {0, 1} - assert all(r.c == 0 and r.t == 0 for r in records) + assert all(r.t == 0 and r.channel_count == 1 for r in records) def test_table_matches_the_pixels(experiment): @@ -144,11 +144,14 @@ def test_table_matches_the_pixels(experiment): pixels = {rel: tifffile.imread(experiment / rel) for rel in IMAGES} for record in records: plane = pixels[record.source_relative_path][record.z] - assert record.histogram.pixel_min == plane.min() - assert record.histogram.pixel_max == plane.max() - assert record.histogram.total_pixel_count == plane.size - assert record.histogram.counts.sum() == record.histogram.valid_pixel_count - assert record.histogram.is_exact + assert record.channel_histogram(0).pixel_min == plane.min() + assert record.channel_histogram(0).pixel_max == plane.max() + assert record.channel_histogram(0).total_pixel_count == plane.size + assert ( + record.channel_histogram(0).counts.sum() + == record.channel_histogram(0).valid_pixel_count + ) + assert record.channel_histogram(0).is_exact # Pooled counts reproduce the pooled-pixel statistics exactly at every level. for level in ("condition", "biological_repeat", "technical_repeat", "image"): for members in group_records(records, level).values(): @@ -223,7 +226,7 @@ def test_every_row_has_its_position_id(experiment): } # Position key plus plane coordinates identifies every row. rows = [ - (r.condition, r.biological_repeat, r.technical_repeat, r.position_id, r.z, r.c, r.t) + (r.condition, r.biological_repeat, r.technical_repeat, r.position_id, r.z, r.t) for r in records ] assert len(set(rows)) == len(rows) == 2 * len(IMAGES) @@ -232,7 +235,7 @@ def test_every_row_has_its_position_id(experiment): rel: rel.rsplit("/", 1)[-1].removesuffix(".tif") for rel in IMAGES } data = json.loads(result.manifest_path.read_text(encoding="utf-8")) - assert data["manifest_version"] == "1" and data["schema_version"] == "2" + assert data["manifest_version"] == "1" and data["schema_version"] == "3" assert all("position_id" in f for f in data["files"]) @@ -329,7 +332,9 @@ def test_only_changed_file_is_read(experiment, reads): after = {r.plane_id: r for r in read_plane_histograms(result.csv_path)} assert after.keys() == before.keys() for plane_id, record in after.items(): - same = np.array_equal(record.histogram.counts, before[plane_id].histogram.counts) + same = np.array_equal( + record.channel_histogram(0).counts, before[plane_id].channel_histogram(0).counts + ) assert same == (not plane_id.startswith(changed)) @@ -382,7 +387,9 @@ def test_changed_bin_count_rebuilds(experiment, reads): result = run_experiment(experiment, ZYX, bin_count=100) assert result.rebuild_reason == "the bin count changed from 256 to 100" assert len(reads) == len(IMAGES) - assert {r.histogram.bin_count for r in read_plane_histograms(result.csv_path)} == {100} + assert {r.channel_histogram(0).bin_count for r in read_plane_histograms(result.csv_path)} == { + 100 + } def test_changed_axis_mapping_rebuilds(experiment): @@ -655,16 +662,52 @@ def broken(series, index, layout, path): def test_missing_axes_need_confirmation(tmp_path): root = tmp_path / "exp" - write_data(root / "a.tif", stack(0, n_z=2)[:, None].repeat(2, axis=1), axes="ZCYX") + write_data(root / "a.tif", stack(0, n_z=2)[None].repeat(2, axis=0), axes="TZYX") write_data(root / "b.tif", stack(1, n_z=2), axes="ZYX") - mapping = prefill_mapping("ZCYX") + mapping = prefill_mapping("TZYX") blocked = run_experiment(root, mapping) assert codes(blocked.issues) == ["missing_axes", "different_plane_counts"] result = run_experiment(root, mapping, missing_as_singleton=True) assert result.written rows = [r for r in read_plane_histograms(result.csv_path) if r.source_relative_path == "b.tif"] - assert [(r.z, r.c) for r in rows] == [(0, 0), (1, 0)] - assert read_manifest(result.manifest_path).file("b.tif").singleton_roles == ("C",) + assert [(r.z, r.t) for r in rows] == [(0, 0), (1, 0)] + assert read_manifest(result.manifest_path).file("b.tif").singleton_roles == ("T",) + + +def test_missing_channel_axis_next_to_several_channels_is_an_error(tmp_path, reads): + # A file without a C axis has one channel; the others have two (N43). + root = tmp_path / "exp" + write_data(root / "a.tif", stack(0, n_z=2)[:, None].repeat(2, axis=1), axes="ZCYX") + write_data(root / "b.tif", stack(1, n_z=2), axes="ZYX") + write_data(root / "c.tif", stack(2, n_z=2), axes="ZYX") + pre = preflight(root) + assert issue(pre.issues, "different_channel_counts").paths == ("a.tif",) + result = run_experiment(root, prefill_mapping("ZCYX"), missing_as_singleton=True) + found = issue(result.issues, "different_channel_counts") + assert found.severity is Severity.ERROR and found.paths == ("a.tif",) + assert "1 channel (2 images), 2 channels (1 image)" in found.message + assert "separate experiment" in found.action + assert not result.written and reads == [] + + +def test_one_row_per_xyc_plane_with_every_channel(tmp_path): + root = tmp_path / "exp" + data = np.stack([stack(seed, n_z=3) for seed in range(2)], axis=1) # Z, C, Y, X + write_data(root / "a.tif", data, axes="ZCYX") + result = run_experiment(root, prefill_mapping("ZCYX")) + assert result.written and result.plane_count == 3 + rows = read_plane_histograms(result.csv_path) + assert [r.plane_id for r in rows] == [f"a.tif::z={z},t=0" for z in range(3)] + for row in rows: + assert row.channel_count == 2 + for c in range(2): + plane = data[row.z, c] + hist = row.channel_histogram(c) + assert (hist.pixel_min, hist.pixel_max) == (plane.min(), plane.max()) + np.testing.assert_array_equal( + hist.counts, np.bincount(plane.ravel() - plane.min(), minlength=256)[:256] + ) + assert read_manifest(result.manifest_path).file("a.tif").plane_count == 3 # Exclusions (open question 6) --------------------------------------------------------- @@ -712,7 +755,10 @@ def test_non_finite_pixels_are_a_warning(tmp_path): found = issue(result.issues, "non_finite_pixels") assert "4 pixels" in found.message and found.paths == ("a.tif",) first = read_plane_histograms(result.csv_path)[0] - assert first.histogram.total_pixel_count - first.histogram.valid_pixel_count == 4 + assert ( + first.channel_histogram(0).total_pixel_count - first.channel_histogram(0).valid_pixel_count + == 4 + ) def test_empty_plane_is_a_warning(tmp_path): @@ -724,7 +770,7 @@ def test_empty_plane_is_a_warning(tmp_path): assert result.written assert issue(result.issues, "no_finite_pixels").paths == ("a.tif",) rows = read_plane_histograms(result.csv_path) - assert rows[1].histogram.is_empty and rows[1].histogram.pixel_min is None + assert rows[1].channel_histogram(0).is_empty and rows[1].channel_histogram(0).pixel_min is None def test_no_usable_planes_is_an_error(tmp_path): @@ -745,7 +791,7 @@ def test_every_integer_dtype_runs(tmp_path, dtype): result = run_experiment(root, ZYX) assert result.written rows = read_plane_histograms(result.csv_path) - assert [r.histogram.pixel_max for r in rows] == [data[0].max(), data[1].max()] + assert [r.channel_histogram(0).pixel_max for r in rows] == [data[0].max(), data[1].max()] # Preflight ---------------------------------------------------------------------------- diff --git a/tests/test_schema_v2_equivalence.py b/tests/test_schema_v2_equivalence.py new file mode 100644 index 00000000..c98adeb2 --- /dev/null +++ b/tests/test_schema_v2_equivalence.py @@ -0,0 +1,195 @@ +"""Schema version 3 holds exactly what version 2 held, channel by channel (follow-up P3b). + +The fixture in ``tests/data`` was written by the code before P3b (commit +``55f1074`` with morphobricks-normalization v0.1.0, ``make_v2_fixture.py``): +simulated pixels and the version-2 ``plane_histograms.csv`` its ``run_experiment`` +wrote for them, one row per plane and channel. Here the same images go through +the current code, read whole and in blocks; every value of every channel must be +the same text as in the version-2 row of that channel. +""" + +import csv +import json +import shutil + +import numpy as np +import pytest +from morphobricks_normalization.csv_io import CsvFormatError, read_plane_histograms +from morphobricks_normalization.histogram import PlaneHistogram +from morphobricks_normalization.records import PlaneRecord + +from morphobricks_preprocessing import ( + plane_layout, + prefill_mapping, + preflight, + read_manifest, + review_axes, + run_experiment, +) +from morphobricks_preprocessing.manifest import file_sha256 +from morphobricks_preprocessing.report import group_summaries, group_table +from v2_fixture import DATA, EXPERIMENTS, PIXELS, write_experiment + +# Version-2 column -> version-3 column of channel c. +CHANNEL_COLUMNS = { + "pixel_min": "Intensity_MinIntensity_C{c}", + "pixel_max": "Intensity_MaxIntensity_C{c}", + "valid_pixel_count": "Histogram_ValidPixelCount_C{c}", + "binning_min": "Histogram_BinningMin_C{c}", + "binning_max": "Histogram_BinningMax_C{c}", +} +# Columns both versions share, with the same meaning. +SHARED_COLUMNS = ( + "position_id", + "image_file", + "z", + "t", + "condition", + "biological_repeat", + "technical_repeat", + "source_relative_path", + "dtype", + "bit_depth", + "total_pixel_count", + "bin_count", +) + + +@pytest.fixture(scope="module") +def pixels(): + with np.load(PIXELS) as data: + return {key: data[key] for key in data.files} + + +def read_rows(path): + with open(path, encoding="utf-8", newline="") as f: + return list(csv.DictReader(f)) + + +def needs_zarr(dtype): + if any(rel.endswith(".ome.zarr") for rel, _, _ in EXPERIMENTS[dtype][2]): + pytest.importorskip("zarr") + + +def smallest_blocks_limit(root, mapping, bins): + """A memory limit at which every image of the experiment is read in blocks.""" + review = review_axes(preflight(root).metadata, mapping) + return max( + plane_layout(root / rel, m, memory_limit=1, bin_count=bins).block_memory_estimate + for rel, m in review.file_mappings.items() + ) + + +@pytest.mark.parametrize("blocks", [False, True], ids=["whole", "in-blocks"]) +@pytest.mark.parametrize("dtype", list(EXPERIMENTS)) +def test_histograms_equal_those_of_version_2(tmp_path, pixels, dtype, blocks): + needs_zarr(dtype) + root = tmp_path / "exp" + bins, axes = write_experiment(root, dtype, pixels) + mapping = prefill_mapping(axes) + limit = smallest_blocks_limit(root, mapping, bins) if blocks else 2**30 + result = run_experiment(root, mapping, bin_count=bins, memory_limit=limit) + assert result.written + in_blocks = [i for i in result.issues if i.code == "planes_read_in_blocks"] + assert bool(in_blocks) == blocks + if blocks: + assert len(in_blocks[0].paths) == len(EXPERIMENTS[dtype][2]) + + old = {} + for row in read_rows(DATA / f"v2_{dtype}.csv"): + assert row["schema_version"] == "2" + old[(f"{row['source_relative_path']}::z={row['z']},t={row['t']}", int(row["c"]))] = row + new = read_rows(result.csv_path) + matched = 0 + for row in new: + assert row["schema_version"] == "3" + for c in range(int(row["channel_count"])): + before = old[(row["plane_id"], c)] + for column in SHARED_COLUMNS: + assert row[column] == before[column], (row["plane_id"], column) + for v2_column, v3_column in CHANNEL_COLUMNS.items(): + assert row[v3_column.format(c=c)] == before[v2_column], (row["plane_id"], c) + counts = [row[f"Histogram_Count_C{c}_{i:05d}"] for i in range(bins)] + assert counts == [before[f"histogram_bin_{i:05d}"] for i in range(bins)] + matched += 1 + # Every version-2 row is a channel of exactly one version-3 row. + assert matched == len(old) + assert len(new) == len(old) // int(new[0]["channel_count"]) + + +def v2_records(dtype): + """The version-2 rows as one-channel records, as the code before P3b read them.""" + records = [] + for row in read_rows(DATA / f"v2_{dtype}.csv"): + number = int if np.dtype(row["dtype"]).kind in "iu" else float + bins = int(row["bin_count"]) + + def value(text, number=number): + return None if text == "" else number(text) + + histogram = PlaneHistogram( + dtype=np.dtype(row["dtype"]), + bin_count=bins, + binning_min=value(row["binning_min"]), + binning_max=value(row["binning_max"]), + counts=np.array([int(row[f"histogram_bin_{i:05d}"]) for i in range(bins)]), + total_pixel_count=int(row["total_pixel_count"]), + valid_pixel_count=int(row["valid_pixel_count"]), + ) + labels = (row["condition"], row["biological_repeat"], row["technical_repeat"]) + records.append(PlaneRecord(row["source_relative_path"], *labels, histogram)) + return records + + +@pytest.mark.parametrize("level", ["condition", "biological_repeat", "image"]) +@pytest.mark.parametrize("dtype", list(EXPERIMENTS)) +def test_step_4_statistics_are_those_of_version_2(tmp_path, pixels, dtype, level): + needs_zarr(dtype) + root = tmp_path / "exp" + bins, axes = write_experiment(root, dtype, pixels) + result = run_experiment(root, prefill_mapping(axes), bin_count=bins) + now = group_summaries(read_plane_histograms(result.csv_path), level, (1.0, 99.8)) + before = group_summaries(v2_records(dtype), level, (1.0, 99.8)) + assert now == before + assert group_table(now, level) == group_table(before, level) + + +def test_a_version_2_table_is_recomputed_on_the_next_run(tmp_path, pixels): + root = tmp_path / "exp" + bins, axes = write_experiment(root, "float32", pixels) + mapping = prefill_mapping(axes) + first = run_experiment(root, mapping, bin_count=bins) + table = first.csv_path.read_bytes() + # Leave the output as a run of version 2 did: its table, and a manifest naming it. + shutil.copyfile(DATA / "v2_float32.csv", first.csv_path) + manifest = json.loads(first.manifest_path.read_text(encoding="utf-8")) + manifest["schema_version"] = "2" + manifest["csv_sha256"] = file_sha256(first.csv_path) + first.manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + with pytest.raises(CsvFormatError, match="schema version 2"): + read_plane_histograms(first.csv_path) + + second = run_experiment(root, mapping, bin_count=bins) + assert second.written + assert second.rebuild_reason == "the earlier table uses a different format version" + assert second.read == tuple(sorted(rel for rel, _, _ in EXPERIMENTS["float32"][2])) + assert second.csv_path.read_bytes() == table + data = json.loads(second.manifest_path.read_text(encoding="utf-8")) + assert (data["manifest_version"], data["schema_version"]) == ("1", "3") + assert read_manifest(second.manifest_path).csv_sha256 == file_sha256(second.csv_path) + + +def test_a_version_2_table_under_a_current_manifest_is_recomputed(tmp_path, pixels): + # A table replaced by hand with an earlier one cannot be read, so it is recomputed. + root = tmp_path / "exp" + bins, axes = write_experiment(root, "float32", pixels) + mapping = prefill_mapping(axes) + first = run_experiment(root, mapping, bin_count=bins) + table = first.csv_path.read_bytes() + shutil.copyfile(DATA / "v2_float32.csv", first.csv_path) + manifest = json.loads(first.manifest_path.read_text(encoding="utf-8")) + manifest["csv_sha256"] = file_sha256(first.csv_path) + first.manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + second = run_experiment(root, mapping, bin_count=bins) + assert second.rebuild_reason == "the earlier histogram table could not be read" + assert second.csv_path.read_bytes() == table diff --git a/tests/test_zarr_export.py b/tests/test_zarr_export.py index 52e8930c..3dfa1e81 100644 --- a/tests/test_zarr_export.py +++ b/tests/test_zarr_export.py @@ -386,11 +386,11 @@ def test_channel_names_and_display_ranges(tmp_path): @needs_zarr def test_missing_axis_becomes_length_one(tmp_path): - write_data(tmp_path / "exp/a.tif", random_data((2, 3, 8, 8)), axes="ZCYX") + write_data(tmp_path / "exp/a.tif", random_data((2, 3, 8, 8)), axes="ZTYX") data = random_data((2, 8, 8), seed=1) write_data(tmp_path / "exp/b.tif", data, axes="ZYX") result = run_experiment( - tmp_path / "exp", prefill_mapping("ZCYX"), convert_tiffs=True, missing_as_singleton=True + tmp_path / "exp", prefill_mapping("ZTYX"), convert_tiffs=True, missing_as_singleton=True ) group = open_copy(result.output_folder, "b.tif") assert group["0"].shape == (1, 2, 8, 8) @@ -482,13 +482,18 @@ def test_switching_to_conversion_recomputes_the_tiff_rows_from_copies(experiment @needs_zarr def test_histograms_are_computed_from_the_converted_copies(experiment, monkeypatch): read_paths = [] - original = run_module.iter_planes - def spy(path, *args, **kwargs): - read_paths.append(Path(path)) - return original(path, *args, **kwargs) + def spy(name): + original = getattr(run_module, name) - monkeypatch.setattr(run_module, "iter_planes", spy) + def reading(path, *args, **kwargs): + read_paths.append(Path(path)) + return original(path, *args, **kwargs) + + monkeypatch.setattr(run_module, name, reading) + + spy("iter_planes") # conversion + spy("iter_xyc_planes") # histograms run_experiment(experiment, ZYX, convert_tiffs=True) copies = [p for p in read_paths if p.name.endswith(".ome.zarr")] sources = [p for p in read_paths if p.suffix == ".tif"] @@ -552,7 +557,8 @@ def test_table_is_the_same_with_or_without_copies(experiment, tmp_path): b = read_plane_histograms(without.csv_path) assert [r.plane_id for r in a] == [r.plane_id for r in b] assert all( - np.array_equal(x.histogram.counts, y.histogram.counts) for x, y in zip(a, b, strict=True) + np.array_equal(x.channel_histogram(0).counts, y.channel_histogram(0).counts) + for x, y in zip(a, b, strict=True) ) diff --git a/tests/test_zarr_input.py b/tests/test_zarr_input.py index 28e6a839..6f1048c3 100644 --- a/tests/test_zarr_input.py +++ b/tests/test_zarr_input.py @@ -450,7 +450,10 @@ def test_review_of_a_mixed_experiment(): found = issue(review.issues, "ome_zarr_axes_mismatch") assert found.severity is Severity.ERROR and found.paths == ("d.ome.zarr",) confirmed = review_axes(files[:3], EXPERIMENT, missing_as_singleton=True) - assert confirmed.ok and confirmed.file_mappings["c.ome.zarr"].singleton_roles == ("C",) + assert confirmed.file_mappings["c.ome.zarr"].singleton_roles == ("C",) + # Read as one channel, c.ome.zarr differs from the two channels of the others (N43). + assert issue(confirmed.issues, "different_channel_counts").paths == ("c.ome.zarr",) + assert len(confirmed.issues) == 1 # Reading planes ------------------------------------------------------------------------ @@ -634,16 +637,16 @@ def test_ome_zarr_experiment_end_to_end(zarr_experiment): result = run_experiment(zarr_experiment, pre.suggested_mapping) assert result.written and result.converted == () records = read_plane_histograms(result.csv_path) - assert len(records) == 18 + assert len(records) == 9 # 3 stores x 3 depth slices, each row with 2 channels import zarr for record in records: data = zarr.open_group(str(zarr_experiment / record.source_relative_path), mode="r")["0"] - plane = np.asarray(data[record.c, record.z]) - expected = compute_plane_histogram(plane) - np.testing.assert_array_equal(record.histogram.counts, expected.counts) - assert record.t == 0 - assert records[0].plane_id == "ctrl/rep1/a.ome.zarr::z=0,c=0,t=0" + assert record.channel_count == 2 and record.t == 0 + for c in range(2): + expected = compute_plane_histogram(np.asarray(data[c, record.z])) + np.testing.assert_array_equal(record.channel_histogram(c).counts, expected.counts) + assert records[0].plane_id == "ctrl/rep1/a.ome.zarr::z=0,t=0" assert {(r.condition, r.position_id) for r in records} == { ("ctrl", "a"), ("ctrl", "b"), @@ -682,24 +685,26 @@ def test_mixed_tiff_and_ome_zarr_experiment(tmp_path): pre = preflight(root) assert pre.suggested_mapping == prefill_mapping("ZCYX") blocked = run_experiment(root, pre.suggested_mapping) - assert codes(blocked.issues)[:2] == ["missing_axes", "different_plane_counts"] - result = run_experiment(root, pre.suggested_mapping, missing_as_singleton=True) + assert codes(blocked.issues)[0] == "missing_axes" + # Its missing channel axis is read as one channel, but the others have two (N43). + one_channel = run_experiment(root, pre.suggested_mapping, missing_as_singleton=True) + assert issue(one_channel.issues, "different_channel_counts").paths == ("drug/c.ome.zarr",) + write_ngff(root / "drug/d.ome.zarr", random_data((3, 2, 8, 8), "uint8", seed=3), "zcyx") + result = run_experiment( + root, pre.suggested_mapping, missing_as_singleton=True, exclude=["drug/c.ome.zarr"] + ) assert result.written rows = {r.plane_id: r for r in read_plane_histograms(result.csv_path)} # The same coordinates mean the same plane, whatever the storage order. - b = rows["ctrl/b.ome.zarr::z=2,c=1,t=0"] + b = rows["ctrl/b.ome.zarr::z=2,t=0"] np.testing.assert_array_equal( - b.histogram.counts, compute_plane_histogram(zarr_data[1, 2]).counts + b.channel_histogram(1).counts, compute_plane_histogram(zarr_data[1, 2]).counts ) - a = rows["ctrl/a.tif::z=2,c=1,t=0"] + a = rows["ctrl/a.tif::z=2,t=0"] np.testing.assert_array_equal( - a.histogram.counts, compute_plane_histogram(tiff_data[2, 1]).counts + a.channel_histogram(1).counts, compute_plane_histogram(tiff_data[2, 1]).counts ) - assert {(r.z, r.c) for r in rows.values() if r.source_relative_path == "drug/c.ome.zarr"} == { - (0, 0), - (1, 0), - (2, 0), - } + assert {r.z for r in rows.values() if r.source_relative_path == "drug/d.ome.zarr"} == {0, 1, 2} def test_conversion_leaves_ome_zarr_images_alone(tmp_path): @@ -733,7 +738,7 @@ def test_ome_ngff_0_5_experiment(tmp_path): root = tmp_path / "exp" write_ngff(root / "a.ome.zarr", random_data((2, 3, 8, 8), "uint8"), "czyx", version="0.5") result = run_experiment(root, prefill_mapping("CZYX")) - assert result.written and result.plane_count == 6 + assert result.written and result.plane_count == 3 def test_cli_check_names_the_formats(tmp_path, capsys): diff --git a/tests/v2_fixture.py b/tests/v2_fixture.py new file mode 100644 index 00000000..67112c5e --- /dev/null +++ b/tests/v2_fixture.py @@ -0,0 +1,82 @@ +"""The experiments of the schema-version-2 fixture in ``tests/data`` (follow-up P3b). + +``tests/data/make_v2_fixture.py`` wrote their ``plane_histograms.csv`` with the +code of version 2 (one row per plane and channel); ``test_schema_v2_equivalence.py`` +writes the same images again and compares the current table with it. +""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np + +from tiff_tree import write_data + +DATA = Path(__file__).parent / "data" +PIXELS = DATA / "v2_pixels.npz" + +# Per pixel type: the bin count, the experiment axis roles and its images as +# (relative path, axes, writer options). TIFF images share one arrangement of axes; +# the OME-Zarr image stores its axes in another order, matched by role. +EXPERIMENTS = { + "uint16": ( + 500, # fewer bins than values in most channels: inexact binning too + "ZCYX", + [ + ("ctrl/rep1/a.tif", "ZCYX", {}), + ("ctrl/rep1/b.tif", "ZCYX", {"compression": "zlib", "tile": (16, 16)}), + ("drug/rep1/a.tif", "ZCYX", {"imagej": True}), + ("drug/rep1/c.ome.zarr", "czyx", {}), + ], + ), + "float32": ( + 64, + "TCYX", + [ + ("ctrl/rep1/a.tif", "TCYX", {}), + ("drug/rep1/a.tif", "TCYX", {"compression": "zlib", "rowsperstrip": 5}), + ], + ), +} + + +def make_pixels(seed: int = 0) -> dict[str, np.ndarray]: + """Simulate the pixels of every image, keyed ``"/"``.""" + rng = np.random.default_rng(seed) + pixels = {} + for rel, axes, _ in EXPERIMENTS["uint16"][2]: + # Channels with narrow (exact) and wide (binned) ranges of values. + shape = {"Z": 2, "C": 3, "Y": 20, "X": 18} + data = np.stack( + [ + rng.integers(100, 100 + spread, (shape["Z"], shape["Y"], shape["X"])) + for spread in (300, 5_000, 60_000) + ], + axis=1, + ).astype(np.uint16) + if axes == "czyx": + data = np.ascontiguousarray(data.transpose(1, 0, 2, 3)) + pixels[f"uint16/{rel}"] = data + for i, (rel, _, _) in enumerate(EXPERIMENTS["float32"][2]): + data = rng.normal(50, 20, (2, 2, 15, 13)).astype(np.float32) # T, C, Y, X + data[0, 0, 1, :3] = [np.nan, np.inf, -np.inf] + if i == 1: + data[1, 1] = np.nan # a channel without finite pixels at one time point + data[0, 1] = 7.5 # a constant channel + pixels[f"float32/{rel}"] = data + return pixels + + +def write_experiment(root: Path, dtype: str, pixels: dict[str, np.ndarray]) -> tuple[int, str]: + """Write the images of one experiment under ``root``; return its bin count and axes.""" + bins, axes, images = EXPERIMENTS[dtype] + for rel, image_axes, kwargs in images: + data = pixels[f"{dtype}/{rel}"] + if rel.endswith(".ome.zarr"): + from zarr_tree import write_ngff + + write_ngff(root / rel, data, image_axes, chunks=(1, 1, 8, 8)) + else: + write_data(root / rel, data, axes=image_axes, **kwargs) + return bins, axes