Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,9 @@ src/morphobricks_preprocessing/
table, unbalanced_images, repeat_missing_conditions, CONSISTENCY_CODES (P2)
design.py experiment design (P3): ExperimentDesign/DesignRow, default pairing, reconcile with
the folders on reruns, design_issues, read/write experiment_design.csv, load_design
image_type.py image type (P4): FLUORESCENCE/RGB, review_image_type (suggestion from
TiffMetadata.rgb_samples and the confirmed roles, whether RGB is possible via the
brick's check_rgb), image_type_issues (rgb_not_possible)
validation.py ValidationIssue (severity, code, message, paths, action), error()/warning() builders
axes.py AxisMapping (role per dimension), prefill, check_mapping, review_axes (one mapping
applied to every file, missing axes as singletons only when confirmed)
Expand Down Expand Up @@ -368,6 +371,21 @@ functions the tests use.
reconciled template and exits 1. **API** `run_experiment(design=None)` is optional; with
a design it is written with the table and `experiment_design_sha256` goes in the manifest.

### Image type (contract §13, follow-up P4)

- **Types:** `fluorescence` and `rgb`; RGB needs three channels of an integer type
(the normalization brick's `check_rgb`), else `rgb_not_possible` in a run.
- **Suggestion:** `rgb` only when every analysed image is a TIFF stored as RGB colour
(`TiffMetadata.rgb_samples`: photometric RGB, three samples per pixel, axis `S`), the
confirmed C role is on `S`, and RGB is possible. Never applied without confirmation.
- **Notebook:** step 2's "Image type" card appears once the axis roles are confirmed
(also when embedded); radio buttons with the suggestion preselected, disabled at
fluorescence when RGB is impossible; `ConfirmToggle` **Yes, this is the right image
type**; changing roles or folder resets it; step 3 gated on `confirmed_image_type`.
- **CLI:** `--image-type`, else the suggestion, printed with its reason.
- **Manifest:** optional top-level `image_type` (null without one) and
`channels_from_rgb_samples`; changing the type recomputes nothing.

### Validation messages (contract §8)

Every expected problem is a `ValidationIssue` with:
Expand Down Expand Up @@ -407,6 +425,7 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
| Normalization brick (P1) | The modules `dtypes`, `histogram`, `records`, `csv_io`, `pooling` and `normalization` moved unchanged to `morphobricks_normalization` (normalization plan N1, N2, N6) and are imported from there. Their public names stay importable from `morphobricks_preprocessing` and from the old submodules for one version: module `__getattr__` stand-ins (`_moved.py`), one `DeprecationWarning` per old path naming `morphobricks_normalization.<module>.<name>`, same objects. The package's own code never uses the old paths. Outputs unchanged (CSV schema 2); the manifest's `versions` adds `morphobricks-normalization`; Step 0 checks for it; contract §3–§7 text moved to the normalization contract (§1.2, now §6.2 for the CSV) |
| Dataset consistency (Q21, P2) | Step 1 "Dataset consistency" panel: balance table (images and technical-repeat folders per condition x biological repeat), `unbalanced_images` (< 0.5 or > 2 x median), `repeat_missing_conditions`, `different_pixel_sizes`, `missing_pixel_size` (OME-Zarr scale included), `different_plane_counts`; `different_sizes` narrowed to width/height; consistency issues shown only in that panel, not in Messages. Warnings only; outputs unchanged |
| Experiment design (Q22, P3) | Step 1 after the labels: pairing dropdowns, one yellow "Yes, they are comparable" question per repeat group (single whole-experiment question without biological-repeat folders), control checkboxes; `experiment_design.csv` written after every answer; reruns load it and fold the questions behind a Change button; step 3 gated. CLI `run` reads it or `--design`, else writes a template and exits 1. `run_experiment(design=...)` optional; manifest `experiment_design_sha256` (optional key, version 1). Two folders of one condition in a group block that group |
| Image type (Q29, P4) | Step 2 card after the axis roles: Fluorescence or RGB, suggestion preselected, confirmed with **Yes, this is the right image type**; RGB suggested for TIFF photometric RGB with three samples on the C axis; RGB needs three integer channels (`check_rgb`), else disabled with the reason and `rgb_not_possible` in a run. Manifest keys `image_type`, `channels_from_rgb_samples` (optional, version 1); type changes recompute nothing. CLI `--image-type`, default the printed suggestion |
| Position identity (Q20, D02) | `position_id` = image name without extension, in every CSV row and manifest file entry; FOV key = labels + `position_id`; duplicates in one leaf folder are the error `duplicate_position_id`. CSV columns and schema 2 unchanged; manifest version 1 with an optional key |
| Notebook style | Follow the lab's NucleiSky and LabChronicle notebooks (see §7) |
| Widget widths | **99%**, never 100% |
Expand Down Expand Up @@ -695,8 +714,9 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb
- **Released as v0.1.0** (27 October 2026), with everything above; the normalization brick
is pinned to its v0.2.0 (question 28). The next tag, v0.2.0, follows follow-ups P4 and
P5.
- **Next (not started):** follow-ups P4–P5 of the normalization plan (image type,
intensity checks and plots).
- **Since v0.1.0:** follow-up P4 of the normalization plan, the image type
(question 29).
- **Next:** follow-up P5 of the normalization plan (intensity checks and plots).
- **Not yet verified:**
- the exact look of the notebook in a real JupyterLab;
- the optional `jl-hidecode` JupyterLab extension, which keeps code cells collapsed. It
Expand Down
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ Pre-alpha. Implemented and tested:
blocks with identical results (`planes`)
- The experiment design: pairing biological repeats across conditions, confirming they
are comparable, and marking controls, in `experiment_design.csv` (`design`)
- The image type, fluorescence or RGB: suggested from the metadata (RGB colour samples),
confirmed by the user and recorded in the run manifest (`image_type`)
- Running an experiment end to end and updating earlier output incrementally, with a
run manifest (`run`, `manifest`)
- A step-by-step notebook ([notebooks/MorphoBricks_Preprocessing.ipynb](notebooks/MorphoBricks_Preprocessing.ipynb))
Expand Down Expand Up @@ -70,7 +72,7 @@ From a terminal:

```bash
morphobricks-preprocessing check /data/exp1
morphobricks-preprocessing run /data/exp1 --axes ZCYX
morphobricks-preprocessing run /data/exp1 --axes ZCYX --image-type fluorescence
```

## Running an experiment
Expand All @@ -92,7 +94,9 @@ recomputes everything, and `exclude=[...]` leaves out damaged files. The noteboo
command line also ask for the experiment design (which biological repeats pair across
conditions, and that they are comparable) and write `experiment_design.csv`; in Python,
pass it with `design=` (see the [user guide](docs/user_guide.md) §9). Later bricks read it
with `read_experiment_design`.
with `read_experiment_design`. They also ask for the image type, `fluorescence` or `rgb`
(suggested by `review_image_type`), and record it in the manifest; in Python, pass it with
`image_type=`.

## Example

Expand Down
20 changes: 19 additions & 1 deletion docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ method is an offset and a scale, `(pixels − offset) / scale`.

| Severity | Situations | Effect |
|---|---|---|
| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; output folder inside the experiment; no usable planes | Processing is blocked |
| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; output folder inside the experiment; no usable planes | Processing is blocked |
| Warning | Files that are not images; empty folders; images outside leaf folders; images excluded by the user, or excluded paths that match no image; channel names differing between files; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; planes over the memory limit that are read in blocks | Processing continues; the item is reported |

Every message states what happened in plain language, lists the affected paths, and
Expand Down Expand Up @@ -150,6 +150,8 @@ file's schema or statistical interpretation.
| `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count` (its rows in the table: XYC planes), `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `ome_zarr_path` (for a TIFF converted to OME-Zarr, the copy relative to the output folder; null otherwise or when the key is absent), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent), `position_id` (the image's position identifier, §1; null when absent, in manifests written before it). For an OME-Zarr image, `size` is the total size of its files and `mtime_ns` the time of its newest file. |
| `csv_sha256` | SHA-256 of the table written with this manifest. |
| `experiment_design_sha256` | SHA-256 of the `experiment_design.csv` written with this run (§12), so later bricks can check that the design they read is the one used; null when the run had no design, or absent in manifests written before it. Optional key: the manifest stays version 1. |
| `image_type` | `fluorescence` or `rgb`, the image type the user confirmed (§13); null when the run was given none, or absent in manifests written before it. Optional key: the manifest stays version 1. Changing it recomputes nothing. |
| `channels_from_rgb_samples` | `true` when the C role is the RGB samples axis of images stored as RGB colour (§13), `false` otherwise; absent in manifests written before it. Optional key. |
| `versions` | Versions of `morphobricks-preprocessing`, `morphobricks-normalization` (which computes the histograms and writes the CSV; absent in manifests written before it), `numpy`, `tifffile` and Python. Information only: a different version never causes a rebuild. |

Channel names and significant bits come from OME metadata and are for information only.
Expand Down Expand Up @@ -239,3 +241,19 @@ folder.
| Notebook | Step 1 asks once the labels are confirmed and writes the file after every answer (unconfirmed groups with `confirmed=false`). A design confirmed for the same folders is loaded with the questions folded away behind **Change the experiment design**. Step 3 runs only once every group is confirmed. |
| Command line | `run` reads the file from the output folder, or `--design FILE`. Without a fully confirmed design it writes the design brought up to date with the folders (a template with `confirmed=false` for the groups still to confirm, keeping the answers already given) to the output folder, explains the columns, and stops with status 1. Confirmed rows without a date get the date of the run. |
| Run | `run_experiment(..., design=...)` brings the design up to date with the folders and stops with `design_not_confirmed` or `repeat_group_conflict` if it is not fully confirmed. Otherwise the file is written with the table and published before the manifest, which records its SHA-256 (§9). Without a design the run is unchanged: no file is written, an existing one is left as it is, and `experiment_design_sha256` is null. `plane_histograms.csv` does not depend on the design. |

## 13. Image type

Later bricks normalize `fluorescence` images channel by channel and `rgb` images with
StarDist's percentile method over the three colour channels together ([normalization
contract][normalization contract] §3.3 and §5). The type is the user's choice (normalization
plan N34); this brick suggests it, asks for it after the axis roles, and records it.

| Rule | Detail |
|---|---|
| Types | `fluorescence` (any number of channels, every pixel type) and `rgb` (exactly three channels of an integer pixel type, the rule of the normalization brick's `check_rgb`). |
| RGB samples | A TIFF image is stored as RGB colour when its first series has the photometric interpretation RGB with three samples per pixel (`tifffile` axis `S`; `TiffMetadata.rgb_samples`). RGB with an alpha sample, several grey samples and OME-Zarr images are not. |
| Suggestion | `rgb` when every analysed image is stored as RGB colour, the confirmed C role is on its samples axis, and `rgb` is possible; `fluorescence` otherwise. `review_image_type` gives the suggestion, why, and why `rgb` is not possible (from the headers and the confirmed axis roles; no pixels are read). |
| Confirmation | Never applied without the user's confirmation. The notebook asks in step 2 once the axis roles are confirmed, with the suggestion preselected and a **Yes, this is the right image type** button; `rgb` cannot be chosen, with the reason shown, when it is not possible. Changing the axis roles or the folder asks again. Step 3 runs only once the type is confirmed. The command line takes `--image-type`, or uses the suggestion and prints it. |
| Run | `run_experiment(..., image_type=...)` records the type and `channels_from_rgb_samples` in the manifest (§9). `rgb` for images that cannot be RGB is the error `rgb_not_possible`; any other value raises `ValueError`. Without a type the manifest records null. |
| Outputs | `plane_histograms.csv` and `experiment_design.csv` do not depend on the type: a plane of an RGB image is one row with channels C0, C1 and C2, as for any image. Changing the type on a later run reads no image again. |
28 changes: 28 additions & 0 deletions docs/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,12 @@ API, CLI). The notebook and CLI only call the API.
columns; its channels read one at a time (`planes.iter_xyc_planes`); images with
different numbers of channels are an error before any image is read; earlier tables are
recomputed on the next run. *Implemented, see question 27.*
- **Image type (follow-up P4 of the normalization plan):** after the axis roles, step 2
asks whether the images are fluorescence or RGB, suggesting RGB for TIFF images stored as
RGB colour on the channel axis (`image_type`, `metadata.TiffMetadata.rgb_samples`); RGB
needs three channels of an integer type. The manifest records the type; the command line
takes `--image-type` or prints the suggestion. Outputs are otherwise unchanged.
*Implemented, see question 29.*

## 11. Tests and acceptance criteria

Expand Down Expand Up @@ -691,3 +697,25 @@ Raised during review; to be decided before the milestones that need them:
depends on the normalization brick's release v0.2.0 (tag, not commit) instead of the
Milestone 2b merge commit `0e53e67`; v0.2.0 adds only RGB and intensity checks, which
this brick does not call yet. The next tag, v0.2.0, follows follow-ups P4 and P5.*
29. How is the image type, fluorescence or RGB, chosen and recorded (normalization plan
N33, N34, its contract §5, follow-up P4)?
*Decided (27 October 2026): after the axis roles are confirmed, step 2 shows an
"Image type" card with the two types, the suggestion preselected and the yellow question
"Is this the right image type for your images?" answered with **Yes, this is the right
image type** (Undo to change it); nothing is used until it is confirmed, and step 3 stays
locked until then. RGB is suggested when every analysed image is a TIFF stored as RGB
colour (photometric RGB, three samples per pixel, read from the header as
`TiffMetadata.rgb_samples`), the confirmed C role is on its samples axis, and RGB is
possible; fluorescence otherwise (OME-Zarr images, grey samples, RGB with alpha). RGB
needs exactly three channels of an integer pixel type, the rule of the normalization
brick's `check_rgb`; otherwise the choice stays at fluorescence with the reason shown
(ipywidgets cannot disable one radio button, so the whole choice is disabled), and
`run_experiment(image_type="rgb")` is the error `rgb_not_possible`. Changing the axis
roles or the folder asks again; the card is the same when step 2 is embedded by another
brick (question 26). Agreed with the owner: the manifest gains two optional top-level
keys, `image_type` (null when the run was given none, as `run_experiment` takes it
optionally like the design of question 22) and `channels_from_rgb_samples`; it stays
version 1. Changing only the type recomputes nothing. The command line takes
`--image-type fluorescence|rgb`, or uses the suggestion and prints it with the reason;
`check` prints the suggestion. `plane_histograms.csv` and `experiment_design.csv` do not
change.*
Loading
Loading