Intensity checks and plots from the histogram table (P5)

AGENTS.md

@@ -76,6 +76,9 @@ src/morphobricks_preprocessing/
   image_type.py     image type (P4): FLUORESCENCE/RGB, review_image_type (suggestion from
                     TiffMetadata.rgb_samples and the confirmed roles, whether RGB is possible via the
                     brick's check_rgb), image_type_issues (rgb_not_possible)
+  intensity.py      intensity checks (P5): check_intensities/check_table (the brick's image_summaries
+                    and intensity_checks, IntensityReport), repeat_boxes and repeat_distributions for
+                    the notebook's plots, intensity_text for the CLI
   validation.py     ValidationIssue (severity, code, message, paths, action), error()/warning() builders
   axes.py           AxisMapping (role per dimension), prefill, check_mapping, review_axes (one mapping
                     applied to every file, missing axes as singletons only when confirmed)
@@ -386,6 +389,18 @@ functions the tests use.
 - **Manifest:** optional top-level `image_type` (null without one) and
   `channels_from_rgb_samples`; changing the type recomputes nothing.
 
+### Intensity checks (contract §14, follow-up P5)
+
+- **From the table only:** `run_experiment` checks its rows in memory (`RunResult.intensity`,
+  apart from `RunResult.issues`, never blocking); `check_table` reads any table. The four
+  codes are the normalization brick's; comparisons stay within a condition.
+- **Limits:** `IntensityCheckSettings` (3.0, 0.01, 5), recorded in the manifest
+  (`intensity_checks`, optional); changing them recomputes nothing.
+- **Notebook:** `IntensityPanel` in step 3 (after a successful run, `run_checks`) and step 4
+  (`table_checks`): status, tiles, warnings table with actions, collapsed **Check
+  settings**, channel buttons, box plots (`Issue.channel` marks images) and cumulative
+  distributions per biological repeat. Nothing written. CLI prints them, no plots.
+
 ### Validation messages (contract §8)
 
 Every expected problem is a `ValidationIssue` with:
@@ -426,6 +441,7 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
 | Dataset consistency (Q21, P2) | Step 1 "Dataset consistency" panel: balance table (images and technical-repeat folders per condition x biological repeat), `unbalanced_images` (< 0.5 or > 2 x median), `repeat_missing_conditions`, `different_pixel_sizes`, `missing_pixel_size` (OME-Zarr scale included), `different_plane_counts`; `different_sizes` narrowed to width/height; consistency issues shown only in that panel, not in Messages. Warnings only; outputs unchanged |
 | Experiment design (Q22, P3) | Step 1 after the labels: pairing dropdowns, one yellow "Yes, they are comparable" question per repeat group (single whole-experiment question without biological-repeat folders), control checkboxes; `experiment_design.csv` written after every answer; reruns load it and fold the questions behind a Change button; step 3 gated. CLI `run` reads it or `--design`, else writes a template and exits 1. `run_experiment(design=...)` optional; manifest `experiment_design_sha256` (optional key, version 1). Two folders of one condition in a group block that group |
 | Image type (Q29, P4) | Step 2 card after the axis roles: Fluorescence or RGB, suggestion preselected, confirmed with **Yes, this is the right image type**; RGB suggested for TIFF photometric RGB with three samples on the C axis; RGB needs three integer channels (`check_rgb`), else disabled with the reason and `rgb_not_possible` in a run. Manifest keys `image_type`, `channels_from_rgb_samples` (optional, version 1); type changes recompute nothing. CLI `--image-type`, default the printed suggestion |
+| Intensity checks (Q30, P5) | Warnings from the table alone with the brick's `image_summaries`/`intensity_checks`, shown automatically after step 3 and in step 4 for any table; limits under a collapsed **Check settings** panel, recorded in the manifest (`intensity_checks`); per-channel box plots with flagged images and cumulative distributions per biological repeat, notebook only; the CLI prints the warnings (`--outlier-limit`, `--saturation-limit`, `--min-images`) |
 | Position identity (Q20, D02) | `position_id` = image name without extension, in every CSV row and manifest file entry; FOV key = labels + `position_id`; duplicates in one leaf folder are the error `duplicate_position_id`. CSV columns and schema 2 unchanged; manifest version 1 with an optional key |
 | Notebook style | Follow the lab's NucleiSky and LabChronicle notebooks (see §7) |
 | Widget widths | **99%**, never 100% |
@@ -714,9 +730,8 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb
 - **Released as v0.1.0** (27 October 2026), with everything above; the normalization brick
   is pinned to its v0.2.0 (question 28). The next tag, v0.2.0, follows follow-ups P4 and
   P5.
-- **Since v0.1.0:** follow-up P4 of the normalization plan, the image type
-  (question 29).
-- **Next:** follow-up P5 of the normalization plan (intensity checks and plots).
+- **Since v0.1.0:** follow-ups P4 (the image type, question 29) and P5 (intensity checks
+  and plots, question 30) of the normalization plan. The owner tags v0.2.0 next.
 - **Not yet verified:**
   - the exact look of the notebook in a real JupyterLab;
   - the optional `jl-hidecode` JupyterLab extension, which keeps code cells collapsed. It

README.md

@@ -36,6 +36,9 @@ Pre-alpha. Implemented and tested:
   confirmed by the user and recorded in the run manifest (`image_type`)
 - Running an experiment end to end and updating earlier output incrementally, with a
   run manifest (`run`, `manifest`)
+- Intensity checks of the written table: images far from the others of their biological
+  repeat, shifted biological repeats, saturated images and images without spread, with
+  plots per channel in the notebook (`intensity`)
 - A step-by-step notebook ([notebooks/MorphoBricks_Preprocessing.ipynb](notebooks/MorphoBricks_Preprocessing.ipynb))
   and a command line (`morphobricks-preprocessing check` / `run`)
 
@@ -96,7 +99,8 @@ conditions, and that they are comparable) and write `experiment_design.csv`; in
 pass it with `design=` (see the [user guide](docs/user_guide.md) §9). Later bricks read it
 with `read_experiment_design`. They also ask for the image type, `fluorescence` or `rgb`
 (suggested by `review_image_type`), and record it in the manifest; in Python, pass it with
-`image_type=`.
+`image_type=`. After writing, the run checks the images' intensities from the table
+(`result.intensity`); `check_table(path)` checks any table.
 
 ## Example
 

docs/contract.md

@@ -108,7 +108,7 @@ method is an offset and a scale, `(pixels − offset) / scale`.
 | Severity | Situations | Effect |
 |---|---|---|
 | Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; output folder inside the experiment; no usable planes | Processing is blocked |
-| Warning | Files that are not images; empty folders; images outside leaf folders; images excluded by the user, or excluded paths that match no image; channel names differing between files; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; planes over the memory limit that are read in blocks | Processing continues; the item is reported |
+| Warning | Intensity checks of the written table (`image_intensity_outlier`, `repeat_intensity_shift`, `saturated_image`, `no_intensity_spread`, §14), reported apart from the run's issues; files that are not images; empty folders; images outside leaf folders; images excluded by the user, or excluded paths that match no image; channel names differing between files; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; planes over the memory limit that are read in blocks | Processing continues; the item is reported |
 
 Every message states what happened in plain language, lists the affected paths, and
 suggests an action. Technical details go to the diagnostic log.
@@ -152,6 +152,7 @@ file's schema or statistical interpretation.
 | `experiment_design_sha256` | SHA-256 of the `experiment_design.csv` written with this run (§12), so later bricks can check that the design they read is the one used; null when the run had no design, or absent in manifests written before it. Optional key: the manifest stays version 1. |
 | `image_type` | `fluorescence` or `rgb`, the image type the user confirmed (§13); null when the run was given none, or absent in manifests written before it. Optional key: the manifest stays version 1. Changing it recomputes nothing. |
 | `channels_from_rgb_samples` | `true` when the C role is the RGB samples axis of images stored as RGB colour (§13), `false` otherwise; absent in manifests written before it. Optional key. |
+| `intensity_checks` | The limits of the intensity checks run on the table (§14): `IntensityCheckSettings.to_dict()` of the normalization brick (`library_version`, `outlier_limit`, `saturation_limit`, `min_images`); absent in manifests written before it. Optional key. Changing the limits recomputes nothing. |
 | `versions` | Versions of `morphobricks-preprocessing`, `morphobricks-normalization` (which computes the histograms and writes the CSV; absent in manifests written before it), `numpy`, `tifffile` and Python. Information only: a different version never causes a rebuild. |
 
 Channel names and significant bits come from OME metadata and are for information only.
@@ -257,3 +258,19 @@ plan N34); this brick suggests it, asks for it after the axis roles, and records
 | Confirmation | Never applied without the user's confirmation. The notebook asks in step 2 once the axis roles are confirmed, with the suggestion preselected and a **Yes, this is the right image type** button; `rgb` cannot be chosen, with the reason shown, when it is not possible. Changing the axis roles or the folder asks again. Step 3 runs only once the type is confirmed. The command line takes `--image-type`, or uses the suggestion and prints it. |
 | Run | `run_experiment(..., image_type=...)` records the type and `channels_from_rgb_samples` in the manifest (§9). `rgb` for images that cannot be RGB is the error `rgb_not_possible`; any other value raises `ValueError`. Without a type the manifest records null. |
 | Outputs | `plane_histograms.csv` and `experiment_design.csv` do not depend on the type: a plane of an RGB image is one row with channels C0, C1 and C2, as for any image. Changing the type on a later run reads no image again. |
+
+## 14. Intensity checks
+
+After the table is written, the images' intensities are checked from its rows alone, with
+the normalization brick's `image_summaries` and `intensity_checks` ([normalization
+contract][normalization contract] §7, plan N38): no image is read and nothing is written
+apart from the limits in the manifest.
+
+| Rule | Detail |
+|---|---|
+| Summaries | One per image, all Z and T planes pooled per channel: median, 1st and 99.8th percentiles, saturated fraction (normalization contract §7.1). |
+| Warnings | `image_intensity_outlier` (an image's median or 99.8th percentile in a channel more than `outlier_limit` robust deviations from the other images of its condition and biological repeat), `repeat_intensity_shift` (a biological repeat's median of image medians far from the other repeats' images of its condition), `saturated_image` (more than `saturation_limit` of the pixels at the type's maximum), `no_intensity_spread` (1st and 99.8th percentiles equal). One warning per image or repeat, channel and compared value, naming the channel `C<i>`, the image or `<condition>/<biological repeat>`, the value and the group value or limit. Comparisons stay within a condition, so treatment effects are never reported. |
+| Limits | `IntensityCheckSettings`: `outlier_limit` 3.0, `saturation_limit` 0.01, `min_images` 5 (provisional in the normalization brick). `run_experiment(..., intensity_settings=...)` uses them and records them in the manifest (`intensity_checks`, §9); they change no row. |
+| Run | `RunResult.intensity` holds the summaries and warnings (`IntensityReport`) of a written table, None otherwise. The warnings never block and are kept apart from `RunResult.issues`. |
+| Notebook | Step 3 shows the checks of the table just written after every successful run, step 4 those of any chosen table: a status line with the limits, counts per kind, a table of the warnings (check, channel, image or repeat, what was found) and what to do about each kind, a collapsed **Check settings** panel (changing a limit updates the warnings at once; in step 3 the next run records it), and per channel (chosen with buttons) box plots of the image medians and high percentiles per condition and biological repeat, with images that have a warning in that channel (`Issue.channel`, N52) as red diamonds and shifted repeats as amber boxes, and the cumulative intensity distribution of each biological repeat, one panel per condition (shifted repeats dashed; at most 12 conditions). The plots are drawn in the notebook only; no file is written. |
+| Command line | `run` prints the warnings after the run, grouped by kind with what to do once per kind, and draws no plots; `--outlier-limit`, `--saturation-limit` (a fraction) and `--min-images` set the limits. |

docs/plan.md

@@ -346,6 +346,12 @@ API, CLI). The notebook and CLI only call the API.
   needs three channels of an integer type. The manifest records the type; the command line
   takes `--image-type` or prints the suggestion. Outputs are otherwise unchanged.
   *Implemented, see question 29.*
+- **Intensity checks (follow-up P5 of the normalization plan):** after every successful
+  run, and in step 4 for any table, the normalization brick's `image_summaries` and
+  `intensity_checks` give the four intensity warnings from the table alone (`intensity`);
+  the notebook shows them with adjustable limits and per-channel box plots and
+  distributions, the command line prints them, and the manifest records the limits.
+  *Implemented, see question 30.*
 
 ## 11. Tests and acceptance criteria
 
@@ -719,3 +725,26 @@ Raised during review; to be decided before the milestones that need them:
     `--image-type fluorescence|rgb`, or uses the suggestion and prints it with the reason;
     `check` prints the suggestion. `plane_histograms.csv` and `experiment_design.csv` do not
     change.*
+30. How are the intensity checks shown (normalization plan N38, N50–N53, its contract §7,
+    follow-up P5)?
+    *Decided (27 October 2026): `run_experiment` checks the rows it wrote, in memory, with
+    the normalization brick's `image_summaries` and `intensity_checks`
+    (`intensity.check_intensities`, `RunResult.intensity`), and `check_table` checks any
+    table; no image is read. The four warnings are kept apart from the run's issues and
+    never block. Agreed with the owner: the warnings and the plots are shown both in step 3,
+    automatically after every successful run, and in step 4 for the chosen table; the
+    manifest records the limits used (optional key `intensity_checks`,
+    `IntensityCheckSettings.to_dict()`; version 1), changing them afterwards only updates
+    the warnings on screen and is recorded by the next run; the distributions are
+    cumulative curves, like step 4's existing plot. Each panel has a status line, tiles per
+    kind, a table (check, channel `C<i>`, image or repeat, what was found) with what to do
+    per kind, a collapsed **Check settings** panel (`outlier_limit` 3.0, `saturation_limit`
+    1 %, `min_images` 5, and **Restore the defaults**), channel buttons, box plots of the
+    image medians and high percentiles per condition and biological repeat (images with a
+    warning in that channel, from `Issue.channel`, as red diamonds; shifted repeats as amber
+    boxes), and the cumulative distribution of each biological repeat, one panel per
+    condition (shifted repeats dashed; at most 12 conditions). Plots are drawn in the
+    notebook only and nothing is written. The command line prints the warnings grouped by
+    kind with the action once per kind, takes `--outlier-limit`, `--saturation-limit` and
+    `--min-images`, and draws no plots. Comparisons stay within a condition, as the brick
+    does. `plane_histograms.csv` and `experiment_design.csv` do not change.*