Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,8 @@ src/morphobricks_preprocessing/
report.py plain-language summaries shared by CLI and notebook, group statistics tables
mask_text.py the masks' texts (mask_summary, mask_report_summary) and the notebook's
tables of the mask step (mask_type_rows, mask_relationship_rows,
mask_hierarchy_lines); moved from report.py in PP-10d
mask_hierarchy_lines, with the mask_hierarchy_cycle warning, #105);
moved from report.py in PP-10d
discovery_report.py the run report (discovery_report.json and .txt, design v0.4 §4): status,
manifest SHA-256, what was found, reused, recomputed and replaced
cli.py, __main__ `morphobricks-preprocessing check|run|masks|calibrate|design|convert` (also `python -m morphobricks_preprocessing`);
Expand Down Expand Up @@ -230,7 +231,7 @@ src/morphobricks_preprocessing/
notebook_axis_step.py step 2: role dropdowns, axis check, missing-axes card (_AxisStep)
notebook_axis_preview.py step 2: preview, confirmation, image type (_AxisPreview)
notebook_run.py step 3: settings, what will happen, the run (_RunStep)
notebook_results.py step 4: pooled statistics per group (_ResultsStep)
notebook_results.py step 4: pooled statistics per group and channel (_ResultsStep)
notebook_plots.py the PNG plots and IntensityPanel (the intensity checks card)
notebook_render.py HTML helpers: issues, balance, labels, role effect, captions, tiles
notebook_options.py the steps' choices and fixed settings (ROLE_OPTIONS, ...)
Expand Down Expand Up @@ -444,8 +445,12 @@ functions the tests use.
histogram equals the version-2 row's, whole and in blocks
(`tests/test_schema_v2_equivalence.py` with the committed fixture in `tests/data`,
written by `tests/data/make_v2_fixture.py` with the pre-P3b code).
- **Step 4** pools every channel of a group together, as with version 2
(`report.channel_histograms`), so its table and plot are unchanged.
- **Step 4** summarises each channel of a group on its own (issue #104): one
`GroupSummary` per group and channel (`channel`, `xyc_plane_count`,
`channel_histogram_count`), a `channel` column in the table and one plot panel per
channel. Pooling every channel together (as with version 2) only with
`group_summaries(..., pool_channels=True)`, labelled `all channels (pooled)`;
`GroupSummary.plane_count` is a deprecated alias of `channel_histogram_count`.
- **`position_id`** is filled in every row (D02, plan §12 question 20). Earlier tables with
it blank are filled on the next run without reading pixels.
- **Reading:** labels are always read back as text (`NA`, `001`, `TRUE` stay as typed).
Expand Down
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,37 @@ The design is `docs/restart/preprocessing-design-v0.4.md` in MorphoFeature. The
are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the
[migration guide](docs/migration-0.3-to-0.4.md).

## Unreleased

- **Group statistics per channel (#104).** `group_summaries` and the notebook's step 4 no
longer pool distinct channels together: each group has one summary per channel, and the
intensity curves are drawn in one panel per channel. Single-channel tables give the same
numbers as before.
- `GroupSummary` gains `channel` (the channel index, or `ALL_CHANNELS`/None for an
aggregate), `xyc_plane_count` (XYC planes) and `channel_histogram_count` (histograms
pooled). `plane_count`, which counted each channel of a plane as a plane, remains as a
deprecated property equal to `channel_histogram_count` and warns
(`DeprecationWarning`). The positional constructor changed: build summaries by
keyword, or use `group_summaries`.
- `group_table` adds the columns `channel`, `XYC planes` and `channel histograms` (in
place of `planes`).
- The earlier all-channel aggregate is available on request,
`group_summaries(..., pool_channels=True)`, and is labelled `all channels (pooled)`.
- A group whose planes have different numbers of channels is refused (`ValueError`).
- **Migration:** no file changes. `plane_histograms.csv` (schema 4), the manifest and
the other outputs are unchanged; step 4 writes no file. Code reading
`GroupSummary.plane_count` should read `xyc_plane_count` or `channel_histogram_count`
and select rows by `channel`; to reproduce a pre-#104 table, pass
`pool_channels=True`. No other MorphoBricks repository uses these names.

- **Cyclic mask hierarchies are named (#105).** When the direct-parent relations of the mask
step form a cycle (possible because pairs of types may be seen together in different
images), the hierarchy text no longer prints an empty or truncated tree: it starts with
the warning `mask_hierarchy_cycle`, naming every type and link of the cycle, and shows
the group as one `[cycle: …]` entry; every type is listed, and no direction is guessed.
New `MaskReport.hierarchy_cycles` and `mask_scan.hierarchy_cycles`. Acyclic hierarchies
print as before; `mask_relationships.json` is unchanged (report version 1).

## v0.4.0 (5 November 2026)

The final v0.4.0, with the small changes since v0.4.0rc5 (#147) and the release step: the
Expand Down
17 changes: 17 additions & 0 deletions docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,14 @@ on the next run without reading pixels.
Defined by the [normalization brick's contract][normalization contract] §1.2.3: groups
pool their planes' (representative, count) pairs and never average per-plane statistics.

This brick's optional group statistics (`group_summaries`, the notebook's step 4) pool
**one channel at a time** (issue #104): one summary per group and channel, with
`xyc_plane_count` (XYC planes) and `channel_histogram_count` (histograms pooled) reported
apart. Channels are pooled together only when asked (`pool_channels=True`), and the
summary then says so (`channel` None, shown as `all channels (pooled)`). A group whose
planes have different numbers of channels is refused (`ValueError`). These statistics are
shown only; no file is written, so no file schema depends on them.

## 7. Normalization parameters

Defined by the [normalization brick's contract][normalization contract] §1.2.4: every
Expand Down Expand Up @@ -538,6 +546,15 @@ relation nucleoli inside cells is kept in the report but not in the hierarchy. A
have several direct parents (cells inside an organoid and inside a region of the slide), and
two types that are the same objects are both kept.

**Cycles (issue #105).** Relations come from pairwise evidence summed over the images with
both masks, so types seen together in different images can form a cycle (A inside B, B
inside C, C inside A). No parent-child hierarchy can then be inferred, and none is guessed:
`MaskReport.hierarchy_cycles` names each group of types in a cycle, and the hierarchy text
(`mask_report_summary`, `mask_hierarchy_lines`, the notebook) starts with the named warning
`mask_hierarchy_cycle` listing every type and link of the cycle, then shows the group as one
`[cycle: A, B, C]` entry; every type is listed. `mask_relationships.json` is unchanged
(report version 1): its `hierarchy` keeps every direct link, cycles included.

The report is UTF-8 JSON (two-space indent, LF), written atomically to the output folder:

| Key | Meaning |
Expand Down
8 changes: 7 additions & 1 deletion docs/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -1000,7 +1000,13 @@ Raised during review; to be decided before the milestones that need them:
schema version 1 or 2 is recomputed completely on the next run; the manifest stays
version 1, records schema version 3, and a file's `plane_count` counts its XYC planes.
Step 4 reads the new layout and, as before, pools every channel of a group together,
so its table (whose "planes" still counts channel planes) and plot do not change. The
so its table (whose "planes" still counts channel planes) and plot do not change.
*Amended (6 November 2026, issue #104): step 4 no longer pools channels. Distinct
channels are different measurements (two fluorescence markers, or red, green and
blue), so each group is summarised and plotted channel by channel; the table has a
`channel` column, counts `XYC planes` and `channel histograms` apart, and an
all-channel aggregate is only made on explicit request (`pool_channels=True`) and
labelled `all channels (pooled)`. No file changes: step 4 writes nothing.* The
warning `no_finite_pixels` counts the planes with a channel without finite pixels. The
deprecated re-exports of the moved names (question 19) keep pointing at the brick's
current objects, now with its Milestone 2b signatures; names it added later are not
Expand Down
15 changes: 10 additions & 5 deletions docs/user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -543,11 +543,14 @@ results or comparing another grouping.
- A spinning wheel shows that work is in progress while the table is read, statistics
are calculated and the plot is drawn. The button says **Calculating…** and stays
disabled until the calculation finishes or an error is shown; you can then try again.
- The table lists pooled statistics per group: minimum, maximum, mean, standard
deviation, median and percentiles. Every channel of a group is pooled together, as
before; its **planes** column counts each channel of a plane as one plane.
- The plot compares the groups' intensity distributions: curves further right are
brighter.
- The table lists pooled statistics per group **and channel**: minimum, maximum, mean,
standard deviation, median and percentiles. Each channel (**C0**, **C1**, …) has its
own row, since different fluorescence markers (or the red, green and blue of an RGB
image) are different measurements and are never mixed. **XYC planes** counts the
planes (an image at one time point) and **channel histograms** the histograms pooled
in that row.
- The plot compares the groups' intensity distributions, one panel per channel: curves
further right are brighter.
- **Intensity checks** shows the same warnings, settings and plots as step 3 for the
chosen table, so tables of other experiments can be checked too. It needs only the
table, not the images.
Expand Down Expand Up @@ -1102,6 +1105,7 @@ continues.
| `channel_without_finite_pixels` | Warning (intensity check) | An image's channel has no finite pixels (every value NaN or infinite), so its intensities were not checked and it cannot be normalised from its own pixels. Exclude the image, or fix the channel in the source data and run again. |
| `channel_names_differ` | Error | Channel names differ between images at the same channel number, so different stains would be pooled as one channel. Spaces at the start or end of a name are ignored (and noted in the manifest); otherwise names must match exactly, and names that differ only in inner spaces are quoted and reported as such. Check the channel order; export the images again with the same channel order and names, or exclude the images whose channels differ. |
| `non_finite_pixels` | Warning | NaN or infinite pixels are left out of the histograms and counted. |
| `mask_hierarchy_cycle` | Warning (mask step) | The direct parents of some mask types form a cycle (for example A inside B in some images, B inside C in others and C inside A), so no parent-child hierarchy can be inferred for them. The hierarchy lists them together as `[cycle: …]` with every link. Check those masks; nothing is guessed. |
| `no_finite_pixels` | Warning | Some planes have a channel without finite pixels; those channels are left out of group statistics. |
| `excluded_images`, `excluded_path_not_found` | Warning | Images excluded as requested, or an excluded path that matches no image. |
| `masks_folder_not_found` | Error | The masks folder does not exist. Select the folder that holds one folder per kind of mask. |
Expand Down Expand Up @@ -1230,6 +1234,7 @@ masks = check.masks # MaskInventory: types, files, issues
cells = masks.for_image("control/bio_rep1/tech_rep1/img_01.tif")["Cells"] # MaskFile
report = scan_masks(masks, "/data/exp1") # reads the pixels, one image at a time
report.hierarchy # {"Nuclei": ("Cells",), "Cells": (), ...}: direct parents
report.hierarchy_cycles # (): types whose direct parents form a cycle, if any
write_mask_report(report, "/data/exp1_MorphoBricks_Preprocessing/mask_relationships.json")
```

Expand Down
40 changes: 39 additions & 1 deletion src/morphobricks_preprocessing/mask_scan.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@
import json
import logging
import os
from collections.abc import Callable, Mapping
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from types import MappingProxyType
Expand Down Expand Up @@ -195,6 +195,16 @@ def hierarchy(self) -> Mapping[str, tuple[str, ...]]:
parents[pair.child].append(pair.parent)
return MappingProxyType({name: tuple(sorted(found)) for name, found in parents.items()})

@property
def hierarchy_cycles(self) -> tuple[tuple[str, ...], ...]:
"""The groups of mask types whose direct parents form a cycle (issue #105).

Pairwise evidence from different images can make A inside B, B inside C and C
inside A: no parent-child hierarchy can then be inferred for them. Each group is
sorted, and the groups too; empty when the hierarchy has no cycle.
"""
return hierarchy_cycles(self.hierarchy)

@property
def same_objects(self) -> tuple[tuple[str, str], ...]:
"""Pairs of types that segment the same objects, each pair once."""
Expand Down Expand Up @@ -228,6 +238,34 @@ def to_dict(self) -> dict:
}


def hierarchy_cycles(hierarchy: Mapping[str, Sequence[str]]) -> tuple[tuple[str, ...], ...]:
"""Return the groups of names whose parents form a cycle, in ``{child: parents}``.

A group is a strongly connected set of two or more names, or one name that is its own
parent. Each group is sorted, and the groups too; empty for an acyclic hierarchy.
"""
names = {*hierarchy, *(parent for parents in hierarchy.values() for parent in parents)}
reach = {name: _ancestors(hierarchy, name) for name in names}
groups = {
tuple(sorted(other for other in found if name in reach[other]))
for name, found in reach.items()
if name in found
}
return tuple(sorted(groups))


def _ancestors(hierarchy: Mapping[str, Sequence[str]], name: str) -> set[str]:
"""Return the names reached from ``name`` through one or more parent links."""
found: set[str] = set()
todo = list(hierarchy.get(name, ()))
while todo:
parent = todo.pop()
if parent not in found:
found.add(parent)
todo.extend(hierarchy.get(parent, ()))
return found


def mask_thresholds(changes: Mapping[str, float] | None = None) -> Mapping[str, float]:
"""Return the default thresholds with some replaced, after checking them.

Expand Down
56 changes: 45 additions & 11 deletions src/morphobricks_preprocessing/mask_text.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,12 @@

from collections.abc import Mapping, Sequence

from .mask_scan import DEFAULT_THRESHOLDS, MaskReport, PairRelation
from .mask_scan import (
DEFAULT_THRESHOLDS,
MaskReport,
PairRelation,
hierarchy_cycles,
)
from .masks import MaskInventory
from .validation import plural

Expand Down Expand Up @@ -185,18 +190,47 @@ def _relationship_line(pair: PairRelation) -> str:


def _hierarchy_lines(hierarchy: Mapping[str, Sequence[str]]) -> list[str]:
"""Return the mask types as an indented tree; a type with two parents shows twice."""
children: dict[str, list[str]] = {name: [] for name in hierarchy}
"""Return the mask types as an indented tree; a type with two parents shows twice.

Every type is listed. Types whose direct parents form a cycle (issue #105) have no
parent-child hierarchy: a named warning ``mask_hierarchy_cycle`` comes first, with
every type and link of the cycle, and the tree shows them together as one
``[cycle: ...]`` entry (the strongly connected components), never guessing a direction.
"""
cycles = hierarchy_cycles(hierarchy)
group = {name: (name,) for name in hierarchy}
group.update({parent: (parent,) for parents in hierarchy.values() for parent in parents})
group.update({name: cycle for cycle in cycles for name in cycle})
children: dict[tuple[str, ...], set[tuple[str, ...]]] = {g: set() for g in group.values()}
has_parent: set[tuple[str, ...]] = set()
for name, parents in hierarchy.items():
for parent in parents:
children[parent].append(name)
lines: list[str] = []

def show(name: str, depth: int) -> None:
lines.append(f"{' ' * (depth + 1)}{name}")
for child in sorted(children[name]):
if group[parent] != group[name]:
children[group[parent]].add(group[name])
has_parent.add(group[name])
lines = [_cycle_warning(cycle, hierarchy) for cycle in cycles]

def show(node: tuple[str, ...], depth: int) -> None:
label = f"[cycle: {', '.join(node)}]" if node in cycles else node[0]
lines.append(f"{' ' * (depth + 1)}{label}")
for child in sorted(children[node]):
show(child, depth + 1)

for name in sorted(n for n, parents in hierarchy.items() if not parents):
show(name, 0)
for node in sorted(set(group.values()) - has_parent):
show(node, 0)
return lines


def _cycle_warning(cycle: tuple[str, ...], hierarchy: Mapping[str, Sequence[str]]) -> str:
"""Name a cycle's types and links, and say no hierarchy can be inferred for them."""
links = [
f"{name} in {parent}"
for name in cycle
for parent in hierarchy.get(name, ())
if parent in cycle
]
return (
f" Warning (mask_hierarchy_cycle): the direct parents of {', '.join(cycle)} form a "
f"cycle ({'; '.join(links)}), so no parent-child hierarchy can be inferred for "
"them; they are shown together below. Check these masks."
)
Loading
Loading