Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
| System files (Q5) | Ignored silently; `single` check and sibling uniqueness are case-insensitive |
| Too-deep folders (Q7) | Error |
| Folders named `single` (Q23) | Accepted as the level's value `single` when lower case and the only folder in its place (MorphoBricks one-member level, MorphoRelate M3); another case or other folders beside it stay `reserved_folder_name` |
| Channel names (Q1) | Manifest only (from OME), warning `channel_names_differ`; CSV stays at schema 1 |
| Channel names (Q1) | Manifest only (from OME); `channel_names_differ` was a warning, an error since issue #37 (channels are matched by number, no override); names trimmed of leading/trailing whitespace at read time for every image (manifest `channel_names_trimmed`), inner whitespace compared exactly; copies leave unnamed channels unlabelled; CSV stays at schema 1 |
| Plane coordinates (Q17) | `z`, `c`, `t` always written (0 for absent axes), `plane_id` always `::z=…,c=…,t=…`; CSV schema 2, version-1 tables recomputed on the next run. Superseded for the CSV by Q27: rows have `z`, `t` and channels in columns |
| Channels across columns (Q27, P3b) | CSV schema 3 with the brick's writer: one row per XYC plane, per-channel CellProfiler-style columns, `plane_id` `::z=…,t=…`. Channels read one at a time (owner's choice); memory estimate adds the plane's finished bin counts. `different_channel_counts` error before any read (headers in step 1, confirmed roles in step 2); `different_plane_counts` without channels. Schema 1/2 tables recomputed; manifest version 1 records schema 3; step 4 output unchanged. Brick first pinned by commit `0e53e67`, now by its tag v0.2.0 (Q28) |
| OME-NGFF labels (Q18) | Read-only helpers `ngff_label_names`/`open_ngff_labels` for later bricks (CellPatcher masks); label images are never analysed and the outputs do not change |
Expand Down
17 changes: 10 additions & 7 deletions docs/contract.md

Large diffs are not rendered by default.

13 changes: 12 additions & 1 deletion docs/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ a run manifest (`run_manifest.json`) records the confirmed labels and axis mappi
settings, package versions and each source file's size and modification time. It also
records each file's channel names and significant bits when metadata provides them, for
information only; channel names that differ between files at the same channel index are
reported as a warning.
an error (question 1, revised after the data-flow audit, issue #37).

## 7. Incremental updates

Expand Down Expand Up @@ -406,6 +406,17 @@ Raised during review; to be decided before the milestones that need them:
*Decided (October 2026): record channel names per file in the run manifest, not in the
CSV, and warn when the names at the same channel index differ between files. The CSV
stays at schema version 1.*
*Revised (October 2026, data-flow audit, issue #37): names that differ at the same
channel index are the error `channel_names_differ`, because channels are matched by
number and a different stain would be pooled and scaled as the same channel. Only named
channels are compared; there is no override, so such images are exported again or
excluded. Owner decision (October 2026, review of #42): leading and trailing whitespace
is removed from channel names at read time, for every image (sources and this brick's
copies), and recorded in the manifest (`channel_names_trimmed`); whitespace inside a
name is still compared exactly (`DA PI` and `DAPI` differ, and the message quotes names
that differ only in whitespace). OME-Zarr copies
leave unnamed channels without a label instead of the placeholder `Channel <i>`, so a
copy never names a channel its source does not.*
2. Multi-series and pyramidal TIFFs: use only full-resolution level 0, and block files
with several positions?
*Decided (October 2026): yes. Pyramidal files are read at full-resolution level 0 only;
Expand Down
4 changes: 2 additions & 2 deletions docs/user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -467,7 +467,7 @@ The default output folder is next to the experiment folder:
`ome_zarr/control/bio_rep1/tech_rep1/img_01.ome.zarr`. The full-resolution level holds
exactly the source pixel values; smaller levels are averages for fast viewing (for
example in napari with the `napari-ome-zarr` plugin). Pixel sizes and channel names are
copied from the metadata when present, each axis with its own unit. Copies of unchanged
copied from the metadata when present (an unnamed channel stays unnamed), each axis with its own unit. Copies of unchanged
images are kept on later runs, copies of deleted or excluded images are removed. Copies
written by an earlier version with the X unit on every axis are converted again once,
with their own units; the table does not change. Switching back to reading the
Expand Down Expand Up @@ -556,7 +556,7 @@ continues.
| `repeat_intensity_shift` | Warning (intensity check) | A biological repeat is brighter or darker than the other repeats of its condition. Check whether it was prepared or acquired differently; normalising per biological repeat in later bricks removes such differences. |
| `saturated_image` | Warning (intensity check) | More than the limit (default 1 %) of an image's pixels in a channel are at the largest value of its pixel type, so their true intensity is lost. Check the exposure, gain or laser power. |
| `no_intensity_spread` | Warning (intensity check) | An image's channel has the same 1st and 99.8th percentile (empty or blank), so it cannot be normalised on its own. Check the channel, or exclude the image. |
| `channel_names_differ` | Warning | Channel names differ between images at the same position. Check the channel order. |
| `channel_names_differ` | Error | Channel names differ between images at the same channel number, so different stains would be pooled as one channel. Spaces at the start or end of a name are ignored (and noted in the manifest); otherwise names must match exactly, and names that differ only in inner spaces are quoted and reported as such. Check the channel order; export the images again with the same channel order and names, or exclude the images whose channels differ. |
| `non_finite_pixels` | Warning | NaN or infinite pixels are left out of the histograms and counted. |
| `no_finite_pixels` | Warning | Some planes have a channel without finite pixels; those channels are left out of group statistics. |
| `excluded_images`, `excluded_path_not_found` | Warning | Images excluded as requested, or an excluded path that matches no image. |
Expand Down
3 changes: 2 additions & 1 deletion notebooks/MorphoBricks_Preprocessing.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,8 @@
" biological repeat that a condition does not have. Below it, **Show consistency warnings**\n",
" lists differences between the images: pixel type, dimensions, sizes, numbers of channels,\n",
" slices or time points, channel names and pixel sizes. Warnings never stop you; each says\n",
" what to check. Different numbers of channels are an error: every image needs the same channels.\n",
" what to check. Different numbers of channels, or different channel names at the same channel\n",
" number, are errors: every image needs the same channels in the same order.\n",
"\n",
"> 💡 **Tip:** to change a label, rename the folder and click **Check folder** again.\n",
">\n",
Expand Down
8 changes: 8 additions & 0 deletions src/morphobricks_preprocessing/manifest.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,9 @@ class FileEntry:
ome_zarr_path
For a TIFF converted to OME-Zarr, the converted copy relative to the
output folder (its rows are computed from it); otherwise None.
channel_names_trimmed
True when leading or trailing whitespace was removed from the image's
channel names as they were read (names are recorded trimmed).
read_in_blocks
True when the image's planes are too large for the memory limit and are
read in blocks. For information only: the rows are the same either way.
Expand All @@ -137,6 +140,7 @@ class FileEntry:
ome_zarr_path: str | None = None
read_in_blocks: bool = False
position_id: str | None = None
channel_names_trimmed: bool = False

def same_source(self, other: FileEntry) -> bool:
"""Return True if ``other`` describes the same unchanged file, read the same way."""
Expand Down Expand Up @@ -403,6 +407,7 @@ def file_entry_for(
significant_bits: int | None = None,
ome_zarr_path: str | None = None,
read_in_blocks: bool = False,
channel_names_trimmed: bool = False,
) -> FileEntry:
"""Describe a source image as it is on disk now, with its position identifier."""
size, mtime_ns = source_fingerprint(Path(root) / relative_path)
Expand All @@ -419,6 +424,7 @@ def file_entry_for(
ome_zarr_path=ome_zarr_path,
read_in_blocks=read_in_blocks,
position_id=position_id(relative_path),
channel_names_trimmed=channel_names_trimmed,
)


Expand Down Expand Up @@ -524,6 +530,7 @@ def _file_dict(entry: FileEntry) -> dict:
"ome_zarr_path": entry.ome_zarr_path,
"read_in_blocks": entry.read_in_blocks,
"position_id": entry.position_id,
"channel_names_trimmed": entry.channel_names_trimmed,
}


Expand All @@ -546,6 +553,7 @@ def _file_entry(data: dict) -> FileEntry:
ome_zarr_path=None if data.get("ome_zarr_path") is None else str(data["ome_zarr_path"]),
read_in_blocks=bool(data.get("read_in_blocks", False)),
position_id=None if data.get("position_id") is None else str(data["position_id"]),
channel_names_trimmed=bool(data.get("channel_names_trimmed", False)),
)


Expand Down
54 changes: 40 additions & 14 deletions src/morphobricks_preprocessing/metadata.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,13 @@
from .discovery import DiscoveryResult
from .physical_sizes import PhysicalSizes, _tiff_sizes
from .validation import ValidationIssue, error, has_errors, plural, warning
from .zarr_input import NgffError, ZarrUnavailableError, is_zarr_name, open_ngff
from .zarr_input import (
NgffError,
ZarrUnavailableError,
is_zarr_name,
open_ngff,
read_channel_names,
)

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -68,6 +74,11 @@ class TiffMetadata:
channel_names
Channel names from OME metadata, by channel index; None for a channel
without a name. Empty when the file has no OME channel information.
Leading and trailing whitespace is removed from every name; whitespace
inside a name is kept and compared exactly.
channel_names_trimmed
True when leading or trailing whitespace was removed from a channel name
(recorded in the run manifest).
significant_bits
Bits per pixel that carry data (e.g. 12 for a 12-bit camera in 16-bit
files), from OME metadata; None when not given. For information only.
Expand Down Expand Up @@ -98,6 +109,7 @@ class TiffMetadata:
kind: str = TIFF
physical_sizes: PhysicalSizes = PhysicalSizes()
rgb_samples: bool = False
channel_names_trimmed: bool = False

def length(self, axis: str) -> int:
"""Return the length of the dimension labelled ``axis``, 1 if there is none."""
Expand Down Expand Up @@ -222,6 +234,7 @@ def inspect_ome_zarr(path: str | os.PathLike, relative_path: str | None = None)
series_count=image.image_count,
page_count=0,
channel_names=image.channel_names,
channel_names_trimmed=image.channel_names_trimmed,
kind=OME_ZARR,
physical_sizes=image.physical_sizes,
)
Expand Down Expand Up @@ -287,12 +300,12 @@ def check_metadata(files: Iterable[TiffMetadata]) -> list[ValidationIssue]:
unsupported pixel types, more than one pixel type, TIFF files whose
dimensions are arranged differently, and images with different numbers of
channels (``different_channel_counts``; the C axis, or the RGB samples
without one, until the axis roles are confirmed). OME-Zarr images declare their axes,
without one, until the axis roles are confirmed), and channel names that differ at the
same channel (``channel_names_differ``). OME-Zarr images declare their axes,
so their arrangement is checked by role during the axis review instead.

Reports, as warnings: images of different widths or heights
(``different_sizes``), channel names that differ at the same channel
(``channel_names_differ``), different numbers of depth slices, time points
(``different_sizes``), different numbers of depth slices, time points
or planes along another dimension (``different_plane_counts``),
and physical pixel sizes that differ (``different_pixel_sizes``) or are
known for only some images (``missing_pixel_size``).
Expand Down Expand Up @@ -423,12 +436,13 @@ def _ome_details(tif: tifffile.TiffFile) -> dict:
pixels = next((e for e in root.iter() if _local_name(e.tag) == "Pixels"), None)
if pixels is None:
return {}
names = tuple(
channel.get("Name") or None for channel in pixels if _local_name(channel.tag) == "Channel"
names, trimmed = read_channel_names(
channel.get("Name") for channel in pixels if _local_name(channel.tag) == "Channel"
)
bits = pixels.get("SignificantBits")
return {
"channel_names": names if any(names) else (),
"channel_names": names,
"channel_names_trimmed": trimmed,
"significant_bits": int(bits) if bits and bits.isdigit() else None,
}

Expand All @@ -455,21 +469,33 @@ def _check_channel_names(files: list[TiffMetadata]) -> list[ValidationIssue]:
descriptions: list[str] = []
for index in range(max(len(f.channel_names) for f in named)):
at_index = [f for f in named if index < len(f.channel_names) and f.channel_names[index]]
if len({f.channel_names[index] for f in at_index}) > 1:
summary, minority = _breakdown(at_index, lambda f, i=index: f.channel_names[i], str)
descriptions.append(f"channel {index}: {summary}")
names = {f.channel_names[index] for f in at_index}
if len(names) > 1:
# Names are compared exactly. When they differ only in whitespace, quote them
# and say so: the difference would otherwise be invisible in the message.
only_spaces = len({" ".join(name.split()) for name in names}) == 1
summary, minority = _breakdown(
at_index,
lambda f, i=index: f.channel_names[i],
(lambda name: f'"{name}"') if only_spaces else str,
)
note = " (the names differ only in spaces or other whitespace)" if only_spaces else ""
descriptions.append(f"channel {index}: {summary}{note}")
differing.extend(minority)
if not differing:
return []
# An error (issue #37): channels are matched by number, so a different stain at the
# same number would be pooled and scaled as one channel.
return [
warning(
error(
"channel_names_differ",
"The images name their channels differently at the same channel position ("
+ "; ".join(descriptions)
+ "). Planes with the same channel number may show different stains.",
+ "). Channels are matched by their number, so different stains would be pooled "
"as one channel.",
sorted(set(differing)),
"No action needed if the names only differ in spelling. Otherwise check the "
"channel order of these images before comparing channels.",
"Check the channel order of these images. Export them again with the same channel "
"order and names, or exclude the images whose channels differ.",
)
]

Expand Down
7 changes: 7 additions & 0 deletions src/morphobricks_preprocessing/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@
from .zarr_export import (
ConversionSettings,
OmeZarrWriter,
copy_channels_current,
copy_units_current,
is_copy_path,
ome_zarr_relative_path,
Expand Down Expand Up @@ -515,7 +516,12 @@ def stop(**counts) -> RunResult:
significant_bits=header.significant_bits,
ome_zarr_path=copy,
read_in_blocks=rel in in_blocks,
channel_names_trimmed=header.channel_names_trimmed,
)
if header.channel_names_trimmed:
logger.info(
"Leading or trailing whitespace was removed from the channel names of %s", rel
)
before = previous.file(rel) if previous is not None else None
if before is None:
new.append(rel)
Expand All @@ -532,6 +538,7 @@ def stop(**counts) -> RunResult:
(output / copy).is_dir()
and same_conversion
and copy_units_current(output / copy, header.physical_sizes)
and copy_channels_current(output / copy, header.channel_names)
)
)
):
Expand Down
55 changes: 46 additions & 9 deletions src/morphobricks_preprocessing/zarr_export.py
Original file line number Diff line number Diff line change
Expand Up @@ -475,18 +475,26 @@ def _omero(self) -> dict:
channels = []
for c in range(self.shape[self.roles.index("C")]):
low, high = self._channel_ranges.get(c, (0.0, 1.0))
label = self.channel_names[c] if c < len(self.channel_names) else None
channels.append(
{
"label": label or f"Channel {c}",
"active": True,
"color": "FFFFFF",
"window": {"min": low, "max": high, "start": low, "end": high},
}
)
channel = {
"active": True,
"color": "FFFFFF",
"window": {"min": low, "max": high, "start": low, "end": high},
}
# Only names from the source's metadata are written. An unnamed channel gets
# no label rather than a made-up one, so the copy reads back unnamed, like its
# source, and never fails the exact channel-name check (issue #37).
label = _channel_label(self.channel_names, c)
if label is not None:
channel["label"] = label
channels.append(channel)
return {"version": NGFF_VERSION, "channels": channels}


def _channel_label(names: Sequence[str | None], channel: int) -> str | None:
"""Return the name of ``channel`` in ``names``; None when it is unnamed."""
return (names[channel] if channel < len(names) else None) or None


def is_copy_path(output_folder: str | os.PathLike, relative_path: str) -> bool:
"""Return True if ``relative_path`` names an OME-Zarr copy inside ``<output>/ome_zarr``.

Expand Down Expand Up @@ -523,6 +531,35 @@ def copy_units_current(copy: str | os.PathLike, sizes: PhysicalSizes) -> bool:
return False


def copy_channels_current(copy: str | os.PathLike, channel_names: Sequence[str | None]) -> bool:
"""Return True if a copy's channel labels are exactly the names its source gives now.

Copies written before unnamed channels were left without a label carry the
placeholder ``Channel <i>``; such copies are written again on the next run, so the
copy never names a channel its source does not. Only the copy's metadata file is
read. A copy whose metadata cannot be read counts as out of date.

Parameters
----------
copy
The copy's folder (``<output>/ome_zarr/....ome.zarr``).
channel_names
The source image's channel names, as its header gives them (None when unnamed).
"""
try:
attrs = json.loads((Path(copy) / ".zattrs").read_text(encoding="utf-8"))
if "omero" not in attrs:
return True
channels = attrs["omero"]["channels"]
return all(
channel.get("label") == _channel_label(channel_names, c)
for c, channel in enumerate(channels)
)
except (OSError, ValueError, KeyError, IndexError, TypeError, AttributeError):
logger.debug("Cannot read the channel labels of the copy %s", copy, exc_info=True)
return False


# zarr 2 / 3 ---------------------------------------------------------------------------


Expand Down
Loading
Loading