Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,15 @@ are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the

## Unreleased

- **Images without header channel names are listed in a described run (follow-up of #218, owner
decision 2026-11-08).** An image whose header names no channel, or not every one, still passes
the per-unit header checks (§4.3, fixture v12: no evidence needed, the confirmed layout is
used), with no warning, but is now recorded: the unit's record in the manifest's
`experiment_description` entry gets `images_without_header_channel_names` (image, `none` or
`partial`, and the zero-based channel positions without a name; omitted for a unit whose images
are all named), `read_manifest` reads it strictly, `RunInputs.images_without_header_channel_names()`
returns it read-only (None without a description), and the report text and JSON list the images.
A run without a description is unchanged: no new manifest or report key.
- **A draft written under another Spec commit is refused at resume (#223, owner decision of
2026-11-08).** `load_draft` and `publish_draft` raise `DraftFileError` with the new code
`spec_commit_mismatch` when the draft's `spec_commit` is not the pinned `SPEC_COMMIT`;
Expand Down
2 changes: 1 addition & 1 deletion docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ interpretation.
| `narrow_histogram_ranges` | The channel planes whose range check found the central range too narrow (§4), sorted by plane and channel: `relative_path`, `plane_id`, `channel` and the fields of the normalization brick's `RangeCheck` (`histogram_range`, `central_percentiles`, `central_bins`, `robust_quantiles`, `robust_limits`, `outside_robust_count`, `robust_central_bins`, `min_central_bins`). Empty when there are none; absent in manifests written before follow-up P6. Optional key: the manifest stays version 1. |
| `conversion_record_sha256` | SHA-256 of the conversion record (`.morphobricks-conversion.json`) in the experiment root when the experiment is a tree of named-axis copies (design v0.4 §5, §17); null otherwise. `RunInputs` checks it (owner decision of 2 November 2026, plan question 59): a record changed or gone since the run is refused, as for `calibration_sha256`. A changed record changes no row (the copies are checked by their fingerprints like any image). |
| `calibration_sha256` | SHA-256 of the output folder's `calibration.csv` as the run read it (§9.3); null when there was none (or absent in manifests written before PP-8). `RunInputs` checks it. |
| `experiment_description` | Only for a run with a published `morphobricks.toml` (the key is left out otherwise; experiment-v1 §4.5a, §4.6, §9.3). An object: `sha256` (of the exact bytes the run read: provenance, never a cache key), `schema_version`, `section_digests` (`conditions`, `channels`, `repeats`, `masks`), `units` (one object per declared unit: its repeat, condition and technical repeat IDs and `image_type_header_verified`, false for a format whose header states no colour interpretation), `masks` (per declared mask ID: `source_type`, `representation`, `status`, `images_with_masks`, `images` and each file with its fingerprint and its own `status`) and `records` (the lines of `morphobricks.records.jsonl`, copied unchanged). A mask file is `checked` only when the mask step recorded it in `mask_representation.json` with the same fingerprint, representation and source type; otherwise `not yet checked`. A run refuses units declaring more than one channel layout (`R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`). |
| `experiment_description` | Only for a run with a published `morphobricks.toml` (the key is left out otherwise; experiment-v1 §4.5a, §4.6, §9.3). An object: `sha256` (of the exact bytes the run read: provenance, never a cache key), `schema_version`, `section_digests` (`conditions`, `channels`, `repeats`, `masks`), `units` (one object per declared unit: its repeat, condition and technical repeat IDs and `image_type_header_verified`, false for a format whose header states no colour interpretation; and, only for a unit that has such images, `images_without_header_channel_names`: a list sorted by `image` (relative path) of objects `image`, `header_names` (`none` when the header names no channel, `partial` when only some) and `channel_positions` (the zero-based C positions without a header name). Such an image passes the header checks, needs no evidence and the confirmed layout is used (§4.3); it is listed, not warned about, and the key is omitted for a unit whose images are all fully named; RGB units list nothing), `masks` (per declared mask ID: `source_type`, `representation`, `status`, `images_with_masks`, `images` and each file with its fingerprint and its own `status`) and `records` (the lines of `morphobricks.records.jsonl`, copied unchanged). A mask file is `checked` only when the mask step recorded it in `mask_representation.json` with the same fingerprint, representation and source type; otherwise `not yet checked`. A run refuses units declaring more than one channel layout (`R_HETEROGENEOUS_LAYOUT_NOT_ENABLED`). |
| Version 1 | A manifest written before v0.4.0 (`"1"`) is still read, for reporting, but no row of it is reused: its axes were never checked. The next run recomputes every row ("the earlier run was written before v0.4.0, without the axis check"). Bricks refuse it. |
| Row reuse | A file's rows are reused only when its row key is the earlier run's (`fingerprint.RowKey`, one rule for "the same file, read the same way", audit D5): its fingerprint (`fingerprint.Fingerprint`: size and modification time, and for an OME-Zarr image the inventory digest), labels, axes, roles, `singleton_roles`, `axes_from` and the path it is read from (so moving or converting the experiment recomputes it), and the settings that change rows (bins, histogram range, robust quantiles, range-check limit). **The identity rule is the fingerprint, which is metadata** (design v0.4 §1): its known blind spot is a file rewritten with the same size whose modification time (for an OME-Zarr image, every changed file's) is set back to its earlier value; its earlier rows are then reused, its staged rows resumed, and `RunInputs` accepts it. |
| Content verification (on by default, owner 7 November 2026, #159; opt-in before) | Every run does it unless it opts out with `RunMode(name, verify_content=False)` (`run_experiment(mode=...)`, CLI `--no-verify-content`, the notebook's "Verify each image's content" box unticked); the opt-out is recorded as `settings.verify_content: false`. It closes that blind spot: each analysed image's `content_sha256` is taken after its fingerprint and recorded; it is part of the row key, so an earlier or staged row is reused only for the same content; an image read must still have it after its pixels are read (else `source_changed_during_run`); every recorded image (reused, resumed or read; an OME-Zarr image by the same store digest) is hashed once more as the last step before the commit, after the generation is staged, flushed and recorded and right before the manifest's rename (`staged_generation(before_commit=...)`, review of rc5), and a mismatch is `source_changed_during_run`: the publication folder is discarded and the earlier generation is intact, as for a fingerprint change; an image that cannot be hashed is `unreadable_planes`; and `RunInputs.open` of such a run checks every image's content SHA-256 once (reads after `open` check the fingerprint). It is a recorded input (`settings.verify_content`): a run given none keeps the recorded choice (true without one: a first run, or a version 1 manifest); turning it on or off recomputes every row ("content verification (verify_content) was turned on/off"), is reported as the change `verify_content`, and discards staged rows of the other mode with that reason. It reads every analysed image whole twice more per run (when its entry is made and before the commit), once more for each image read, and once for each `RunInputs.open`; the cost is in the pull request that made it the default (about 3 seconds per 1.5 GB read, measured at 1 GB/s of SHA-256). A run that verifies writes manifest version 3. Masks are hashed too (issue #160): each mask's content SHA-256 is taken once the header checks pass and again as the last step before the commit, and a mismatch (or a mask that cannot be hashed) is `mask_changed_during_run`; it is not recorded in the manifest (no mask has rows to reuse), so the manifest's mask entries are unchanged. Not covered: a change between the header read and the first hashing with its metadata kept (the rows are of the hashed content, the header may be of the earlier one); and a change in the moment between the last hashing and the rename, or after the commit (`RunInputs.open` hashes again). |
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,7 @@ baseline = "tools/code_rules_baseline.json"
"morphobricks_preprocessing.run_inputs_image" = "pure"
"morphobricks_preprocessing.description_layouts" = "pure"
"morphobricks_preprocessing.description_record" = "pure"
"morphobricks_preprocessing.description_unnamed" = "pure"
"morphobricks_preprocessing.description_run_text" = "pure"
"morphobricks_preprocessing.mask_representation" = "io"
"morphobricks_preprocessing.description_transition" = "pure"
Expand Down
43 changes: 27 additions & 16 deletions src/morphobricks_preprocessing/description_record.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from collections.abc import Mapping

from .description import Description
from .description_unnamed import FIELD, checked_items, unnamed_items
from .description_write import section_digests

KEY = "experiment_description"
Expand All @@ -22,33 +23,40 @@


def unit_records(
description: Description, verified: Mapping[tuple[str, str, str], bool]
description: Description,
verified: Mapping[tuple[str, str, str], bool],
unnamed: Mapping[tuple[str, str, str], Mapping[str, tuple[int, ...]]] | None = None,
) -> list[dict[str, object]]:
"""Return every declared unit with whether its image type was checked against a header.

``verified`` maps ``(condition folder, repeat folder, technical repeat folder)`` to the
check's outcome; a unit it does not name is not verified. It is False for a unit whose
format states no colour interpretation (OME-Zarr: the declaration stands on the user's
confirmation) and for a unit with no readable image.

``unnamed`` maps the same keys to the images whose header names not every channel and the
positions without a name (:mod:`.description_unnamed`); a unit with such images gets the
list under ``images_without_header_channel_names``, any other unit has no such key.
"""
return [
{
"repeat": repeat_id,
"condition": condition_id,
"technical_repeat": unit_id,
"image_type_header_verified": verified.get(
(
out = []
for repeat_id, repeat in sorted(description.repeats.items()):
for condition_id, member in sorted(repeat.members.items()):
for unit_id, unit in sorted(member.technical_repeats.items()):
key = (
description.conditions[condition_id].source_folder,
member.source_repeat,
unit.source_folder,
),
False,
),
}
for repeat_id, repeat in sorted(description.repeats.items())
for condition_id, member in sorted(repeat.members.items())
for unit_id, unit in sorted(member.technical_repeats.items())
]
)
record: dict[str, object] = {
"repeat": repeat_id,
"condition": condition_id,
"technical_repeat": unit_id,
"image_type_header_verified": verified.get(key, False),
}
if (unnamed or {}).get(key):
record[FIELD] = unnamed_items(unnamed[key], len(unit.channels))
out.append(record)
return out


def description_record(
Expand Down Expand Up @@ -94,4 +102,7 @@ def recorded_description(value: object) -> dict | None:
):
if not isinstance(value[name], kind):
raise ValueError(f"{KEY}.{name} is not a {kind.__name__}")
for index, unit in enumerate(value["units"]):
if isinstance(unit, dict) and FIELD in unit:
checked_items(unit[FIELD], f"{KEY}.units[{index}]")
return value
95 changes: 95 additions & 0 deletions src/morphobricks_preprocessing/description_unnamed.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
"""Images of a described run whose header names no channel, or not every one (experiment-v1 §4.3).

Such an image passes the per-unit header checks: it gives no evidence against the user's
confirmed layout, which is used. It is not warned about; it is *listed*, so a reader can see
which images stand on the confirmation alone. The list is part of the unit's record in the
manifest's ``experiment_description`` entry (:mod:`.description_record`), under
:data:`FIELD`, and only for a unit that has such images: a unit whose images are all fully
named has no such key. Each item names the image (relative to the experiment root), whether
its header names no channel (``"none"``) or only some (``"partial"``), and the zero-based C
positions without a name. A run without a description records nothing of this.
"""

from __future__ import annotations

from collections.abc import Iterable, Mapping

from .validation import plural

FIELD = "images_without_header_channel_names"
"""Key of a unit record that has images with unnamed channels."""
NONE = "none"
PARTIAL = "partial"
_KEYS = {"image", "header_names", "channel_positions"}


def unnamed_items(
unnamed: Mapping[str, tuple[int, ...]], declared_channels: int
) -> list[dict[str, object]]:
"""Return the manifest items of one unit's unnamed images, sorted by path."""
return [
{
"image": path,
"header_names": NONE if len(positions) == declared_channels else PARTIAL,
"channel_positions": list(positions),
}
for path, positions in sorted(unnamed.items())
]


def checked_items(value: object, where: str) -> None:
"""Refuse a malformed list of unnamed images of one unit record.

Raises
------
ValueError
When it is not a non-empty list of items with exactly the recorded keys, a string
image, ``none`` or ``partial``, and ascending distinct non-negative whole positions
(all of the unit's positions are not known here, so only ``none`` and ``partial``
are told apart by the writer).
"""
if not isinstance(value, list) or not value:
raise ValueError(f"{where}.{FIELD} must be a non-empty list")
for item in value:
if not isinstance(item, dict) or set(item) != _KEYS:
raise ValueError(f"{where}.{FIELD} items must hold exactly {', '.join(sorted(_KEYS))}")
if not isinstance(item["image"], str) or not item["image"]:
raise ValueError(f"{where}.{FIELD} image is not a path")
if item["header_names"] not in (NONE, PARTIAL):
raise ValueError(f"{where}.{FIELD} header_names is not {NONE} or {PARTIAL}")
positions = item["channel_positions"]
whole = isinstance(positions, list) and all(
isinstance(p, int) and not isinstance(p, bool) and p >= 0 for p in positions
)
if not whole or not positions or positions != sorted(set(positions)):
raise ValueError(f"{where}.{FIELD} channel_positions must be ascending positions")
paths = [item["image"] for item in value]
if paths != sorted(set(paths)):
raise ValueError(f"{where}.{FIELD} images must be sorted and distinct")


def listed_units(units: Iterable[Mapping[str, object]]) -> list[Mapping[str, object]]:
"""Return the unit records that list unnamed images."""
return [unit for unit in units if unit.get(FIELD)]


def text_lines(units: Iterable[Mapping[str, object]]) -> list[str]:
"""Describe the listed images for the report text; empty when there are none."""
listed = listed_units(units)
if not listed:
return []
count = sum(len(unit[FIELD]) for unit in listed) # type: ignore[arg-type]
lines = [
f"Images without header channel names: {plural(count, 'image')} in "
f"{plural(len(listed), 'unit')} (checked as passing; your confirmed layout is used)"
]
for unit in listed:
lines.append(
f" repeat {unit['repeat']} / condition {unit['condition']} / "
f"technical repeat {unit['technical_repeat']}"
)
for item in unit[FIELD]: # type: ignore[union-attr]
spots = ", ".join(str(p) for p in item["channel_positions"])
what = "no channel named" if item["header_names"] == NONE else "unnamed positions"
lines.append(f" {item['image']}: {what} (channel positions {spots}, from 0)")
return lines
13 changes: 13 additions & 0 deletions src/morphobricks_preprocessing/discovery_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@
from pathlib import Path
from typing import TYPE_CHECKING

from .description_unnamed import FIELD as UNNAMED_FIELD
from .description_unnamed import listed_units, text_lines
from .image_names import position_id
from .recorded_sizes import HEADER, PIXELS, STATED, RecordedSize
from .validation import ValidationIssue, plural
Expand Down Expand Up @@ -148,6 +150,7 @@ def text(self) -> str:
lines.extend(_folder_lines(data))
lines.extend(_image_lines(data))
lines.extend(_mask_lines(data))
lines.extend(text_lines(data.get(UNNAMED_FIELD, ())))
lines.extend(_outcome_lines(self.outcome))
lines.extend(_issue_lines(self.issues))
return "\n".join(lines) + "\n"
Expand All @@ -162,6 +165,7 @@ def build_report(
design: ExperimentDesign | None = None,
image_type: str | None = None,
outcome: RunOutcome | None = None,
unit_records: Sequence[Mapping[str, object]] | None = None,
) -> DiscoveryReport:
"""Build the report of one run from what it found and did.

Expand All @@ -181,6 +185,10 @@ def build_report(
The image type, or None.
outcome
What the run did with the earlier results.
unit_records
The manifest's unit records of a run with a published description, or None. Their
images without header channel names are listed under ``images_without_header_channel_names``
(:mod:`.description_unnamed`); the key is left out for a run without a description.

Returns
-------
Expand All @@ -204,6 +212,11 @@ def build_report(
"image_type": image_type,
"masks": _masks(pre),
}
if unit_records is not None:
data[UNNAMED_FIELD] = [
{k: v for k, v in unit.items() if k != "image_type_header_verified"}
for unit in listed_units(unit_records)
]
ordered = tuple(sorted(issues, key=lambda issue: not issue.is_error))
return DiscoveryReport(data, ordered, outcome)

Expand Down
Loading
Loading