Strict description reader, phases 1-3 (Stage 1, PR 2 of 3)

AGENTS.md

@@ -92,6 +92,13 @@ src/morphobricks_preprocessing/
                     read_confirmed_design/ConfirmedDesign (the strict reader other bricks use, §16)
   design_values.py  the design file's header check (no column twice or unnamed, #83), booleans,
                     counts, dates and DesignFormatError (pure; re-exported by design.py)
+  description_rules.py  the one table of the experiment-description rules (Stage 1, Spec
+                    experiment-v1): codes, field tables, forbidden names, enumerations; compared
+                    with the vendored text by tests/test_spec_rules_table.py
+  description_report.py DescriptionIssue (code, table path, value), DescriptionReport
+  description_read.py   the reference reader's bytes -> phases: decode (1), header (2), the
+                    driver (read_structure: phases 1-3)
+  description_structure.py phase 3: closed fields, forbidden names, exact types, ranges, enums
   image_type.py     image type (P4): FLUORESCENCE/RGB, review_image_type (suggestion from
                     TiffMetadata.rgb_samples and the confirmed roles, whether RGB is possible via the
                     brick's check_rgb), image_type_issues (rgb_not_possible)

CHANGELOG.md

@@ -8,6 +8,17 @@ are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the
 
 ## Unreleased
 
+- **Experiment-description reader, phases 1-3 (Stage 1, PR 2).** New modules
+  `description_rules`, `description_report`, `description_read` and `description_structure`:
+  `read_structure(data: bytes)` runs the strict phases 1 to 3 of Spec experiment-v1 (bytes and
+  encoding, duplicate keys, header and version, structure, forbidden fields) and reports every
+  error of the first failing phase with its code, table path and value. Not used by any run,
+  notebook or CLI command yet.
+- **Spec fixtures vendored (Stage 1, PR 1).** `tests/spec_fixtures/` holds the
+  MorphoBricks-Spec conformance fixtures at a pinned commit, with a git tree-hash pin test,
+  a gate that makes a skipped `tests/test_spec_*.py` test a failure, and the "no fallback
+  rules" scan (conformance plan sections 1 and 2). Tests only: no change to the package,
+  the CLI, the notebook or any output.
 - **Group statistics per channel (#104).** `group_summaries` and the notebook's step 4 no
   longer pool distinct channels together: each group has one summary per channel, and the
   intensity curves are drawn in one panel per channel. Single-channel tables give the same

docs/development.md

@@ -101,6 +101,23 @@ is pinned to the release in `RELEASE_TAG` (plan §12 question 58);
 - Every acceptance check in the plan has a test.
 - A bug fix comes with a test that fails without it.
 
+## Spec conformance tests
+
+The experiment-description reader is tested against the fixtures of
+[MorphoBricks-Spec](https://github.com/CellMigrationLab/MorphoBricks-Spec)
+(conformance plan, sections 1 and 2). They are vendored in `tests/spec_fixtures/` at the full
+commit SHA in its `PIN.toml`, never a branch.
+
+- **Pin integrity:** `tests/test_spec_pin.py` fails when one byte of the vendored `fixtures/`
+  differs, or a file is added or missing (git tree hash against the pin), so a local edit
+  of a fixture cannot make a reader pass. Never edit the vendored files; updating the pin
+  is a reviewed change (`tests/spec_fixtures/README.md`).
+- **Skip is failure:** every test module named `tests/test_spec_*.py` is held by
+  `tests/spec_gate.py`: a skipped or xfailed test there fails the run.
+- **No second copy of the rules:** `tests/test_spec_no_fallback.py` scans the reader's
+  modules (`description*.py`) for a second table of codes or names and for lenient, older or
+  default-version paths.
+
 ## Versioning
 
 - Semantic versioning for the package.

pyproject.toml

@@ -66,7 +66,8 @@ addopts = "-ra"
 line-length = 100
 target-version = "py311"
 src = ["src", "tests", "tools"]
-extend-exclude = ["tests/guard_fixtures"]
+# The vendored Spec files are byte-exact copies (tests/spec_fixtures/README.md).
+extend-exclude = ["tests/guard_fixtures", "tests/spec_fixtures"]
 
 [tool.ruff.lint]
 select = [
@@ -171,6 +172,10 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing.dataset" = "pure"
 "morphobricks_preprocessing.design" = "io"
 "morphobricks_preprocessing.design_values" = "pure"
+"morphobricks_preprocessing.description_read" = "pure"
+"morphobricks_preprocessing.description_report" = "pure"
+"morphobricks_preprocessing.description_rules" = "pure"
+"morphobricks_preprocessing.description_structure" = "pure"
 "morphobricks_preprocessing.design_edit" = "pure"
 "morphobricks_preprocessing.discovery" = "io"
 "morphobricks_preprocessing.links" = "io"

src/morphobricks_preprocessing/description_read.py

@@ -0,0 +1,108 @@
+"""The reference reader of ``morphobricks.toml``: bytes in, phases and issues out.
+
+Experiment-v1 sections 2 and 6. The reader is given raw bytes (never decoded text, so a byte
+order mark or invalid UTF-8 reaches the encoding check) and runs the validation phases in
+order, stopping after the first phase that reports an error. Nothing is converted, trimmed,
+normalized or defaulted, and the only rules are those of ``description_rules`` and the phase
+modules. This module holds phases 1 (decode) and 2 (header) and the driver.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import re
+import tomllib
+from collections.abc import Callable, Mapping
+
+from .description_report import DescriptionReport, Issues
+from .description_rules import SCHEMA_ID, SUPPORTED_VERSIONS, Code
+from .description_structure import check_structure
+
+BYTE_ORDER_MARK = b"\xef\xbb\xbf"
+DUPLICATE = re.compile(
+    r"Cannot (?:declare .* twice|overwrite a value|mutate immutable namespace|redefine namespace)"
+    r"|Duplicate inline table key"
+)
+"""The ``tomllib`` messages for a key or table declared twice, in any spelling TOML treats as
+the same key (bare or quoted, table, dotted or inline). The Python 3.11 and 3.13 parsers share
+them; ``tests/test_spec_reader_phases_1_3.py`` covers each form."""
+
+
+def decode(data: bytes) -> tuple[dict[str, object] | None, Issues]:
+    """Phase 1: the bytes must be UTF-8 without a byte-order mark and valid TOML 1.0.
+
+    Returns
+    -------
+    tuple
+        The parsed document (None when phase 1 failed) and the phase's issues:
+        ``E_FILE_ENCODING``, ``E_DUPLICATE_KEY`` or ``E_TOML_SYNTAX``.
+    """
+    out = Issues()
+    if data.startswith(BYTE_ORDER_MARK):
+        out.add(Code.E_FILE_ENCODING, "<file>", "byte-order mark")
+        return None, out
+    try:
+        text = data.decode("utf-8")
+    except UnicodeDecodeError as error:
+        out.add(Code.E_FILE_ENCODING, "<file>", str(error))
+        return None, out
+    try:
+        return tomllib.loads(text), out
+    except tomllib.TOMLDecodeError as error:
+        message = str(error)
+        out.add(
+            Code.E_DUPLICATE_KEY if DUPLICATE.match(message) else Code.E_TOML_SYNTAX,
+            "<file>",
+            message,
+        )
+    except RecursionError:
+        out.add(Code.E_TOML_SYNTAX, "<file>", "nested too deeply to read")
+    return None, out
+
+
+def check_header(doc: Mapping[str, object]) -> Issues:
+    """Phase 2: ``schema`` and ``schema_version``, exact types, the supported version only.
+
+    A missing, unsupported or mistyped version stops the validation: no field of a version the
+    reader does not implement is interpreted (experiment-v1 section 6).
+    """
+    out = Issues()
+    if "schema" not in doc:
+        out.add(Code.E_SCHEMA_ID, "schema")
+    elif type(doc["schema"]) is not str:
+        out.add(Code.E_TYPE, "schema", doc["schema"])
+    elif doc["schema"] != SCHEMA_ID:
+        out.add(Code.E_SCHEMA_ID, "schema", doc["schema"])
+    if "schema_version" not in doc:
+        out.add(Code.E_SCHEMA_VERSION_MISSING, "schema_version")
+    elif type(doc["schema_version"]) is not int:
+        out.add(Code.E_TYPE, "schema_version", doc["schema_version"])
+    elif doc["schema_version"] not in SUPPORTED_VERSIONS:
+        out.add(Code.E_SCHEMA_VERSION_UNSUPPORTED, "schema_version", doc["schema_version"])
+    return out
+
+
+PHASES: tuple[tuple[int, Callable[[Mapping[str, object]], Issues]], ...] = (
+    (2, check_header),
+    (3, check_structure),
+)
+"""The document phases after decoding, in order, as (number, check)."""
+
+
+def read_structure(data: bytes) -> DescriptionReport:
+    """Run phases 1 to 3 on the raw bytes of a file and report the first failing phase.
+
+    This is the structural reading only: a file it accepts is not yet a validated
+    description (identities and semantics are phases 4 and 5).
+    """
+    if not isinstance(data, bytes):
+        raise TypeError("the reader is given the raw bytes of the file, not decoded text")
+    digest = hashlib.sha256(data).hexdigest()
+    doc, issues = decode(data)
+    if doc is None:
+        return DescriptionReport(digest, 1, tuple(issues.items), (), None)
+    for phase, check in PHASES:
+        issues = check(doc)
+        if issues:
+            return DescriptionReport(digest, phase, tuple(issues.items), (), doc)
+    return DescriptionReport(digest, PHASES[-1][0], (), (), doc)

src/morphobricks_preprocessing/description_report.py

@@ -0,0 +1,77 @@
+"""What the experiment-description reader reports: issues with code, table path and value.
+
+Experiment-v1 section 6: every error of the first failing phase is reported, each with its
+code, the table path (for example ``repeats.Day1.members.CTRL.technical_repeats.single.
+channels[2]``) and the offending value. Warnings are reported only for a file without errors.
+"""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Mapping
+from dataclasses import dataclass
+
+from .description_rules import Code
+
+BARE_KEY = re.compile(r"[A-Za-z0-9_-]+")
+
+
+def key_path(base: str, key: str) -> str:
+    """Return the table path of ``key`` under ``base``, quoting a key TOML would quote."""
+    name = key if BARE_KEY.fullmatch(key) else f'"{key}"'
+    return f"{base}.{name}" if base else name
+
+
+@dataclass(frozen=True)
+class DescriptionIssue:
+    """One error or warning: its code, table path and the offending value (None if absent)."""
+
+    code: Code
+    path: str
+    value: object = None
+
+
+class Issues:
+    """The issues of one validation phase, collected in the order they are found."""
+
+    def __init__(self) -> None:
+        self.items: list[DescriptionIssue] = []
+
+    def add(self, code: Code, path: str, value: object = None) -> None:
+        """Record one issue."""
+        self.items.append(DescriptionIssue(code, path, value))
+
+    def __bool__(self) -> bool:
+        return bool(self.items)
+
+
+@dataclass(frozen=True)
+class DescriptionReport:
+    """The outcome of reading one file.
+
+    ``phase`` is the phase that reported ``errors`` (1 to 5), or, for a file without errors,
+    the last phase that ran. ``document`` is the parsed TOML (never modified, never
+    normalized) once phase 1 succeeded, else None. ``sha256`` is the digest of the exact
+    bytes read: provenance, not a cache key (experiment-v1 section 2.6).
+    """
+
+    sha256: str
+    phase: int
+    errors: tuple[DescriptionIssue, ...]
+    warnings: tuple[DescriptionIssue, ...]
+    document: Mapping[str, object] | None
+
+    @property
+    def ok(self) -> bool:
+        """Return True when the file has no error."""
+        return not self.errors
+
+    @property
+    def codes(self) -> frozenset[Code]:
+        """Return the set of error codes (what conformance compares)."""
+        return frozenset(issue.code for issue in self.errors)
+
+    @property
+    def warning_codes(self) -> frozenset[Code]:
+        """Return the set of warning codes."""
+        return frozenset(issue.code for issue in self.warnings)

src/morphobricks_preprocessing/description_rules.py

@@ -0,0 +1,130 @@
+"""The one table of the experiment-description rules (experiment-v1 sections 3, 4 and 6).
+
+Codes, field tables, the forbidden names and the fixed enumerations live here and nowhere
+else in the reader (conformance plan section 1: no second copy of the rules).
+``tests/test_spec_rules_table.py`` compares every table with the vendored normative text;
+a rule that changes in the Spec changes here in the same pull request, with its fixtures.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from enum import StrEnum
+
+
+class Code(StrEnum):
+    """Every error and warning code of experiment-v1 section 6.
+
+    ``R_*`` and ``A_*`` are not the reader's: preprocessing's publication and the analysis
+    layer own them.
+    """
+
+    E_FILE_ENCODING = "E_FILE_ENCODING"
+    E_TOML_SYNTAX = "E_TOML_SYNTAX"
+    E_DUPLICATE_KEY = "E_DUPLICATE_KEY"
+    E_SCHEMA_ID = "E_SCHEMA_ID"
+    E_SCHEMA_VERSION_MISSING = "E_SCHEMA_VERSION_MISSING"
+    E_SCHEMA_VERSION_UNSUPPORTED = "E_SCHEMA_VERSION_UNSUPPORTED"
+    E_TYPE = "E_TYPE"
+    E_UNKNOWN_FIELD = "E_UNKNOWN_FIELD"
+    E_FORBIDDEN_FIELD = "E_FORBIDDEN_FIELD"
+    E_MISSING_FIELD = "E_MISSING_FIELD"
+    E_EMPTY = "E_EMPTY"
+    E_TEXT_EMPTY = "E_TEXT_EMPTY"
+    E_ENUM = "E_ENUM"
+    E_DATE_FORMAT = "E_DATE_FORMAT"
+    E_VALUE_RANGE = "E_VALUE_RANGE"
+    E_REPEAT_UNCONFIRMED = "E_REPEAT_UNCONFIRMED"
+    E_ID_TEXT = "E_ID_TEXT"
+    E_ID_SYNTAX = "E_ID_SYNTAX"
+    E_ID_RESERVED = "E_ID_RESERVED"
+    E_ID_NOT_NFC = "E_ID_NOT_NFC"
+    E_ID_CASE_COLLISION = "E_ID_CASE_COLLISION"
+    E_SINGLE_MISUSE = "E_SINGLE_MISUSE"
+    E_UNKNOWN_CONDITION = "E_UNKNOWN_CONDITION"
+    E_UNKNOWN_CHANNEL = "E_UNKNOWN_CHANNEL"
+    E_DUPLICATE_CHANNEL_IN_UNIT = "E_DUPLICATE_CHANNEL_IN_UNIT"
+    E_DUPLICATE_VALUE = "E_DUPLICATE_VALUE"
+    E_RGB_UNIT_CHANNELS = "E_RGB_UNIT_CHANNELS"
+    E_RGB_TOKEN_MISUSE = "E_RGB_TOKEN_MISUSE"
+    E_RGB_ALIAS = "E_RGB_ALIAS"
+    E_DISPLAY_NAME_COLLISION = "E_DISPLAY_NAME_COLLISION"
+    E_USED_ID_MISSING = "E_USED_ID_MISSING"
+    E_SOURCE_FOLDER_REUSED = "E_SOURCE_FOLDER_REUSED"
+    E_SOURCE_FOLDER_CONFLICT = "E_SOURCE_FOLDER_CONFLICT"
+    E_CONDITION_WITHOUT_MEMBERS = "E_CONDITION_WITHOUT_MEMBERS"
+    E_SOURCE_REPEAT_REUSED = "E_SOURCE_REPEAT_REUSED"
+    E_REPEAT_ID_SHADOWS_SOURCE = "E_REPEAT_ID_SHADOWS_SOURCE"
+    E_MASK_SOURCE_TYPE_REUSED = "E_MASK_SOURCE_TYPE_REUSED"
+    E_REFERENCE_UNKNOWN = "E_REFERENCE_UNKNOWN"
+    E_REFERENCE_SELF = "E_REFERENCE_SELF"
+    E_REFERENCE_CYCLE = "E_REFERENCE_CYCLE"
+    W_UNUSED_CHANNEL = "W_UNUSED_CHANNEL"
+    W_REPEAT_MISSING_CONDITIONS = "W_REPEAT_MISSING_CONDITIONS"
+    W_REFERENCE_UNPAIRED = "W_REFERENCE_UNPAIRED"
+
+
+@dataclass(frozen=True)
+class TableSpec:
+    """The closed set of fields of one table, in the order of the Spec's table.
+
+    ``required`` names the ones that must be present.
+    """
+
+    fields: tuple[str, ...]
+    required: frozenset[str]
+
+
+SCHEMA_ID = "morphobricks-experiment"
+SUPPORTED_VERSIONS = frozenset({1})
+"""The schema versions this reader implements: a constant, checked against nothing fetched
+(experiment-v1 section 7)."""
+
+REPRESENTATIONS = ("instances", "binary")
+IMAGE_TYPES = ("rgb",)
+
+TOP = TableSpec(
+    ("schema", "schema_version", "conditions", "channels", "repeats", "masks", "used_ids"),
+    frozenset({"schema", "schema_version", "conditions", "channels", "repeats", "used_ids"}),
+)
+CONDITION = TableSpec(
+    ("source_folder", "display_name", "reference", "target", "treatment_key"),
+    frozenset({"source_folder"}),
+)
+CHANNEL = TableSpec(("display_name", "header_names"), frozenset())
+REPEAT = TableSpec(
+    ("display_name", "confirmed", "confirmed_on", "members"),
+    frozenset({"confirmed", "confirmed_on", "members"}),
+)
+MEMBER = TableSpec(
+    ("source_repeat", "image_count", "technical_repeats"),
+    frozenset({"source_repeat", "image_count", "technical_repeats"}),
+)
+UNIT = TableSpec(
+    ("source_folder", "display_name", "channels", "image_type"),
+    frozenset({"source_folder", "channels"}),
+)
+MASK = TableSpec(
+    ("source_type", "representation", "source_channels", "display_name"),
+    frozenset({"source_type", "representation"}),
+)
+USED_IDS = TableSpec(
+    ("conditions", "channels", "repeats", "technical_repeats"),
+    frozenset({"conditions", "channels", "repeats", "technical_repeats"}),
+)
+
+FORBIDDEN = frozenset(
+    {
+        "axes", "shared", "plane_scope", "z", "projection",
+        "pixel_size", "calibration",
+        "fovs", "files", "paths", "overrides", "exclude", "fingerprints",
+        "bricks", "routes", "features", "normalization", "statistics", "contrasts",
+        "binary_mask_policy", "mask_conversion",
+        "control", "roles", "experiment",
+        "source_index", "channel_count",
+        "retired_repeats", "replaced_by",
+        "labchronicle", "provenance", "source_database",
+    }
+)  # fmt: skip
+"""Field names and ID keys refused as ``E_FORBIDDEN_FIELD``, in any table, compared exactly
+and case-sensitively (experiment-v1 section 4.7)."""