Publication writes the removed and image_count_changed records (RR2, RR1)

CHANGELOG.md

@@ -8,6 +8,23 @@ are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the
 
 ## Unreleased
 
+- **Re-pin the vendored Spec to 2975e0e (MorphoBricks-Spec PR 13, owner decision RR3).** Text only: the
+  `image_count_changed` record is written for every confirmed member whose count changed, also
+  beside other changes, and the order of the `removed` lines is stated; fixtures and the reference
+  validator are unchanged (the fixtures README alone differs). Every fixture is exact and the
+  cross-check against the reference validator has 0 differences.
+- **Publication writes the `removed` and `image_count_changed` records (owner decisions RR2 and
+  RR1).** `publish_description` takes the user's `removals` (each with its confirmation date) and
+  `recorded_on`. Every unit the candidate drops must be named by a removal (else `ValueError`,
+  nothing written) and is appended to `morphobricks.records.jsonl` as a `removed` line with all
+  six keys; a condition, repeat or technical repeat whose ID the removal leaves held nowhere gets
+  its own line first (keys it does not name are `null`). A change of a confirmed member's
+  `image_count` writes `image_count_changed` with the old and new counts and `recorded_on`
+  (required then, never read from the clock), also when the same publication changes other
+  things (RR3). Entity lines come first, each unit line once after them; the date of an entity
+  is the first removal answer that touches it; with an ID correction the lines carry the
+  published IDs. All lines go in the same single atomic append as
+  the re-confirmations, so the publication is whole or absent.
 - **`convert --project max|mean|sum [--z-range START:STOP]` (part of #176).** The command line of
   the recorded Z projection: no default method (`--project` alone or an unknown method is a
   parser error naming the three), `--z-range` zero-based and half-open for the whole

docs/conformance.md

@@ -8,7 +8,7 @@ real run; a skip is never counted as a pass.
 | | |
 |---|---|
 | Base commit | `68b2a4b` (`origin/main`, merge of PR 211); the run included the changes of PR 17 |
-| Spec pin | the full SHA in `tests/spec_fixtures/PIN.toml` (vendored `experiment-v1.md` at `b197ac1`) |
+| Spec pin | the full SHA in `tests/spec_fixtures/PIN.toml` (vendored `experiment-v1.md` at `2975e0e`) |
 | Command | `python tools/check.py` (Ruff, Ruff format, then `pytest -q` over the whole `tests/` folder), then `python tools/check.py --rules` |
 | Whole suite | 4324 passed, 4 skipped, 0 failed (301 s) |
 | Spec modules (`tests/test_spec_*.py`) | 1535 passed, 0 skipped, 0 failed (the Spec gate fails a skip or xfail there) |

pyproject.toml

@@ -272,6 +272,7 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing.run.planned" = "orchestration"
 "morphobricks_preprocessing.run.design_file" = "io"
 "morphobricks_preprocessing.run.description_file" = "io"
+"morphobricks_preprocessing.run.description_records" = "pure"
 "morphobricks_preprocessing.run.draft_file" = "io"
 "morphobricks_preprocessing.run.description_run" = "orchestration"
 "morphobricks_preprocessing.run.lock" = "io"

src/morphobricks_preprocessing/draft_format.py

@@ -33,7 +33,7 @@
 
 DRAFT_FORMAT = "morphobricks-description-draft"
 DRAFT_VERSION = 1
-SPEC_COMMIT = "b197ac19d7a577cabe5281a81fa0fa59aef37146"
+SPEC_COMMIT = "2975e0ee09d408d4a87da49d953f7988d4f9cd30"
 """The MorphoBricks-Spec commit the rules of this brick are pinned to (``tests/spec_fixtures``)."""
 BASE_MISSING = "R_DRAFT_BASE_MISSING"
 SHA256 = re.compile(r"[0-9a-f]{64}")

src/morphobricks_preprocessing/run/description_file.py

@@ -7,7 +7,9 @@
 compares the candidate with the published file (:mod:`..description_transition`), and only then
 replaces the file atomically. Every refusal is reported as a set and writes nothing.
 
-The user's re-confirmations that the candidate needed are appended, in the same publication, to
+The user's re-confirmations that the candidate needed, a ``removed`` record for every removal
+the user confirmed and an ``image_count_changed`` record for a changed image count
+(:mod:`.description_records`) are appended, in the same publication, to
 the append-only ``morphobricks.records.jsonl`` next to the file (one JSON object per line, of one
 of the three Spec record types, each with its fixed key set). The records are never edited or
 dropped: the new file is the old bytes plus the new lines, written in full to a temporary file and
@@ -40,6 +42,7 @@
     read_file_bytes,
     sha256_hex,
 )
+from ..description_removal import Removal
 from ..description_report import DescriptionIssue, key_path
 from ..description_rules import Refusal
 from ..description_transition import (
@@ -50,6 +53,7 @@
     unwaived,
 )
 from ..description_write import canonical_bytes
+from .description_records import image_count_records, removal_records
 from .lock import output_lock
 
 ABSENT = "absent"
@@ -222,8 +226,11 @@ def _compare(
     published: bytes | None,
     candidate: Description,
     confirmations: tuple[Reconfirmation, ...],
-) -> None:
-    """Refuse every ``R_*`` that applies, or a re-confirmation the candidate does not need."""
+) -> Description | None:
+    """Refuse every ``R_*`` that applies, or a re-confirmation the candidate does not need.
+
+    Returns the published description (None when there was none), parsed once.
+    """
     old = description_from_bytes(published) if published is not None else None
     issues = transition_issues(old, candidate) if old is not None else ()
     left = unwaived(issues, confirmations)
@@ -232,6 +239,7 @@ def _compare(
     if unused(issues, confirmations):
         names = ", ".join(f"{c.kind} {c.old_id}" for c in unused(issues, confirmations))
         raise ValueError(f"a re-confirmation names a change the candidate does not make: {names}")
+    return old
 
 
 def publish_description(
@@ -240,6 +248,8 @@ def publish_description(
     *,
     base_sha256: str | None,
     confirmations: tuple[Reconfirmation, ...] = (),
+    removals: tuple[Removal, ...] = (),
+    recorded_on: str | None = None,
 ) -> Publication:
     """Publish ``candidate`` as ``morphobricks.toml``, or refuse and write nothing.
 
@@ -254,6 +264,14 @@ def publish_description(
         (a draft that recorded none) is ``R_DRAFT_BASE_MISSING``.
     confirmations
         The user's recorded re-confirmations of changed IDs; each must be needed.
+    removals
+        The user's confirmations of the removed units (each with its date). Every unit the
+        candidate drops must be named by one; each is written as a ``removed`` record, with the
+        records of the condition, repeat and technical repeat that no longer exist.
+    recorded_on
+        The date (``YYYY-MM-DD``) of the ``image_count_changed`` records: required when any
+        confirmed member's ``image_count`` changes, alone or with other changes; never
+        defaulted and never read from the clock.
 
     Returns
     -------
@@ -271,12 +289,18 @@ def publish_description(
     DescriptionError
         When the candidate's canonical bytes, or the published file, is not a valid description.
     ValueError
-        When a re-confirmation is not needed by the candidate.
+        When a re-confirmation is not needed by the candidate, a dropped unit has no removal, a
+        removal names a unit the candidate keeps, or ``recorded_on`` is missing when needed.
     """
     folder = Path(output_folder)
     with output_lock(folder):
         return publish_locked(
-            folder, candidate, base_sha256=base_sha256, confirmations=confirmations
+            folder,
+            candidate,
+            base_sha256=base_sha256,
+            confirmations=confirmations,
+            removals=removals,
+            recorded_on=recorded_on,
         )
 
 
@@ -286,6 +310,8 @@ def publish_locked(
     *,
     base_sha256: str | None,
     confirmations: tuple[Reconfirmation, ...] = (),
+    removals: tuple[Removal, ...] = (),
+    recorded_on: str | None = None,
 ) -> Publication:
     """Publish like :func:`publish_description` for a caller that already holds the lock.
 
@@ -304,9 +330,14 @@ def publish_locked(
     base = check_base(base_sha256, published)
     data = canonical_bytes(candidate)
     description_from_bytes(data)  # the exact bytes to be written pass the strict reader
-    _compare(published, candidate, confirmations)
+    old = _compare(published, candidate, confirmations)
     new_sha = sha256_hex(data)
     records = tuple(_record(c, base, new_sha) for c in sorted(confirmations, key=_order))
+    if old is not None:
+        records += removal_records(old, candidate, removals, confirmations)
+        records += image_count_records(old, candidate, recorded_on, confirmations)
+    elif removals:
+        raise ValueError("a removal needs a published file to remove from")
     write_text_atomic(path, data.decode("utf-8"))
     try:
         _append_records(folder, records)

src/morphobricks_preprocessing/run/description_records.py

@@ -0,0 +1,221 @@
+"""The ``removed`` and ``image_count_changed`` records of a publication (experiment-v1 4.6, 10).
+
+Both are derived by comparing the candidate with the published file it replaces, plus the user's
+facts that no comparison can supply: the date of each removal confirmation
+(:class:`..description_removal.Removal`) and the date the image-count change is recorded
+(``recorded_on``). Nothing here reads the clock, and a date that is needed and not given is an
+error, never a default. Pure: no file is touched.
+
+``removed`` (owner decision RR2). A published unit is removed when neither its source folders nor
+its IDs (after the confirmed ID corrections) are in the candidate. Every removed unit is written
+as a ``unit`` record naming all six keys. An entity whose ID the removal leaves held nowhere
+(a condition, repeat or technical repeat) is written as its own record, before the unit records:
+all entity records in the order condition, repeat, technical repeat (each by ID), then the unit
+records by (condition ID, repeat ID, technical repeat ID). Labels are the published file's.
+
+``image_count_changed`` (owner decisions RR1, RR3). Written for every member of a confirmed repeat
+whose ``image_count`` differs, whether or not the publication changes anything else (a removal, a
+new member, an ID correction, a reopening). Members are matched through the confirmed ID
+corrections; a member the candidate no longer holds has no new count and is covered by its
+``removed`` records. Lines carry the published IDs and labels, in (condition ID, repeat ID) order.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterable
+from datetime import date
+
+from ..description import Description, Member
+from ..description_reconcile import PublishedUnit, UnitFolders, published_units
+from ..description_removal import Removal
+from ..description_transition import DATE, Reconfirmation
+
+_NONE = dict.fromkeys(
+    (
+        "condition_id",
+        "condition_source_folder",
+        "repeat_id",
+        "source_repeat",
+        "technical_repeat_id",
+        "technical_repeat_source_folder",
+    )
+)
+
+
+def _renames(confirmations: Iterable[Reconfirmation]) -> dict[str, dict[str, str]]:
+    """Return the confirmed ID corrections (old to new) of each kind."""
+    found: dict[str, dict[str, str]] = {"conditions": {}, "repeats": {}, "technical_repeats": {}}
+    for item in confirmations:
+        if item.new_id is not None and item.kind in found:
+            found[item.kind][item.old_id] = item.new_id
+    return found
+
+
+def _paths(description: Description) -> set[tuple[str, str, str]]:
+    return {
+        (repeat_id, condition_id, tid)
+        for repeat_id, repeat in description.repeats.items()
+        for condition_id, member in repeat.members.items()
+        for tid in member.technical_repeats
+    }
+
+
+def _removed_units(
+    old: Description, new: Description, confirmations: Iterable[Reconfirmation]
+) -> dict[UnitFolders, PublishedUnit]:
+    """Return the published units the candidate no longer holds, by source folder labels."""
+    fix = _renames(confirmations)
+    kept_keys, kept_paths = set(published_units(new)), _paths(new)
+    gone = {}
+    for key, unit in published_units(old).items():
+        path = (
+            fix["repeats"].get(unit.repeat_id, unit.repeat_id),
+            fix["conditions"].get(unit.condition_id, unit.condition_id),
+            fix["technical_repeats"].get(unit.technical_id, unit.technical_id),
+        )
+        if key not in kept_keys and path not in kept_paths:
+            gone[key] = unit
+    return gone
+
+
+def removal_records(
+    old: Description,
+    new: Description,
+    removals: tuple[Removal, ...],
+    confirmations: tuple[Reconfirmation, ...] = (),
+) -> tuple[dict, ...]:
+    """Return the ``removed`` records of the units and entities the candidate drops.
+
+    Each takes the date of the first removal answer that names one of its units.
+
+    Raises
+    ------
+    ValueError
+        When a unit is dropped that no removal confirms, or a removal names a unit the
+        candidate keeps or the published file does not have.
+    """
+    gone = _removed_units(old, new, confirmations)
+    asked = {key for removal in removals for key in removal.units}
+    loose = sorted(set(gone) - asked)
+    if loose:
+        names = ", ".join(f"{k.condition}/{k.source_repeat}/{k.technical}" for k in loose)
+        raise ValueError(f"a unit is removed and no removal confirms it: {names}")
+    extra = sorted(asked - set(gone))
+    if extra:
+        names = ", ".join(f"{k.condition}/{k.source_repeat}/{k.technical}" for k in extra)
+        raise ValueError(f"a removal names a unit the candidate does not remove: {names}")
+
+    def day(keys: Iterable[UnitFolders]) -> str:
+        held = set(keys)
+        return next(r.confirmed_on for r in removals if held & set(r.units))
+
+    fix = _renames(confirmations)
+    kept_tids = {tid for _, _, tid in _paths(new)}
+    entities: dict[tuple[int, str], dict] = {}
+    conditions = {u.condition_id: u for u in gone.values()}
+    repeats = {u.repeat_id for u in gone.values()}
+    tids = {u.technical_id: u for u in gone.values()}
+    for cid in conditions:
+        if fix["conditions"].get(cid, cid) not in new.conditions:
+            keys = [k for k, u in gone.items() if u.condition_id == cid]
+            entities[(0, cid)] = _row(
+                "condition",
+                day(keys),
+                condition_id=cid,
+                condition_source_folder=old.conditions[cid].source_folder,
+            )
+    for rid in repeats:
+        if fix["repeats"].get(rid, rid) not in new.repeats:
+            keys = [k for k, u in gone.items() if u.repeat_id == rid]
+            entities[(1, rid)] = _row("repeat", day(keys), repeat_id=rid)
+    for tid, unit in tids.items():
+        if fix["technical_repeats"].get(tid, tid) not in kept_tids:
+            keys = [k for k, u in gone.items() if u.technical_id == tid]
+            entities[(2, tid)] = _row(
+                "technical_repeat",
+                day(keys),
+                technical_repeat_id=tid,
+                technical_repeat_source_folder=unit.unit.source_folder,
+            )
+    rows = [entities[key] for key in sorted(entities)]
+    for key, unit in sorted(
+        gone.items(), key=lambda kv: (kv[1].condition_id, kv[1].repeat_id, kv[1].technical_id)
+    ):
+        rows.append(
+            _row(
+                "unit",
+                day([key]),
+                condition_id=unit.condition_id,
+                condition_source_folder=key.condition,
+                repeat_id=unit.repeat_id,
+                source_repeat=key.source_repeat,
+                technical_repeat_id=unit.technical_id,
+                technical_repeat_source_folder=key.technical,
+            )
+        )
+    return tuple(rows)
+
+
+def _row(entity: str, confirmed_on: str, **named: str) -> dict:
+    """Return a ``removed`` line: every key present, ``null`` where the entity names none."""
+    return {"type": "removed", "entity": entity, **_NONE, **named, "confirmed_on": confirmed_on}
+
+
+def image_count_records(
+    old: Description,
+    new: Description,
+    recorded_on: str | None,
+    confirmations: Iterable[Reconfirmation] = (),
+) -> tuple[dict, ...]:
+    """Return the ``image_count_changed`` record of every confirmed member whose count changed.
+
+    Raises
+    ------
+    ValueError
+        When a record is due and ``recorded_on`` is not a ``YYYY-MM-DD`` date (no default).
+    """
+    fix = _renames(confirmations)
+    changed = []
+    for rid, repeat in old.repeats.items():
+        if not repeat.confirmed:
+            continue
+        kept = new.repeats.get(fix["repeats"].get(rid, rid))
+        for cid, member in repeat.members.items():
+            held = kept.members.get(fix["conditions"].get(cid, cid)) if kept else None
+            if held is not None and held.image_count != member.image_count:
+                changed.append((cid, rid, member, held.image_count))
+    if not changed:
+        return ()
+    try:
+        if recorded_on is None or not DATE.fullmatch(recorded_on):
+            raise ValueError
+        date.fromisoformat(recorded_on)
+    except ValueError:
+        raise ValueError(
+            f"an image count changed: recorded_on is a real YYYY-MM-DD date, not {recorded_on!r}"
+        ) from None
+    return tuple(_count_row(old, *item, recorded_on) for item in sorted(changed, key=_by_ids))
+
+
+def _by_ids(item: tuple) -> tuple[str, str]:
+    return item[0], item[1]
+
+
+def _count_row(
+    old: Description, cid: str, rid: str, member: Member, now: int, recorded_on: str
+) -> dict:
+    """Return the ``image_count_changed`` line of one member (4.6)."""
+    return {
+        "type": "image_count_changed",
+        "condition_id": cid,
+        "condition_source_folder": old.conditions[cid].source_folder,
+        "repeat_id": rid,
+        "source_repeat": member.source_repeat,
+        "technical_repeats": [
+            {"id": tid, "source_folder": unit.source_folder}
+            for tid, unit in sorted(member.technical_repeats.items())
+        ],
+        "old_image_count": member.image_count,
+        "new_image_count": now,
+        "recorded_on": recorded_on,
+    }