From 4eca8ac8adcf21ab2b55d700b6d6bd29ef79f888 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 31 Oct 2026 14:57:37 +0000 Subject: [PATCH] Keep an unreadable experiment_design.csv untouched in step 1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When step 1 could not read a saved experiment_design.csv (for example a confirmed_on that a spreadsheet program rewrote as 26/10/2026, refused since #59), it fell back to the default design and wrote it at once, before any answer, so the confirmed answers in the file were lost. Owner decision of 31 October 2026 (plan §12 question 42): the file is left unchanged and refused. Step 1 shows the refusal in a red box, naming the line and column, starts the questions unanswered and writes nothing until the user corrects the file and confirms the labels again, or answers a question, which replaces the file. Step 3 still runs only once every group is confirmed. The command line already refused such a file without writing it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WX6rdMV1r5jUPUR94kxV5a --- docs/contract.md | 2 +- docs/plan.md | 10 ++++ docs/user_guide.md | 7 +++ src/morphobricks_preprocessing/notebook_ui.py | 51 ++++++++++++---- tests/test_notebook_ui.py | 59 ++++++++++++++++++- 5 files changed, 116 insertions(+), 13 deletions(-) diff --git a/docs/contract.md b/docs/contract.md index 9fe9969a..b8393000 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -259,7 +259,7 @@ folder. | Controls | None, one or several conditions; every row of a condition has the same value. Recorded only. | | Reruns | The file in the output folder is read and brought up to date with the folders found: folders in it keep their group, control mark and confirmation (added or removed images within a folder keep it too, with the counts recorded at confirmation); a new folder joins the group named after it and inherits its condition's control mark, and that group is asked again; folders no longer found are dropped without asking again. | | Reading | `read_experiment_design` reads labels as text; `control` and `confirmed` accept `true`/`false` in any case, also `yes`/`no` and `1`/`0`, and empty as false; a byte-order mark is ignored. The user's edits are checked (owner decision of 31 October 2026): `repeat_group` follows the identity rules of §16 (D-e: not empty, `NA`, `nan`, `none`, `null`, `n/a` or `` in any case, no leading or trailing whitespace, and spelled one way: two groups differing only in case are refused) and `confirmed_on` is empty or a date `YYYY-MM-DD`; a bad edit raises `DesignFormatError` naming the line and column. `design_issues` gives the error `invalid_experiment_design` for such values in a design built in code, so a run stops before writing, and `write_experiment_design` refuses to write them. A missing column, an invalid value, a folder listed twice, rows of one condition disagreeing on `control` or another `schema_version` raise `DesignFormatError`. `ExperimentDesign.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping the normalization brick takes. This lenient reader is for the user's own edits; other bricks read the file with the strict `read_confirmed_design` (§16). | -| Notebook | Step 1 asks once the labels are confirmed and writes the file after every answer (unconfirmed groups with `confirmed=false`). A design confirmed for the same folders is loaded with the questions folded away behind **Change the experiment design**. Step 3 runs only once every group is confirmed. A brick embedding step 1 with `folder_step(show_design=False)` (plan §12 question 31) gets no design: nothing is asked or written, and step 3 runs without one, as `run_experiment(design=None)`. | +| Notebook | Step 1 asks once the labels are confirmed and writes the file after every answer (unconfirmed groups with `confirmed=false`). A design confirmed for the same folders is loaded with the questions folded away behind **Change the experiment design**. A saved file that cannot be read (`DesignFormatError`, e.g. a `confirmed_on` a spreadsheet program rewrote as `26/10/2026`) is refused and left unchanged (owner decision of 31 October 2026, plan §12 question 42): step 1 shows the refusal, which names the line and column, starts the questions unanswered and writes nothing until the user corrects the file and confirms the labels again, or answers a question, which replaces the file. Nothing falls back to a default design silently, and step 3 does not run until every group is confirmed again. Step 3 runs only once every group is confirmed. A brick embedding step 1 with `folder_step(show_design=False)` (plan §12 question 31) gets no design: nothing is asked or written, and step 3 runs without one, as `run_experiment(design=None)`. | | Command line | `run` reads the file from the output folder, or `--design FILE`. Without a fully confirmed design it writes the design brought up to date with the folders (a template with `confirmed=false` for the groups still to confirm, keeping the answers already given) to the output folder, explains the columns, and stops with status 1. Confirmed rows without a date get the date of the run. | | Run | `run_experiment(..., design=...)` brings the design up to date with the folders and stops with `design_not_confirmed` or `repeat_group_conflict` if it is not fully confirmed. Otherwise the file is written with the table and published before the manifest, which records its SHA-256 (§9). Without a design the run is unchanged: no file is written, an existing one is left as it is, and `experiment_design_sha256` is null. `plane_histograms.csv` does not depend on the design. | diff --git a/docs/plan.md b/docs/plan.md index f4fe9579..c4978cac 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -991,3 +991,13 @@ Raised during review; to be decided before the milestones that need them: groups differing only in case, or a `confirmed_on` that is not `YYYY-MM-DD`. The normalization brick stays at v0.3.2. The version in `pyproject.toml` is 0.3.4 before tagging. Every brick pins this tag (D79).* +42. What does step 1 do with a saved `experiment_design.csv` it cannot read (PR #59 + follow-up)? Until now it fell back to the default design and wrote it at once, before + any answer, so the confirmed answers in a file a spreadsheet program had re-saved (a + `confirmed_on` of `26/10/2026`, refused since question 40) were lost. + *Decided (31 October 2026): the file is kept untouched and refused. Step 1 shows the + refusal, naming the line and column, starts the questions unanswered and writes nothing + until the user corrects the file (and confirms the labels again) or answers a question + again, which replaces the file. No silent fallback to a default design; step 3 runs + only once every group is confirmed again. The command line already refused such a file + without writing it.* diff --git a/docs/user_guide.md b/docs/user_guide.md index ef992f7d..7933b4b8 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -209,6 +209,11 @@ results or comparing another grouping. every answer. Next time, if the folders are the same, they are loaded and the questions are folded away; **Change the experiment design** shows them again. A new biological-repeat folder asks again only for the group it joins. + - If the saved **experiment_design.csv** cannot be read (for example a spreadsheet + program rewrote a date such as 2026-10-26 as 26/10/2026), a red box shows why, + naming the line and column, and the file is left as it is. Correct the file and + undo and confirm the labels again to read it, or answer the questions again: the + first answer replaces the file. - **Dataset consistency** shows how many images each condition (columns) and biological repeat (rows; the repeat groups once paired) has, with the number of technical-repeat folders. A cell is amber when it has fewer than half or more than twice the median number of images, or @@ -622,6 +627,8 @@ nor writes one. To record it as the notebook does: from morphobricks_preprocessing import load_design design, changes, problem = load_design(result.output_folder, check.dataset.repeats) +if problem: # an existing file that cannot be read: correct it, do not overwrite it + raise SystemExit(problem) design = design.with_group(("treated", "rep2_day2"), "rep2") # pair a repeat for group in design.groups: design = design.with_confirmed(group, True, "2026-10-26") diff --git a/src/morphobricks_preprocessing/notebook_ui.py b/src/morphobricks_preprocessing/notebook_ui.py index 8c59b1c2..f3de3ab7 100644 --- a/src/morphobricks_preprocessing/notebook_ui.py +++ b/src/morphobricks_preprocessing/notebook_ui.py @@ -221,6 +221,8 @@ class PreprocessingNotebook: def __init__(self, experiment_folder: str = "", output_folder: str = ""): self.preflight: Preflight | None = None self.design: ExperimentDesign | None = None + # True while the saved design cannot be read and has not been replaced by an answer. + self._design_unreadable = False self.dataset = DatasetReport() self.mapping: AxisMapping | None = None self.confirmed_mapping: AxisMapping | None = None @@ -763,7 +765,12 @@ def _design_folder(self) -> Path | None: return None if check_output_folder(folder, output) else output def _open_design(self) -> None: - """Load the saved design, or start a new one, once the labels are confirmed.""" + """Load the saved design, or start a new one, once the labels are confirmed. + + A saved file that cannot be read is left as it is (owner decision of 31 October + 2026): nothing is written until the user corrects it or answers a question again, + and the questions start unanswered, so step 3 cannot run on a guessed design. + """ repeats = self.preflight.dataset.repeats output = self._design_folder() changes, problem, saved = DesignChanges(), None, False @@ -773,6 +780,7 @@ def _open_design(self) -> None: design, changes, problem = load_design(output, repeats) saved = problem is None and (output / DESIGN_FILENAME).exists() self.design = design + self._design_unreadable = problem is not None self._build_design_editor() # A design confirmed earlier for the same folders is not asked again. reopen = not (saved and design.confirmed and not changes.any) @@ -780,7 +788,10 @@ def _open_design(self) -> None: ui.set_visible(self.design_change, not reopen) self._design_note = self._design_load_note(design, changes, problem, saved) ui.set_visible(self.design_card, True) - self._save_design() + if self._design_unreadable: + self._design_saved = False + else: + self._save_design() self._design_changed() def _design_load_note( @@ -788,9 +799,16 @@ def _design_load_note( ) -> str: if problem: return ui.status( - "warn", - f"The saved {DESIGN_FILENAME} could not be read, so the questions are asked again.", - [problem, "Answering replaces the file."], + "error", + f"The saved {DESIGN_FILENAME} cannot be read. It was left unchanged.", + [ + problem, + "Correct the file (a spreadsheet program may have rewritten a date such " + "as 2026-10-26 as 26/10/2026; dates are written YYYY-MM-DD), then undo " + "and confirm the labels again to read it.", + "Or answer the questions below again: your first answer replaces the " + "file with the answers shown here.", + ], ) if saved and changes.any: lines = [] @@ -928,6 +946,11 @@ def _save_design(self) -> None: logging.getLogger(__name__).warning("Cannot save the design", exc_info=True) return self._design_saved = True + if self._design_unreadable: + self._design_unreadable = False + self._design_note = ui.status( + "info", f"Your answers replaced the unreadable {DESIGN_FILENAME}." + ) def _design_changed(self) -> None: """Update the balance, the messages, the status and the later steps after an answer.""" @@ -949,12 +972,18 @@ def _refresh_design_status(self) -> None: return note = getattr(self, "_design_note", "") saved = getattr(self, "_design_saved", False) - where = ( - f"Saved in {html.escape(str(self._design_folder() / DESIGN_FILENAME))}." - if saved - else "Not saved: the output folder is not set or cannot be written; the answers " - "are kept while the notebook is open and saved with the run." - ) + if saved: + where = f"Saved in {html.escape(str(self._design_folder() / DESIGN_FILENAME))}." + elif self._design_unreadable: + where = ( + f"Not saved: the unreadable {DESIGN_FILENAME} is kept as it is until you " + "correct it or answer a question." + ) + else: + where = ( + "Not saved: the output folder is not set or cannot be written; the answers " + "are kept while the notebook is open and saved with the run." + ) controls = ", ".join(html.escape(c) for c in design.controls) or "none" if design.confirmed: dates = sorted({design.confirmed_on(g) for g in design.groups} - {""}) diff --git a/tests/test_notebook_ui.py b/tests/test_notebook_ui.py index d7d61ffe..c1acff32 100644 --- a/tests/test_notebook_ui.py +++ b/tests/test_notebook_ui.py @@ -387,11 +387,68 @@ def test_an_unreadable_design_is_asked_again(tmp_path): (tmp_path / "out").mkdir() (tmp_path / "out" / "experiment_design.csv").write_bytes(b"not a design") nb = labelled(root, tmp_path) - assert "could not be read" in nb.design_status.value + assert "cannot be read" in nb.design_status.value + assert (tmp_path / "out" / "experiment_design.csv").read_bytes() == b"not a design" confirm_repeats(nb) assert read_experiment_design(tmp_path / "out" / "experiment_design.csv").confirmed +def spreadsheet_saved(path): + """Rewrite the ISO dates of a saved design as a spreadsheet program may (26/10/2026).""" + text = path.read_text(encoding="utf-8") + for on in {row.confirmed_on for row in read_experiment_design(path).rows} - {""}: + year, month, day = on.split("-") + text = text.replace(on, f"{day}/{month}/{year}") + path.write_text(text, encoding="utf-8") + return path.read_bytes() + + +def test_an_unreadable_design_is_left_untouched_until_answered(tmp_path): + # PR #59 follow-up, owner decision of 31 October 2026: the confirmed answers in a file + # that cannot be read are never replaced by the default design before the user answers. + root = design_tree(tmp_path / "exp", ["ctrl/rep1", "ctrl/rep2", "drug/rep1", "drug/rep2"]) + nb = labelled(root, tmp_path) + nb.control_checks["ctrl"].value = True + confirm_repeats(nb) + path = tmp_path / "out" / "experiment_design.csv" + answers = path.read_bytes() + unreadable = spreadsheet_saved(path) + + again = labelled(root, tmp_path) + assert path.read_bytes() == unreadable + status = again.design_status.value + assert "cannot be read" in status and "left unchanged" in status + assert "line 2, column confirmed_on" in status + assert "mb-error" in status + assert not again._design_confirmed() + assert ui.is_visible(again.design_editor) + # Undoing and confirming the labels again reads the file again: still nothing written. + again.labels_confirmed.button.click() + again.labels_confirmed.button.click() + assert path.read_bytes() == unreadable + + # The corrected file is read with the answers it holds. + path.write_bytes(answers) + fixed = labelled(root, tmp_path) + assert path.read_bytes() == answers + assert fixed.design.controls == ("ctrl",) + assert fixed.design.confirmed and not ui.is_visible(fixed.design_editor) + + +def test_answering_again_replaces_an_unreadable_design(tmp_path): + root = design_tree(tmp_path / "exp", ["ctrl/rep1", "drug/rep1"]) + nb = labelled(root, tmp_path) + confirm_repeats(nb) + path = tmp_path / "out" / "experiment_design.csv" + spreadsheet_saved(path) + again = labelled(root, tmp_path) + again.control_checks["drug"].value = True + saved = read_experiment_design(path) + assert saved.controls == ("drug",) and not saved.confirmed + assert "replaced the unreadable" in again.design_status.value + assert "mb-error" not in again.design_status.value + + def test_undoing_the_labels_closes_the_design(tmp_path): root = design_tree(tmp_path / "exp", ["ctrl/rep1", "drug/rep1"]) nb = labelled(root, tmp_path)