diff --git a/docs/contract.md b/docs/contract.md index 9fe9969a..b8393000 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -259,7 +259,7 @@ folder. | Controls | None, one or several conditions; every row of a condition has the same value. Recorded only. | | Reruns | The file in the output folder is read and brought up to date with the folders found: folders in it keep their group, control mark and confirmation (added or removed images within a folder keep it too, with the counts recorded at confirmation); a new folder joins the group named after it and inherits its condition's control mark, and that group is asked again; folders no longer found are dropped without asking again. | | Reading | `read_experiment_design` reads labels as text; `control` and `confirmed` accept `true`/`false` in any case, also `yes`/`no` and `1`/`0`, and empty as false; a byte-order mark is ignored. The user's edits are checked (owner decision of 31 October 2026): `repeat_group` follows the identity rules of §16 (D-e: not empty, `NA`, `nan`, `none`, `null`, `n/a` or `` in any case, no leading or trailing whitespace, and spelled one way: two groups differing only in case are refused) and `confirmed_on` is empty or a date `YYYY-MM-DD`; a bad edit raises `DesignFormatError` naming the line and column. `design_issues` gives the error `invalid_experiment_design` for such values in a design built in code, so a run stops before writing, and `write_experiment_design` refuses to write them. A missing column, an invalid value, a folder listed twice, rows of one condition disagreeing on `control` or another `schema_version` raise `DesignFormatError`. `ExperimentDesign.repeat_groups()` gives the `{(condition, biological_repeat): repeat_group}` mapping the normalization brick takes. This lenient reader is for the user's own edits; other bricks read the file with the strict `read_confirmed_design` (§16). | -| Notebook | Step 1 asks once the labels are confirmed and writes the file after every answer (unconfirmed groups with `confirmed=false`). A design confirmed for the same folders is loaded with the questions folded away behind **Change the experiment design**. Step 3 runs only once every group is confirmed. A brick embedding step 1 with `folder_step(show_design=False)` (plan §12 question 31) gets no design: nothing is asked or written, and step 3 runs without one, as `run_experiment(design=None)`. | +| Notebook | Step 1 asks once the labels are confirmed and writes the file after every answer (unconfirmed groups with `confirmed=false`). A design confirmed for the same folders is loaded with the questions folded away behind **Change the experiment design**. A saved file that cannot be read (`DesignFormatError`, e.g. a `confirmed_on` a spreadsheet program rewrote as `26/10/2026`) is refused and left unchanged (owner decision of 31 October 2026, plan §12 question 42): step 1 shows the refusal, which names the line and column, starts the questions unanswered and writes nothing until the user corrects the file and confirms the labels again, or answers a question, which replaces the file. Nothing falls back to a default design silently, and step 3 does not run until every group is confirmed again. Step 3 runs only once every group is confirmed. A brick embedding step 1 with `folder_step(show_design=False)` (plan §12 question 31) gets no design: nothing is asked or written, and step 3 runs without one, as `run_experiment(design=None)`. | | Command line | `run` reads the file from the output folder, or `--design FILE`. Without a fully confirmed design it writes the design brought up to date with the folders (a template with `confirmed=false` for the groups still to confirm, keeping the answers already given) to the output folder, explains the columns, and stops with status 1. Confirmed rows without a date get the date of the run. | | Run | `run_experiment(..., design=...)` brings the design up to date with the folders and stops with `design_not_confirmed` or `repeat_group_conflict` if it is not fully confirmed. Otherwise the file is written with the table and published before the manifest, which records its SHA-256 (§9). Without a design the run is unchanged: no file is written, an existing one is left as it is, and `experiment_design_sha256` is null. `plane_histograms.csv` does not depend on the design. | diff --git a/docs/plan.md b/docs/plan.md index f4fe9579..c4978cac 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -991,3 +991,13 @@ Raised during review; to be decided before the milestones that need them: groups differing only in case, or a `confirmed_on` that is not `YYYY-MM-DD`. The normalization brick stays at v0.3.2. The version in `pyproject.toml` is 0.3.4 before tagging. Every brick pins this tag (D79).* +42. What does step 1 do with a saved `experiment_design.csv` it cannot read (PR #59 + follow-up)? Until now it fell back to the default design and wrote it at once, before + any answer, so the confirmed answers in a file a spreadsheet program had re-saved (a + `confirmed_on` of `26/10/2026`, refused since question 40) were lost. + *Decided (31 October 2026): the file is kept untouched and refused. Step 1 shows the + refusal, naming the line and column, starts the questions unanswered and writes nothing + until the user corrects the file (and confirms the labels again) or answers a question + again, which replaces the file. No silent fallback to a default design; step 3 runs + only once every group is confirmed again. The command line already refused such a file + without writing it.* diff --git a/docs/user_guide.md b/docs/user_guide.md index ef992f7d..7933b4b8 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -209,6 +209,11 @@ results or comparing another grouping. every answer. Next time, if the folders are the same, they are loaded and the questions are folded away; **Change the experiment design** shows them again. A new biological-repeat folder asks again only for the group it joins. + - If the saved **experiment_design.csv** cannot be read (for example a spreadsheet + program rewrote a date such as 2026-10-26 as 26/10/2026), a red box shows why, + naming the line and column, and the file is left as it is. Correct the file and + undo and confirm the labels again to read it, or answer the questions again: the + first answer replaces the file. - **Dataset consistency** shows how many images each condition (columns) and biological repeat (rows; the repeat groups once paired) has, with the number of technical-repeat folders. A cell is amber when it has fewer than half or more than twice the median number of images, or @@ -622,6 +627,8 @@ nor writes one. To record it as the notebook does: from morphobricks_preprocessing import load_design design, changes, problem = load_design(result.output_folder, check.dataset.repeats) +if problem: # an existing file that cannot be read: correct it, do not overwrite it + raise SystemExit(problem) design = design.with_group(("treated", "rep2_day2"), "rep2") # pair a repeat for group in design.groups: design = design.with_confirmed(group, True, "2026-10-26") diff --git a/src/morphobricks_preprocessing/notebook_ui.py b/src/morphobricks_preprocessing/notebook_ui.py index 8c59b1c2..f3de3ab7 100644 --- a/src/morphobricks_preprocessing/notebook_ui.py +++ b/src/morphobricks_preprocessing/notebook_ui.py @@ -221,6 +221,8 @@ class PreprocessingNotebook: def __init__(self, experiment_folder: str = "", output_folder: str = ""): self.preflight: Preflight | None = None self.design: ExperimentDesign | None = None + # True while the saved design cannot be read and has not been replaced by an answer. + self._design_unreadable = False self.dataset = DatasetReport() self.mapping: AxisMapping | None = None self.confirmed_mapping: AxisMapping | None = None @@ -763,7 +765,12 @@ def _design_folder(self) -> Path | None: return None if check_output_folder(folder, output) else output def _open_design(self) -> None: - """Load the saved design, or start a new one, once the labels are confirmed.""" + """Load the saved design, or start a new one, once the labels are confirmed. + + A saved file that cannot be read is left as it is (owner decision of 31 October + 2026): nothing is written until the user corrects it or answers a question again, + and the questions start unanswered, so step 3 cannot run on a guessed design. + """ repeats = self.preflight.dataset.repeats output = self._design_folder() changes, problem, saved = DesignChanges(), None, False @@ -773,6 +780,7 @@ def _open_design(self) -> None: design, changes, problem = load_design(output, repeats) saved = problem is None and (output / DESIGN_FILENAME).exists() self.design = design + self._design_unreadable = problem is not None self._build_design_editor() # A design confirmed earlier for the same folders is not asked again. reopen = not (saved and design.confirmed and not changes.any) @@ -780,7 +788,10 @@ def _open_design(self) -> None: ui.set_visible(self.design_change, not reopen) self._design_note = self._design_load_note(design, changes, problem, saved) ui.set_visible(self.design_card, True) - self._save_design() + if self._design_unreadable: + self._design_saved = False + else: + self._save_design() self._design_changed() def _design_load_note( @@ -788,9 +799,16 @@ def _design_load_note( ) -> str: if problem: return ui.status( - "warn", - f"The saved {DESIGN_FILENAME} could not be read, so the questions are asked again.", - [problem, "Answering replaces the file."], + "error", + f"The saved {DESIGN_FILENAME} cannot be read. It was left unchanged.", + [ + problem, + "Correct the file (a spreadsheet program may have rewritten a date such " + "as 2026-10-26 as 26/10/2026; dates are written YYYY-MM-DD), then undo " + "and confirm the labels again to read it.", + "Or answer the questions below again: your first answer replaces the " + "file with the answers shown here.", + ], ) if saved and changes.any: lines = [] @@ -928,6 +946,11 @@ def _save_design(self) -> None: logging.getLogger(__name__).warning("Cannot save the design", exc_info=True) return self._design_saved = True + if self._design_unreadable: + self._design_unreadable = False + self._design_note = ui.status( + "info", f"Your answers replaced the unreadable {DESIGN_FILENAME}." + ) def _design_changed(self) -> None: """Update the balance, the messages, the status and the later steps after an answer.""" @@ -949,12 +972,18 @@ def _refresh_design_status(self) -> None: return note = getattr(self, "_design_note", "") saved = getattr(self, "_design_saved", False) - where = ( - f"Saved in {html.escape(str(self._design_folder() / DESIGN_FILENAME))}." - if saved - else "Not saved: the output folder is not set or cannot be written; the answers " - "are kept while the notebook is open and saved with the run." - ) + if saved: + where = f"Saved in {html.escape(str(self._design_folder() / DESIGN_FILENAME))}." + elif self._design_unreadable: + where = ( + f"Not saved: the unreadable {DESIGN_FILENAME} is kept as it is until you " + "correct it or answer a question." + ) + else: + where = ( + "Not saved: the output folder is not set or cannot be written; the answers " + "are kept while the notebook is open and saved with the run." + ) controls = ", ".join(html.escape(c) for c in design.controls) or "none" if design.confirmed: dates = sorted({design.confirmed_on(g) for g in design.groups} - {""}) diff --git a/tests/test_notebook_ui.py b/tests/test_notebook_ui.py index d7d61ffe..c1acff32 100644 --- a/tests/test_notebook_ui.py +++ b/tests/test_notebook_ui.py @@ -387,11 +387,68 @@ def test_an_unreadable_design_is_asked_again(tmp_path): (tmp_path / "out").mkdir() (tmp_path / "out" / "experiment_design.csv").write_bytes(b"not a design") nb = labelled(root, tmp_path) - assert "could not be read" in nb.design_status.value + assert "cannot be read" in nb.design_status.value + assert (tmp_path / "out" / "experiment_design.csv").read_bytes() == b"not a design" confirm_repeats(nb) assert read_experiment_design(tmp_path / "out" / "experiment_design.csv").confirmed +def spreadsheet_saved(path): + """Rewrite the ISO dates of a saved design as a spreadsheet program may (26/10/2026).""" + text = path.read_text(encoding="utf-8") + for on in {row.confirmed_on for row in read_experiment_design(path).rows} - {""}: + year, month, day = on.split("-") + text = text.replace(on, f"{day}/{month}/{year}") + path.write_text(text, encoding="utf-8") + return path.read_bytes() + + +def test_an_unreadable_design_is_left_untouched_until_answered(tmp_path): + # PR #59 follow-up, owner decision of 31 October 2026: the confirmed answers in a file + # that cannot be read are never replaced by the default design before the user answers. + root = design_tree(tmp_path / "exp", ["ctrl/rep1", "ctrl/rep2", "drug/rep1", "drug/rep2"]) + nb = labelled(root, tmp_path) + nb.control_checks["ctrl"].value = True + confirm_repeats(nb) + path = tmp_path / "out" / "experiment_design.csv" + answers = path.read_bytes() + unreadable = spreadsheet_saved(path) + + again = labelled(root, tmp_path) + assert path.read_bytes() == unreadable + status = again.design_status.value + assert "cannot be read" in status and "left unchanged" in status + assert "line 2, column confirmed_on" in status + assert "mb-error" in status + assert not again._design_confirmed() + assert ui.is_visible(again.design_editor) + # Undoing and confirming the labels again reads the file again: still nothing written. + again.labels_confirmed.button.click() + again.labels_confirmed.button.click() + assert path.read_bytes() == unreadable + + # The corrected file is read with the answers it holds. + path.write_bytes(answers) + fixed = labelled(root, tmp_path) + assert path.read_bytes() == answers + assert fixed.design.controls == ("ctrl",) + assert fixed.design.confirmed and not ui.is_visible(fixed.design_editor) + + +def test_answering_again_replaces_an_unreadable_design(tmp_path): + root = design_tree(tmp_path / "exp", ["ctrl/rep1", "drug/rep1"]) + nb = labelled(root, tmp_path) + confirm_repeats(nb) + path = tmp_path / "out" / "experiment_design.csv" + spreadsheet_saved(path) + again = labelled(root, tmp_path) + again.control_checks["drug"].value = True + saved = read_experiment_design(path) + assert saved.controls == ("drug",) and not saved.confirmed + assert "replaced the unreadable" in again.design_status.value + assert "mb-error" not in again.design_status.value + + def test_undoing_the_labels_closes_the_design(tmp_path): root = design_tree(tmp_path / "exp", ["ctrl/rep1", "drug/rep1"]) nb = labelled(root, tmp_path)