diff --git a/docs/contract.md b/docs/contract.md index dab8b9a4..1f0e6732 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -33,6 +33,7 @@ must have a supported role. One confirmed mapping per experiment. Plane coordina | Unsupported axes | Position-like axes (tile `R`, mosaic `M`, row `K`, column `J`) cannot have a role. If they hold more than one plane, the file is an error. | | Singleton axes | An axis of length 1 needs no role; its index is always 0. | | Missing axes | A file lacking axes the mapping has is read with those axes as length 1 only when the user confirmed this and only one arrangement fits; its coordinates for them are 0. | +| Notebook readiness | Confirming a valid singleton mapping resolves its missing-axis error for running; folder, exclusion and all other metadata errors still block processing. | | Series | Only the first image series, at full-resolution level 0. A file with more than one series is an error. | | Channel names, significant bits | Recorded in the run manifest when metadata provides them; not part of the CSV. Channel names that differ at the same channel index are a warning. | @@ -140,6 +141,17 @@ suggests an action. Technical details go to the diagnostic log. UTF-8 JSON next to `plane_histograms.csv`, written after the table and only when the run succeeded. +Both files are prepared in a temporary folder before either earlier output is replaced. +Earlier files are backed up before publication. A caught replacement failure restores the +previous state, including the absence of files on a first run. If restoration fails, the +backup folder is retained and its location is reported for manual recovery. Failures to +remove temporary files after a successful publication are logged. + +The two replacements are not jointly atomic: forced process termination between them can +leave a mismatched pair, detected by `csv_sha256` on the next run and rebuilt. Only one run +may write to a given output folder at a time. This write protocol does not change either +file's schema or statistical interpretation. + | Key | Meaning | |---|---| | `manifest_version`, `schema_version` | Manifest format (`1`) and CSV format (`1`). | diff --git a/docs/development.md b/docs/development.md index 55fe7892..568648d8 100644 --- a/docs/development.md +++ b/docs/development.md @@ -46,6 +46,10 @@ functions the tests use. - Floats are written at full precision (`repr`); files are UTF-8. - Outputs are written to a temporary file and replace the previous version only after a complete successful write. +- Experiment runs stage both the CSV and manifest before publishing either, back up + earlier outputs, and restore their previous state if publication raises. Preserve + backups if recovery fails. The CSV hash check still detects forced termination between + replacements; run only one writer per output folder. ## Tests diff --git a/docs/plan.md b/docs/plan.md index 0635573d..8929109f 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -153,8 +153,15 @@ new and changed files without reading unchanged pixels. Provide a force-rebuild On rerun: add rows for new planes; replace rows for changed files or changed axis/bin settings; remove rows for deleted or renamed files (a rename is a delete plus an add). Existing rows of unchanged files are never recomputed, because each row depends only on -its own plane. Write to a temporary file and replace the previous output only after a -complete successful run. +its own plane. Prepare the CSV and manifest in a temporary folder on the output filesystem +before replacing either output. Back up earlier files before publication; if replacing +either file fails, restore the earlier pair (or remove newly created files on a first run). +If restoration itself fails, keep the backups and report their location for manual recovery. +Temporary-file cleanup failures are logged without treating a completed run as failed. + +The two replacements are not a single filesystem transaction. If the process is forcibly +terminated between them, the CSV hash check on the next run detects the mismatch and +rebuilds the output. Runs writing to the same output folder must be performed one at a time. A file is unchanged when its size, modification time, labels and axis layout match the manifest and the table still holds all its rows. Every row is recomputed when there is no @@ -216,15 +223,21 @@ API, CLI). The notebook and CLI only call the API. mapping saved to the manifest. *Implemented: axis mapping, prefill and validation (`axes`), plane-by-plane reading with a memory limit and the preview plane (`planes`), and the axis review step of the notebook.* + *Review fix: confirmed singleton axes clear the corresponding readiness error while + folder, exclusion and other metadata errors continue to block the run.* - **Milestone 3 — Histogram engine:** per-plane histograms, extrema, non-finite and constant planes, pooling, normalization library. *Core implemented and tested.* - **Milestone 4 — CSV and reruns:** write/update the CSV and manifest, detect new/changed/deleted planes, force rebuild. *Implemented: run manifest (`manifest`), preflight and incremental runs with exclusions (`run`).* + *Review fix: stage both files, preserve earlier output on write/publication failures, + and keep backup files with a recovery message if restoration fails.* - **Milestone 5 — Integration and docs:** public API, notebook, CLI, user documentation. *Implemented: summaries (`report`), notebook widgets (`notebook_ui`, `notebooks/preprocessing.ipynb`), `check`/`run` command line (`cli`) and the [user guide](user_guide.md).* + *Step-4 feedback: show an animated spinner through table loading, statistics and plotting; + disable the calculation button while busy and restore it on success or failure.* - **Later:** OME-Zarr and chunked processing for planes above the memory limit. ## 11. Tests and acceptance criteria @@ -278,3 +291,16 @@ Raised during review; to be decided before the milestones that need them: removed.* 7. Folders deeper than the technical-repeat level: warning or error? *Decided (October 2026): error, so that a layer of images is never dropped silently.* +8. High-priority review fixes: how should confirmed singleton axes and failed output + publication behave? + *Decided (October 2026): notebook readiness uses the confirmed axis review together + with all remaining folder, exclusion and metadata issues. Both output files are staged + before publication, and a caught publication failure restores the previous state. + Backups are retained if restoration fails. Forced termination remains covered by the + existing CSV hash check on rerun. The CSV and manifest schemas stay at version 1; + their fields and interpretation are unchanged.* +9. Step-4 feedback while statistics are calculated? + *Decided (October 2026): show the existing animated spinner with the current stage, + change the button to "Calculating…" and disable duplicate calculations. Keep the busy + state until plotting finishes, and restore the button after success or failure. This + is a notebook interaction change; statistical definitions and file schemas are unchanged.* diff --git a/docs/user_guide.md b/docs/user_guide.md index d03835c6..6e10327d 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -107,6 +107,8 @@ confirmed. If you change the experiment folder, everything after it is reset. The confirmed roles are locked until you click **Change axis roles**. - If some images lack a dimension the others have (often a single channel), a yellow box lists them; press **Yes, they have a single plane there** if that is the case. + Once you confirm the axis roles, this resolves the missing-axis error for step 3. + Any other errors still need to be fixed before running. 3. **Compute the histograms.** - **Before you run** lists anything still to do. - **What will happen** names the output folder and files, and says whether an earlier @@ -128,6 +130,9 @@ confirmed. If you change the experiment folder, everything after it is reset. experiment, paste the path of its `plane_histograms.csv` or click **Browse…**. - Choose **Condition**, **Biological repeat**, **Technical repeat** or **Image**, and the percentiles to report, then click **Calculate statistics**. + - A spinning wheel shows that work is in progress while the table is read, statistics + are calculated and the plot is drawn. The button says **Calculating…** and stays + disabled until the calculation finishes or an error is shown; you can then try again. - The table lists pooled statistics per group: minimum, maximum, mean, standard deviation, median and percentiles. - The plot compares the groups' intensity distributions: curves further right are @@ -184,6 +189,15 @@ The default output folder is next to the experiment folder: modification time. Channel names and significant bits are recorded when the metadata provides them. Format in [contract.md](contract.md) §9. +The table and manifest are both prepared before the earlier files are replaced. If a +replacement fails, the earlier files are restored. This needs temporary disk space for +the new outputs and copies of the earlier pair. If restoration also fails (for example, +another program locks an output file), the error names a backup folder: close programs +using the output files and restore those backups before running again. + +Run only one writer per output folder. If the process is forcibly stopped during output +replacement, rerunning checks the table against the manifest and rebuilds a mismatched pair. + ## 7. Running again, and excluding images Run the notebook or `run` again after adding, changing or removing images: diff --git a/src/morphobricks_preprocessing/notebook_ui.py b/src/morphobricks_preprocessing/notebook_ui.py index 4c68811c..573c7408 100644 --- a/src/morphobricks_preprocessing/notebook_ui.py +++ b/src/morphobricks_preprocessing/notebook_ui.py @@ -914,8 +914,24 @@ def _recommended_changed(self) -> None: def _refresh_run_step(self) -> None: pre = self.preflight + folder_ok = pre is not None and pre.ok + if pre is not None and self.confirmed_mapping is not None: + # Axis confirmation can resolve a preflight missing-axes error, but + # folder, exclusion and other metadata errors must still block the run. + review = review_axes( + pre.metadata, + self.confirmed_mapping, + missing_as_singleton=self.missing_as_singleton.value, + ) + folder_ok = not has_errors( + ( + *pre.discovery.issues, + *pre.exclusion_issues, + *review.combined_issues(pre.metadata), + ) + ) items = [ - (pre is not None and pre.ok, "Check the experiment folder and fix any errors (step 1)"), + (folder_ok, "Check the experiment folder and fix any errors (step 1)"), (pre is not None and self.labels_confirmed.value, "Confirm the labels (step 1)"), (self.confirmed_mapping is not None, "Confirm the axis roles (step 2)"), ] @@ -995,7 +1011,7 @@ def run(self) -> RunResult | None: "The run stopped because of an unexpected problem.", [ f"{type(exc).__name__}: {exc}", - "Nothing was written. Open Technical details below for the full report.", + "Open Technical details below for the full report.", ], ) self._show_log() @@ -1166,6 +1182,8 @@ def _statistics_option_changed(self) -> None: def show_statistics(self) -> None: """Show pooled statistics and distributions per group from the histogram table.""" + if self.statistics_button.disabled: + return path = self._csv_path() if path is None or not path.is_file(): if path is None: @@ -1193,21 +1211,39 @@ def show_statistics(self) -> None: ["Use numbers from 0 to 100 separated by commas, for example 1, 99.8."], ) return + self._busy(self.statistics_button, "Calculating…") + self.stats_status.value = ui.working("Reading the histogram table…") + self.stats_tiles.value = self.statistics.value = "" + ui.set_visible(self.statistics_card, False) + ui.set_visible(self.distribution_card, False) + try: + self._calculate_statistics(path, percentiles) + except Exception as exc: + self.log.lines.append(traceback.format_exc()) + self.stats_status.value = ui.status( + "error", + "The statistics could not be calculated.", + [f"{type(exc).__name__}: {exc}", "Check the histogram table and try again."], + ) + ui.set_visible(self.statistics_card, False) + ui.set_visible(self.distribution_card, False) + finally: + self._idle(self.statistics_button, " Calculate statistics", "calculator") + self._show_log() + + def _calculate_statistics(self, path: Path, percentiles: Sequence[float]) -> None: + """Read, summarise and plot the table while reporting the current stage.""" try: records = read_plane_histograms(path) except (CsvFormatError, OSError) as exc: self.stats_status.value = ui.status("error", "The table cannot be read.", [str(exc)]) return + self.stats_status.value = ui.working("Calculating statistics for each group…") level = self.level.value summaries = group_summaries(records, level, percentiles) groups = group_records(records, level) images = {r.source_relative_path for r in records} first = records[0].histogram if records else None - self.stats_status.value = ui.status( - "ok", - f"Statistics of {plural(len(summaries), LEVEL_NAMES[level])} calculated.", - [f"From the table {path}"], - ) self.stats_tiles.value = ui.tiles( [ ("🧪", f"{len(summaries):,}", f"{LEVEL_NAMES[level]} groups"), @@ -1221,9 +1257,15 @@ def show_statistics(self) -> None: depth = len(summaries[0].labels) if summaries else 1 self.statistics.value = ui.table(rows, numeric_from=depth) ui.set_visible(self.statistics_card, True) + self.stats_status.value = ui.working("Drawing the intensity distributions…") png = distribution_png(groups) self.distribution.value = png or b"" ui.set_visible(self.distribution_card, bool(png)) + self.stats_status.value = ui.status( + "ok", + f"Statistics of {plural(len(summaries), LEVEL_NAMES[level])} calculated.", + [f"From the table {path}"], + ) # Shared ------------------------------------------------------------------------------ diff --git a/src/morphobricks_preprocessing/run.py b/src/morphobricks_preprocessing/run.py index 6352a1b2..68718ac9 100644 --- a/src/morphobricks_preprocessing/run.py +++ b/src/morphobricks_preprocessing/run.py @@ -18,7 +18,10 @@ import logging import os -from collections.abc import Callable, Iterable +import shutil +import tempfile +from collections.abc import Callable, Iterable, Iterator +from contextlib import contextmanager from dataclasses import dataclass, replace from datetime import datetime, timezone from pathlib import Path @@ -385,20 +388,21 @@ def stop(**counts) -> RunResult: for record in records: per_file[record.source_relative_path] = per_file.get(record.source_relative_path, 0) + 1 output.mkdir(parents=True, exist_ok=True) - csv_path = write_plane_histograms(records, output / OUTPUT_FILENAME) - manifest = RunManifest( - experiment_root=root.as_posix(), - dtype=dtype.name, - settings=settings, - mapping=mapping, - files=tuple( - replace(entry, plane_count=per_file[rel]) for rel, entry in sorted(entries.items()) - ), - csv_sha256=file_sha256(csv_path), - versions=package_versions(), - written_at=datetime.now(timezone.utc).isoformat(timespec="seconds"), - ) - write_manifest(manifest, output / MANIFEST_FILENAME) + with _staged_outputs(output) as staged: + csv_path = write_plane_histograms(records, staged / OUTPUT_FILENAME) + manifest = RunManifest( + experiment_root=root.as_posix(), + dtype=dtype.name, + settings=settings, + mapping=mapping, + files=tuple( + replace(entry, plane_count=per_file[rel]) for rel, entry in sorted(entries.items()) + ), + csv_sha256=file_sha256(csv_path), + versions=package_versions(), + written_at=datetime.now(timezone.utc).isoformat(timespec="seconds"), + ) + write_manifest(manifest, staged / MANIFEST_FILENAME) logger.info( "Wrote %d planes (%d files read, %d kept) to %s", len(records), @@ -417,6 +421,63 @@ def stop(**counts) -> RunResult: ) +class _OutputRollbackError(RuntimeError): + """Raised when an output update fails and its earlier files cannot be restored.""" + + +@contextmanager +def _staged_outputs(output: Path) -> Iterator[Path]: + """Prepare both outputs before publishing, and retain backups if recovery fails.""" + staged = Path(tempfile.mkdtemp(prefix=".run-", dir=output)) + keep_backups = False + try: + yield staged + _publish_outputs(output, staged) + except _OutputRollbackError: + keep_backups = True + raise + finally: + if not keep_backups: + try: + shutil.rmtree(staged) + except OSError: + # A leftover temporary directory does not invalidate completed output. + logger.warning("Cannot remove temporary output folder %s", staged, exc_info=True) + + +def _publish_outputs(output: Path, staged: Path) -> None: + """Replace the output pair, restoring its previous state on a publication failure.""" + filenames = (OUTPUT_FILENAME, MANIFEST_FILENAME) + backup = staged / "previous" + backup.mkdir() + # Copy before replacing either file, so even a backup failure leaves the + # earlier output intact. Keep backups on the same filesystem for atomic restores. + for name in filenames: + previous = output / name + if previous.exists() or previous.is_symlink(): + shutil.copy2(previous, backup / name, follow_symlinks=False) + published: list[str] = [] + try: + for name in filenames: + os.replace(staged / name, output / name) + published.append(name) + except BaseException: + try: + for name in reversed(published): + previous = backup / name + if previous.exists() or previous.is_symlink(): + os.replace(previous, output / name) + else: + (output / name).unlink() + except BaseException as exc: + raise _OutputRollbackError( + "The output update failed and the earlier files could not be fully restored. " + f"Backup files have been kept in {backup}. Close programs using the output " + "files and restore the backups before running again." + ) from exc + raise + + def _read_file( root: Path, rel: str, labels: tuple[str, str, str], mapping: AxisMapping, settings ) -> list[PlaneRecord]: diff --git a/tests/test_notebook_ui.py b/tests/test_notebook_ui.py index 591efaba..27ed70b3 100644 --- a/tests/test_notebook_ui.py +++ b/tests/test_notebook_ui.py @@ -8,7 +8,7 @@ pytest.importorskip("ipywidgets") pytest.importorskip("matplotlib") -from morphobricks_preprocessing import Plane, read_manifest # noqa: E402 +from morphobricks_preprocessing import Plane, read_manifest, read_plane_histograms # noqa: E402 from morphobricks_preprocessing import notebook_style as ui # noqa: E402 from morphobricks_preprocessing.axes import AxisMapping # noqa: E402 from morphobricks_preprocessing.notebook_ui import ( # noqa: E402 @@ -272,6 +272,49 @@ def test_missing_axes_card(tmp_path): assert not nb.confirm_button.disabled +@pytest.mark.parametrize("blocking_file", [False, True]) +def test_confirmed_missing_axes_allow_running_unless_other_errors_remain(tmp_path, blocking_file): + root = tmp_path / "exp" + write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZCYX") + write_data(root / "b.tif", np.zeros((2, 8, 8), np.uint8), axes="ZYX") + if blocking_file: + write_file(root / "bad.tif", b"garbage") + nb = checked(root) + nb.labels_confirmed.button.click() + nb.preview_button.click() + nb.preview_checked.button.click() + assert nb.run_button.disabled + nb.missing_as_singleton.button.click() + nb.confirm_button.click() + assert nb.confirmed_mapping is not None + assert nb.run_button.disabled == blocking_file + if blocking_file: + assert nb.run() is None + assert "could not be opened" in nb.folder_issues.value + else: + result = nb.run() + assert result.written + rows = read_plane_histograms(result.csv_path) + assert [(r.z, r.c) for r in rows if r.source_relative_path == "b.tif"] == [(0, 0), (1, 0)] + + +def test_undoing_missing_axes_confirmation_blocks_running_again(tmp_path): + root = tmp_path / "exp" + write_data(root / "a.tif", np.zeros((2, 3, 8, 8), np.uint8), axes="ZCYX") + write_data(root / "b.tif", np.zeros((2, 8, 8), np.uint8), axes="ZYX") + nb = checked(root) + nb.labels_confirmed.button.click() + nb.missing_as_singleton.button.click() + nb.preview_button.click() + nb.preview_checked.button.click() + nb.confirm_button.click() + assert not nb.run_button.disabled + nb.confirm_button.click() # unlock the roles before undoing singleton confirmation + nb.missing_as_singleton.button.click() + assert nb.run_button.disabled and nb.confirm_button.disabled + assert nb.run() is None + + def test_conflicting_axes_explain_why_no_roles_are_offered(tmp_path): write_data(tmp_path / "exp/a.tif", np.zeros((2, 8, 8), np.uint8), axes="ZYX") write_data(tmp_path / "exp/b.tif", np.zeros((2, 8, 8), np.uint8), axes="CYX") @@ -369,6 +412,7 @@ def broken(*args, **kwargs): assert nb.run() is None assert "unexpected problem" in nb.run_status.value assert "RuntimeError: disk on fire" in nb.run_status.value + assert "Nothing was written" not in nb.run_status.value assert "Traceback" in nb.details.value assert not nb.run_button.disabled @@ -376,6 +420,58 @@ def broken(*args, **kwargs): # Step 4 ----------------------------------------------------------------------------------- +@pytest.mark.parametrize("failure", [None, "read", "pool", "plot"]) +def test_statistics_stay_busy_until_finished_and_recover_on_failure( + experiment, monkeypatch, failure +): + import morphobricks_preprocessing.notebook_ui as module + + nb = confirmed(experiment) + nb.run() + stages = [] + for stage, name in ( + ("read", "read_plane_histograms"), + ("pool", "group_summaries"), + ("plot", "distribution_png"), + ): + original = getattr(module, name) + + def while_busy(*args, stage=stage, original=original, **kwargs): + assert nb.statistics_button.disabled + assert nb.statistics_button.description.strip() == "Calculating…" + assert nb.statistics_button.icon == "spinner" + assert "mb-spinner" in nb.stats_status.value + assert "calculated." not in nb.stats_status.value + stages.append(stage) + nb.show_statistics() # a second request must not start another calculation + if failure == stage: + if stage == "read": + raise OSError("simulated unreadable table") + raise RuntimeError("simulated calculation failure") + return original(*args, **kwargs) + + monkeypatch.setattr(module, name, while_busy) + nb.statistics_button.click() + assert stages == ["read", "pool", "plot"][: {"read": 1, "pool": 2}.get(failure, 3)] + assert not nb.statistics_button.disabled + assert nb.statistics_button.description.strip() == "Calculate statistics" + assert nb.statistics_button.icon == "calculator" + assert "mb-spinner" not in nb.stats_status.value + if failure: + assert "mb-error" in nb.stats_status.value + assert not ui.is_visible(nb.statistics_card) + assert not ui.is_visible(nb.distribution_card) + if failure == "read": + assert "The table cannot be read" in nb.stats_status.value + else: + assert "simulated calculation failure" in nb.stats_status.value + assert "Traceback" in nb.details.value + else: + assert "Statistics of 2 conditions calculated" in nb.stats_status.value + assert ui.is_visible(nb.statistics_card) + assert ui.is_visible(nb.distribution_card) + + def test_statistics_by_level_and_percentiles(experiment): nb = confirmed(experiment) nb.run() diff --git a/tests/test_run.py b/tests/test_run.py index 739709b8..470b7d6a 100644 --- a/tests/test_run.py +++ b/tests/test_run.py @@ -347,6 +347,131 @@ def test_first_run_with_errors_creates_no_output(experiment): assert not result.output_folder.exists() +@pytest.mark.parametrize("failure", ["csv", "hash", "manifest", "backup"]) +def test_preparing_outputs_preserves_earlier_files_on_failure(experiment, monkeypatch, failure): + first = run_experiment(experiment, ZYX) + before = {path.name: path.read_bytes() for path in (first.csv_path, first.manifest_path)} + touch(experiment / IMAGES[0], stack(99)) + original_hash = run_module.file_sha256 + + def broken_csv(*args, **kwargs): + raise OSError("simulated CSV failure") + + def broken_hash(path): + if Path(path).parent != first.output_folder: + raise OSError("simulated hash failure") + return original_hash(path) + + def broken_manifest(*args, **kwargs): + raise OSError("simulated manifest failure") + + def broken_backup(*args, **kwargs): + raise OSError("simulated backup failure") + + with monkeypatch.context() as patch: + if failure == "csv": + patch.setattr(run_module, "write_plane_histograms", broken_csv) + elif failure == "hash": + patch.setattr(run_module, "file_sha256", broken_hash) + elif failure == "manifest": + patch.setattr(run_module, "write_manifest", broken_manifest) + else: + patch.setattr(run_module.shutil, "copy2", broken_backup) + with pytest.raises(OSError, match="simulated"): + run_experiment(experiment, ZYX) + assert {path.name: path.read_bytes() for path in first.output_folder.iterdir()} == before + retried = run_experiment(experiment, ZYX) + assert retried.written and retried.changed == (IMAGES[0],) + + +@pytest.mark.parametrize("previous", ["none", "both", "csv", "manifest"]) +@pytest.mark.parametrize("failed_name", [OUTPUT_FILENAME, MANIFEST_FILENAME]) +def test_publishing_outputs_restores_the_previous_state( + experiment, monkeypatch, previous, failed_name +): + first = run_experiment(experiment, ZYX) + for path, keep in ((first.csv_path, "csv"), (first.manifest_path, "manifest")): + if previous not in ("both", keep): + path.unlink() + unrelated = first.output_folder / "notes.txt" + unrelated.write_text("keep this file", encoding="utf-8") + before = {path.name: path.read_bytes() for path in first.output_folder.iterdir()} + touch(experiment / IMAGES[0], stack(99)) + original_replace = os.replace + failed = False + + def broken_replace(source, destination): + nonlocal failed + destination = Path(destination) + if destination == first.output_folder / failed_name and not failed: + failed = True + raise PermissionError("simulated publication failure") + return original_replace(source, destination) + + with monkeypatch.context() as patch: + patch.setattr(run_module.os, "replace", broken_replace) + with pytest.raises(PermissionError, match="simulated publication failure"): + run_experiment(experiment, ZYX) + assert failed + assert {path.name: path.read_bytes() for path in first.output_folder.iterdir()} == before + retried = run_experiment(experiment, ZYX) + assert retried.written + assert read_manifest(retried.manifest_path).csv_sha256 == run_module.file_sha256( + retried.csv_path + ) + assert unrelated.read_text(encoding="utf-8") == "keep this file" + + +def test_interrupted_publication_restores_earlier_output(experiment, monkeypatch): + first = run_experiment(experiment, ZYX) + before = {path.name: path.read_bytes() for path in first.output_folder.iterdir()} + touch(experiment / IMAGES[0], stack(99)) + original_replace = os.replace + + def interrupted(source, destination): + if Path(destination) == first.manifest_path: + raise KeyboardInterrupt + return original_replace(source, destination) + + monkeypatch.setattr(run_module.os, "replace", interrupted) + with pytest.raises(KeyboardInterrupt): + run_experiment(experiment, ZYX) + assert {path.name: path.read_bytes() for path in first.output_folder.iterdir()} == before + + +def test_failed_rollback_retains_backups_and_reports_their_location(experiment, monkeypatch): + first = run_experiment(experiment, ZYX) + before = {path.name: path.read_bytes() for path in first.output_folder.iterdir()} + touch(experiment / IMAGES[0], stack(99)) + original_replace = os.replace + + def broken_replace(source, destination): + if Path(destination) == first.manifest_path or Path(source).parent.name == "previous": + raise PermissionError("simulated locked output") + return original_replace(source, destination) + + monkeypatch.setattr(run_module.os, "replace", broken_replace) + with pytest.raises(RuntimeError, match="could not be fully restored") as excinfo: + run_experiment(experiment, ZYX) + (staged,) = first.output_folder.glob(".run-*") + backup = staged / "previous" + assert str(backup) in str(excinfo.value) + assert {path.name: path.read_bytes() for path in backup.iterdir()} == before + + +def test_cleanup_failure_does_not_report_completed_output_as_failed( + experiment, monkeypatch, caplog +): + def broken_cleanup(path): + raise PermissionError("simulated temporary folder in use") + + monkeypatch.setattr(run_module.shutil, "rmtree", broken_cleanup) + result = run_experiment(experiment, ZYX) + assert result.written + assert read_manifest(result.manifest_path).csv_sha256 == run_module.file_sha256(result.csv_path) + assert "Cannot remove temporary output folder" in caplog.text + + def test_output_inside_experiment_is_refused(experiment): result = run_experiment(experiment, ZYX, output_folder=experiment / "results") assert "output_inside_experiment" in codes(result.issues)