diff --git a/AGENTS.md b/AGENTS.md index c2b97f9..656b878 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -834,13 +834,14 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p tests skip without zarr; CI's Python 3.11 jobs pin zarr 2.18 and 3.13 jobs get zarr 3. - **The notebook file is tested too:** - valid nbformat 4, with no outputs; - - code cells hidden and titled `# @title . …` (1. Install, 2. Mount Google Drive, 3. Run preprocessing); + - code cells hidden as Colab forms (`"cellView": "form"`) and titled `# @title . …` (1. Install, 2. Mount Google Drive, 3. Run preprocessing); - Colab-only code and `/content` paths only inside the Colab guard; no magics; - the GitHub install pinned to `RELEASE_TAG`, equal to `current_version` and the version index; - exactly three code cells, in order: 1 installs and reports versions, 2 mounts Google Drive on Colab (a one-line note elsewhere), 3 displays only widgets built by `notebook_ui` and, - run out of order, names the cell to run first (`tests/test_notebook_setup.py`); + run before cell 1, names cell 1 (`tests/test_notebook_setup.py`); every stop of a cell is + one line without a traceback (the cell's `CellStop`, tested in a real kernel); - the buttons it names exist; - its cells execute, by `exec` and in a real kernel (`nbclient`, `tests/test_notebook_file.py`; cell 1's alternatives and the cells out of order in `tests/test_notebook_setup.py`). @@ -874,8 +875,13 @@ plainer version "not interactive enough". Descriptive, guided and forgiving is t - Outputs are stripped. - Three code cells (owner, 10 November 2026, as ZeroCostDL4Mic and NucleiSky): **1. Install**, **2. Mount Google Drive**, **3. Run preprocessing**, each under a Markdown heading that - explains it. Cell 3 checks that cell 1 ran (`SETUP_RECORD`) and, on Colab, that Drive is - mounted, and otherwise stops naming the cell to run; no cell runs another. + explains it. Cell 3 checks that cell 1 ran (`SETUP_RECORD`), and otherwise stops naming + cell 1; on Colab without Drive mounted it opens the app after one line saying that only + files on the runtime are visible (owner, v0.5.1). No cell runs another. + - A cell that cannot go on raises its own `CellStop` (a `RuntimeError` whose + `_render_traceback_` returns the message), so Colab and JupyterLab show one line and no + traceback (owner, v0.5.1). A stop caused by another error (the interface import, the Drive + mount) names that error on the same line; an error that is not a stop keeps its traceback. Cell 1 checks the Python version before `import tomllib`. - Cell 1 is a standalone standard-library bootstrap: nearby checkout, installed package, or GitHub installation of the release `RELEASE_TAG`, with missing requirements installed using the kernel's Python (`subprocess`, never `!pip`); on Colab with the @@ -1051,14 +1057,14 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb --- -## 11. Status (current state, v0.4.0) +## 11. Status (current state, v0.5.1) -- **Released:** v0.4.0 (5 November 2026), with the normalization brick pinned at v0.3.4. The +- **Released:** v0.5.1 (10 November 2026), with the normalization brick pinned at v0.4.0. The per-release history is in [CHANGELOG.md](CHANGELOG.md); the milestones PP-0 to PP-13 are in [docs/archive/plan-milestones-v0.1-v0.4.md](docs/archive/plan-milestones-v0.1-v0.4.md); the older status log of this file is in [docs/archive/agents-status-history-v0.1-v0.3.md](docs/archive/agents-status-history-v0.1-v0.3.md). -- **Unreleased:** see the changelog's "Unreleased" section. +- **Unreleased:** none yet; later changes go in a new "Unreleased" section of the changelog. - **Scope:** 2-D per plane (a Z-stack is refused as `z_stack_not_supported`); TIFF and OME-Zarr input read directly (TIFF to OME-Zarr conversion and the `_moved.py` re-exports were removed); optional masks, the mask step and the notebook's **Describe the masks**; named-axis copies diff --git a/CHANGELOG.md b/CHANGELOG.md index 01239fc..5dc05c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,108 @@ The design is `docs/restart/preprocessing-design-v0.4.md` in MorphoFeature, on b are in [docs/plan.md](docs/plan.md) §12. (The v0.3 to v0.4 migration guide was removed in v0.5.1: no run before v0.5.1 is read.) -## Unreleased (v0.5.1) +## v0.5.1 (10 November 2026) + +The first release in which every run is described, with the shared request resolver for the +other bricks and the seven-step notebook. There is no backward compatibility before the first +pipeline release (MorphoBricks-Spec ADR 0002 AR10): older run folders, records and drafts are +refused, never converted. Normalization is pinned at v0.4.0. + +**Breaking** + +- **Every run needs a published experiment description** (`morphobricks.toml`, saved by the + notebook's *Check and save*). The run, the mask step, `check`/`run`/`masks` and + `RunInputs.open` refuse an output folder without one with `A_DESCRIPTION_REQUIRED`, before + anything is read or written. +- **Older files are refused, not converted**: run manifests of versions 1 to 3 (every run of + v0.5.0 or earlier: run preprocessing again, which reads every image again), conversion + records other than version 3 (convert the originals again), answer drafts saved under the + earlier Spec pin (`spec_commit_mismatch`), and hand-edited manifests missing a key. +- **The legacy `experiment_design.csv` migration is gone**: the file is an output only and + nothing reads it back; the `design` command and `run --design` are removed. +- **Normalization v0.4.0**: `PlaneRecord.position_id` is required, a blank `position_id` in a + table row is refused, and tables of schema versions 1 to 3 are refused. + +**New** + +- **`RunInputs.resolve_request`** (resolver contract version 1): one interpreter of a described + run for every analysis route, channels by canonical ID per position, every refusal at once, + coverage per biological repeat and the provenance. +- **`Fov.image_type`**: each position's declared image type, `"rgb"` or `"channels"`. +- **Lab-tool links**: `run --lab-links FILE` (and `--confirm-repeat-codes`), recorded in the + run manifest only. +- **The seven-step notebook**: Images → Axes → Channels → Masks → Design → Check and save → + Run, with a quick scan and an example picture before the headers are read, progress for every + wait, one column and no box that scrolls on its own. Three code cells shown as Colab forms; + a cell that cannot go on stops with one line, without a traceback. + +**Fixes** + +- The notebook output is no longer flooded with debug messages from ImageJ TIFFs. +- *Is compared against* keeps its dropdown; lists of files draw no empty bullets; every + disabled action says beside it what it waits for. +- On Python 3.10, cell 1 says "Select a Python 3.11 or newer kernel" instead of failing on + `import tomllib`. + +### Migration from v0.5.0 + +**Users.** Open the v0.5.1 notebook, describe the experiment (the seven steps) and run again +into the same output folder: the run reads every image again and replaces the v0.5.0 output. +Nothing in an old `experiment_design.csv` is carried over; describe the design in *Design*. On +the command line, `check`, `run` and `masks` need the published description: write it in the +notebook (`describe` shows what is still open). A `.morphobricks-conversion.json` written before +v0.5.1 is refused: run `convert` again into a new folder. + +**Bricks** (each is unaffected until its pin moves to v0.5.1): + +| v0.5.0 | v0.5.1 | +|---|---| +| A run without a description | Refused (`A_DESCRIPTION_REQUIRED`, `DescriptionRequiredError`); tests use `testing.make_clean_run` (described) or publish a `morphobricks.toml` before `run_experiment` | +| `run_experiment(design=...)`, `run --design`, the `design` command, `edit_design`, `write_check_outputs`/`CheckOutputs` | Removed: the design is derived from the description | +| `read_confirmed_design`, `ConfirmedDesign`, `DesignCheckError`, `RunInputs.design()` | Removed: `Fov.repeat_group` or `RunInputs.resolve_request` | +| `RunInputs.image_type()` | Removed: `Fov.image_type` (or `FovSelection.image_type`) | +| `RunInputs.description()` and its siblings could be None | Never None | +| `preflight`/`remasked`/`masks.find_masks(with_description=...)`, `run.preflight.description_in_use` | Removed: the name refusals always apply | +| `STRICT_MANIFEST_VERSION`, `DESCRIBED_MANIFEST_VERSION`, `LEGACY_MANIFEST_VERSION`, `version_for` | Removed: `MANIFEST_VERSION` is `"4"`, the only version read | +| `source_fingerprint` | Removed: `inventory.source_signature` | +| `GroupSummary.plane_count` | Removed: `xyc_plane_count` or `channel_histogram_count` | +| `discovery.TIFF_SUFFIXES`, `discovery.SYSTEM_NAMES` | Moved to `image_names` (`is_tiff_name` and `is_system_name` still import from `discovery`) | +| The design editor: `design.ExperimentDesign`, `DesignChanges`, `default_design`, `load_design`, `read_experiment_design`; the module `design_edit` (`DesignEdit`, `DesignEditError`, `apply_design_edit`, `parse_repeat_group`); `run.design_file` (`design_for_run`, `write_design`, `write_design_template`); `report.design_summary`, `design_table`; the module `cli_design` | Removed: the design rows are the read-only `DesignTable`, derived from the description | +| `draft.migrate_legacy_design`, `LegacyMigrationRefused`, `MigrationNote`, `Draft.origin`, `Draft.legacy_missing`, `Draft.notes`, `Draft.unresolved_controls` | Removed | +| `PreprocessingNotebook(describe=True)` | `PreprocessingNotebook()`: no `describe` argument; an embedding brick must let the user describe and save before the run | +| `PreprocessingNotebook.design`; the widgets `group_confirmations`, `groups_box`, `controls_box`, `control_checks`, `design_change` | `.pairing`; the widgets are removed | +| `welcome()`: a page with a *What next?* box | A one-line `VBox` (name, version, ready) | +| `results_step()` | After `steps()`, an empty hidden box (*Compare groups* is in the run step); alone, the page it was | +| `check_candidate(..., output)` | No `output` argument | +| `PlaneRecord(...)` (normalization v0.3.4) | `PlaneRecord(..., position_id=...)`, keyword-only and required | + +Added for bricks: `Request`, `MaskRequest`, `Capabilities`, `ResolvedRequest`, `FovSelection`, +`MaskSelection`, `PlaneSelection`, `Refusal`, `RepeatCoverage`, `Provenance`, +`RESOLVER_VERSION`, `VolumeNotSupportedError`; `LabLinks`, `LabLinksError`, `read_lab_links`, +`RunMode(lab_links=...)`; `make_described_run(..., mask_images=...)` and +`testing.described_images`; `progress=` callbacks of `discover_experiment`, `masks.find_masks` +and `run.preflight.remasked`. The pinned surface is `tests/test_brick_api_surface.py` (contract +§9.1). The details are in the sections below. + +### Release v0.5.1: the shipped notebook + +- `RELEASE_TAG` `v0.5.1`, `current_version` `0.5.1`, and `RELEASE_REF` the v0.5.1 preparation + commit `d023ff9`; cell 3 calls `PreprocessingNotebook()` (`test_notebook_file.RELEASE_CALLS` records + the pair). +- Each code cell is a Colab form with its code hidden (`"cellView": "form"`, AGENTS.md §7). +- A cell that cannot go on (cell 3 before cell 1, a refused checkout, a failed install or Drive + mount, Python 3.10) stops with its message on one line and no traceback: it raises the cell's + own `CellStop`, a `RuntimeError` whose `_render_traceback_` is the message (shown so by + IPython, so in Colab and JupyterLab; tested in a real kernel). A stop caused by another error + names it on that line, as in "… its notebook interface could not load (ImportError: …)" or + "Google Drive could not be connected (ValueError: …)", so the one line never hides what + failed; any other error keeps its full traceback (tested in a real kernel). +- **Cell 3 on Colab without Drive opens the app** after one line: "Google Drive is not mounted: + only files on this Colab runtime are visible, and they are deleted when the runtime stops." + (It stopped and named cell 2 before.) +- Cell 1 checks the Python version before `import tomllib`, so Python 3.10 gets the "Python + 3.11 or newer" line. +- The introduction describes the seven-step app in short. ### The legacy `experiment_design.csv` migration is removed (inventory PP-03) diff --git a/README.md b/README.md index e74ac92..876b6b4 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ from there. ## Status -Pre-alpha; the current release is v0.5.0 (what is planned next is in +Pre-alpha; the current release is v0.5.1 (what is planned next is in [docs/roadmap.md](docs/roadmap.md)). Implemented and tested: - Folder discovery and validation, TIFF and OME-Zarr header inspection without reading pixels @@ -57,7 +57,7 @@ v0.5.1 is refused until it is run again; nothing is converted. See the [CHANGELO ## Installation ```bash -pip install "git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.0" +pip install "git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.1" ``` For development, from a clone: @@ -67,7 +67,7 @@ pip install -e ".[dev]" ``` The optional `notebook` extra installs the widget dependencies for the notebook -(`pip install "morphobricks-preprocessing[notebook] @ git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.0"`). +(`pip install "morphobricks-preprocessing[notebook] @ git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.1"`). The notebook's first code cell (**1. Install**) can set this up automatically: it uses a compatible nearby checkout or installed package, and installs missing requirements into the active kernel. A standalone download installs from GitHub if needed (internet access and Git required). @@ -85,9 +85,10 @@ LabConstrictor's code cells, each under a heading that explains it, as in the lab's ZeroCostDL4Mic and NucleiSky notebooks: **1. Install MorphoBricks preprocessing** (prepares the environment and reports versions), **2. Mount Google Drive** (Colab only; elsewhere a one-line note) and - **3. Run preprocessing** (explicitly displays the complete preprocessing UI). Cell 3 run out - of order says which cell to run first and never runs it for the user. Users never edit - code, so LabConstrictor may hide it. + **3. Run preprocessing** (explicitly displays the complete preprocessing UI). Cell 3 run + before cell 1 says in one line to run cell 1 and never runs it for the user; a cell that + cannot go on stops with one line, without a traceback. Users never edit code: each code + cell is a Colab form with its code hidden, and LabConstrictor may hide it. 2. Colab-only setup stays inside `if "google.colab" in sys.modules:` and mounts Google Drive. The UI uses only core ipywidgets supported by both ipywidgets 7.7 and 8; it does not force an ipywidgets upgrade/restart or enable Colab's custom-widget manager. Locally the same notebook @@ -223,7 +224,7 @@ ruff format . pytest ``` -Coding conventions are described in [docs/development.md](docs/development.md). Planned work after v0.5.0 is in [docs/roadmap.md](docs/roadmap.md). +Coding conventions are described in [docs/development.md](docs/development.md). Planned work after v0.5.1 is in [docs/roadmap.md](docs/roadmap.md). ## License diff --git a/docs/colab-release-check.md b/docs/colab-release-check.md index c1b5a70..eabb28d 100644 --- a/docs/colab-release-check.md +++ b/docs/colab-release-check.md @@ -1,124 +1,84 @@ -# Colab and JupyterLab release smoke check - -Automated tests execute the notebook in a local kernel and fake `google.colab`, but they -cannot prove that a browser frontend renders or reacts to widgets correctly. Any release -candidate that changes the notebook, bootstrap, ipywidgets support or notebook UI must be -smoke-tested in both a fresh Google Colab runtime and JupyterLab/LabConstrictor before tagging. - -This is a release gate, not an optional final-major-release check. A frontend failure is a -release blocker even when all headless notebook tests pass. The list collects the hand-check lists of PR #118 -(PP-8d, Colab readiness), PR #123 (PP-10d, Describe the masks) and PR #127 (PP-11b, Make -named-axis copies). Items were updated for later changes: PP-10a replaced the stale-lock code -cell with a button, and PP-12 replaced the `.run-*` backup folders with `.publish` and -`.staging`. - -**The notebook's pin.** The shipped notebook installs the package at its `RELEASE_REF` (set in -cell 1), not main's code, so its cell 3 calls the API of the code at that commit. Between releases main's code -moves on while the cell keeps the released call: after v0.5.1's PP-C, main has no `describe` -argument, while `RELEASE_REF` is still v0.5.0's `fd7afd3`, whose cell calls -`PreprocessingNotebook(describe=True)`. The release PR moves `RELEASE_REF` and cell 3's call -in the same commit and records the new pair in `RELEASE_CALLS` of `tests/test_notebook_file.py`, -which fails for a `RELEASE_REF` without one. - -- [ ] `RELEASE_REF` is the release commit, and cell 3's `PreprocessingNotebook(...)` call is - that commit's API (v0.5.1: `PreprocessingNotebook()`). - -Use a dataset on Google Drive with a masks folder, and a second small experiment of -Y/X-swapped OME-TIFFs for section F. Note the tag, the date and any failure next to each item. -Report a failure as an issue before the final tag. - -## A. Fresh runtime and app rendering - -The notebook has three code cells, run in order: **1. Install MorphoBricks preprocessing**, -**2. Mount Google Drive** and **3. Run preprocessing**. - -1. Open the notebook on a fresh Google Colab runtime and run cell 3 first. - - [ ] It stops at once with "run cell 1 (Install MorphoBricks preprocessing) first", not an - `ImportError`, and installs nothing. -2. Run cell 1 (install). - - [ ] Setup reports the expected notebook, preprocessing, normalization and ipywidgets - versions and the pinned source revision, then names cells 2 and 3. - - [ ] No ipywidgets upgrade or runtime restart is requested, and Drive is not mounted. - - [ ] Run cell 1 again: nothing is installed again and the same versions are reported. -3. Run cell 3 before cell 2. - - [ ] It stops with "Google Drive is not connected … run cell 2 (Mount Google Drive)". -4. Run cell 2 (Drive). - - [ ] Drive authorisation appears, the mount succeeds and the cell says Drive is connected. -5. Run cell 3 (the app). - - [ ] The complete preprocessing app renders inline; Colab does not open the Code snippets - pane instead of the widget. - - [ ] Buttons, dropdowns, accordions, preview image, sliders, progress bars and matplotlib - plots render and react. -6. Repeat cells 1 to 3 in JupyterLab/LabConstrictor. - - [ ] Cell 2 prints its one-line "Not Google Colab" note and does nothing else. - - [ ] The same app renders and uses local paths without executing Colab-only setup. - -## B. Step 1 (PR #118) - -3. - [ ] **Browse…** shows the "could not be opened … Copy path" message. - - [ ] A pasted **/content/drive/MyDrive/…** path checks fine. - - [ ] The default output folder lands on Drive next to the experiment. - -## C. Step 3 on the full dataset (PR #118) - -4. - [ ] The run finishes. - - [ ] On Drive, the output folder has `plane_histograms.csv`, `run_manifest.json`, - `experiment_design.csv` and the report (`discovery_report.txt` and `.json`). - - [ ] There are no "(1)" duplicate files, and no leftover `.publish`, `.staging` or - `...tmp` files. - - [ ] There is no lock file (`.morphobricks-preprocessing.lock`) after the run. -5. - [ ] Run Step 3 again in the same session. Everything is unchanged and nothing is read. -6. - [ ] Disconnect the runtime (Runtime › Disconnect and delete runtime) and open a new one. - Run Steps 0–3 again on the same Drive folders. The rows are reused (unchanged). This - shows the Drive times are stable across mounts. -7. Optional: stop a run part-way (interrupt, or disconnect the runtime). - - [ ] The next check names the lock holder (`output_locked`), and **Remove the lock of a - stopped run** clears it. - - [ ] The next run resumes: the report lists the images read before the stop as - `resumed`. -8. - [ ] Optional: an experiment with OME-Zarr images on Drive checks and runs (the `zarr` - extra is installed). - -## D. Describe the masks (PR #123) - -9. - [ ] In Steps 1–3 with an experiment and a masks folder on Drive, before **Compute - histograms**, the card shows a greyed button: "Available when the output folder holds a - run that recorded masks". -10. - [ ] After the run, the help line says "Describes the run in … written …, of …: N mask - types from …", and the button is enabled. -11. - [ ] Click it: a spinner, a progress bar per image, then a green bar. The tables are - labelled as suggestions, and `mask_relationships.json` appears in the output folder. -12. - [ ] Run Step 3 again: the card clears. -13. - [ ] **Reopen** the notebook (Runtime › Restart session) and choose the same folders in - Step 1 without running. The button is enabled, names the earlier run, and works. -14. - [ ] Move the masks folder on Drive and click: a red card says the masks folder does not - exist, and nothing is written. -15. - [ ] The page scrolls as a whole: no box inside it has its own scroll bar, and there is no - horizontal page scrollbar. - -## E. Make named-axis copies (PR #127) - -16. - [ ] Run Step 0, then Step 1 on an experiment of Y/X-swapped OME-TIFFs on Drive, and - press **Check folder**. -17. - [ ] In Step 2, swap the Y and X roles. The red axis check appears, with the **Make - named-axis copies (optional)** card under it. Its button is disabled. -18. - [ ] Look at the example (drawn by itself) and press **Yes, the example looks right**, then **Yes, write the - copies with these roles**. -19. - [ ] Press **Browse…**: the amber card names **Copy path**. Paste a Drive folder inside - the experiment and check that it is refused. Then paste one next to the experiment. -20. - [ ] Check the plan lists and the space, then press **Make named-axis copies**. The bar - moves, the green card shows the record path, and the pixel-size note lists the copies. -21. - [ ] Press **Check the copies in step 1**. Step 1 shows the copy folder, the same output - folder and a clean check. The roles are suggested from the copies. -22. - [ ] Confirm the labels, preview, roles and image type, then run Step 3. The report says - "named-axis copies of …". -23. - [ ] Optional: disconnect the runtime during a conversion, reconnect, and convert into - the same folder. The lock card appears. Press **Remove the lock of a stopped run**, then - convert again. +# Colab and JupyterLab release check + +The tests run the notebook in a real kernel, with `google.colab` faked, but they cannot show +that a browser draws the widgets and reacts to clicks. So before a release is tagged, the owner +opens the release PR's notebook (its Colab link and its raw download, both pinned to the +`RELEASE_REF` commit) and ticks this list once in a fresh Google Colab runtime and once in +JupyterLab or LabConstrictor. A failure here blocks the tag, even when every test passes. + +Use an experiment on Google Drive (Colab) or on disk (JupyterLab) with two conditions, at least +two biological repeats and a masks folder. Write the date, the commit and any failure (as an +issue number) next to each item. + +**Before opening it.** `RELEASE_REF` in cell 1 is the release preparation commit, and cell 3's +`PreprocessingNotebook(...)` call is that commit's API (v0.5.1: `PreprocessingNotebook()`). The +release PR moves both together and records the pair in `RELEASE_CALLS` of +`tests/test_notebook_file.py`, which fails for a `RELEASE_REF` without one. + +## A. The three cells (Colab) + +1. - [ ] The code of the three cells is hidden: each shows its title (a Colab form). +2. Fresh runtime, run **cell 3** first. + - [ ] One line: "… run cell 1 (Install MorphoBricks preprocessing) first …", no traceback; + nothing is installed. +3. Run **cell 1**. + - [ ] It prints **Ready** with the preprocessing (0.5.1), normalization (0.4.0) and + ipywidgets versions and the pinned source, then names cells 2 and 3. No restart or + ipywidgets upgrade is asked for; Drive is not mounted. + - [ ] Run it again: nothing is installed again, the same versions. +4. Run **cell 3** before cell 2. + - [ ] One line, "Google Drive is not mounted: only files on this Colab runtime are visible, + and they are deleted when the runtime stops.", then the app opens. +5. Run **cell 2**, allow access. + - [ ] "Google Drive is connected: your files are in /content/drive/MyDrive." +6. Run **cell 3** again. + - [ ] No Drive line; the app renders inline (not the Code snippets pane): the one-line + welcome, the strip of seven steps and the first step. + +## B. The seven steps (Colab, the experiment on Drive) + +7. - [ ] **Images:** paste the Drive paths (experiment, masks, output); **Browse…** says it + cannot open here and offers **Copy path**. **Scan the folders** shows the folders and an + example picture with a progress line; **Looks right: read all image headers** shows its + progress (done of total, time left) and the result. +8. - [ ] **Axes:** one row per dimension with its role, the example redrawn when a role changes, + the pixel size in one line, the image type in one line (**Use for all folders**). The roles + are confirmed only by the click. +9. - [ ] **Channels:** name the stains, then tag each folder's channels. +10. - [ ] **Masks:** **Check the masks** runs with progress; each mask type can be described. + **Describe the masks** is greyed out, with its reason, until a run exists. +11. - [ ] **Design:** the grid of conditions by biological repeats; **Looks right: confirm the + design**. +12. - [ ] **Check and save:** the experiment in words; **Yes, save the experiment description** + writes `morphobricks.toml` in the output folder. +13. - [ ] **Run:** **Compute histograms** finishes with progress; the intensity checks, + **Compare groups** and the report are collapsed panels and open. The output folder holds + `plane_histograms.csv`, `run_manifest.json`, `experiment_design.csv` and the report, with + no "(1)" copies, no `.publish`, `.staging` or `.tmp` leftovers and no lock file. +14. - [ ] Back in **Masks**, **Describe the masks** is enabled and writes + `mask_relationships.json`. +15. - [ ] Throughout: one column, the page scrolls as a whole (no box with its own scroll bar, + no horizontal page scroll bar); every greyed-out button says beside it what it waits for; + no flood of debug messages under the cell. + +## C. Again later (Colab) + +16. - [ ] Disconnect and delete the runtime, open a new one, run cells 1-3 on the same Drive + folders: the steps open answered, and **Compute histograms** reuses every row. +17. - [ ] Optional: stop a run part-way. The next check names the lock; **Remove the lock of a + stopped run** clears it, and the next run resumes. + +## D. JupyterLab or LabConstrictor + +18. - [ ] Download the raw notebook; run cell 3 first (the one line naming cell 1), then cells + 1, 2 and 3. Cell 2 prints only "Not Google Colab: there is no Google Drive to mount here, + so nothing was done."; cell 3 prints no Drive line. +19. - [ ] Repeat B (7-15) with local paths; **Browse…** opens a folder chooser on a local + kernel with a display (otherwise it says to paste the path). +20. - [ ] On a Python 3.10 kernel, if one is at hand: cell 1 says in one line to select Python + 3.11 or newer. ## After the check -- Record the result (tag, date, items passed, failures with their issue numbers) in the - release PR of v0.4.0. -- Then update this file's first paragraph, the README's "Notebook" section and plan §10 - (release checklist) to say which release was checked on Colab. +Write the result (date, commit, items passed, failures with their issue numbers) in the release +PR. Then tag. diff --git a/docs/conformance.md b/docs/conformance.md index 0efeef7..5af8d37 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -6,6 +6,46 @@ real run; a skip is never counted as a pass. Owner rulings of 8 November 2026 (i "no skips" applies to the Spec conformance modules (`tests/test_spec_*.py`); a skip elsewhere in the suite is listed as **not run**, with its reason. +## Release record: v0.5.1 (10 November 2026) + +A G1-style record of the v0.5.1 release candidate (as for v0.5.0 below). It does not decide a +gate: the Stage 2 gate (G2) also needs the bricks' items, and deciding a gate is the owner's. + +| | | +|---|---| +| Release candidate | Measured on `9f61cb9` (branch `claude/release-0.5.1`: `main` at `9a5da03`, the v0.5.1 preparation `d023ff9` (commit A), `a4154ad` (commit B), which points the notebook's `RELEASE_REF` at `d023ff91289a2d2204e1ae56f7dee114d1a665e4`, the first record `7607617`, then the review fixes `7beac12` (a cell's one-line stop names the error that caused it; notebook, tests and docs only) and `9f61cb9` (CHANGELOG)). The commit that updates this record changes only this file. `git diff d023ff9 9f61cb9 -- src pyproject.toml` is empty: the installed package is the one `RELEASE_REF` names. Earlier measurement, on `a4154ad` before the review: 5270 passed, 4 skipped; 1732 in the Spec modules, 0 skipped (reproduced by the review) | +| Spec pin | `tests/spec_fixtures/PIN.toml`: MorphoBricks-Spec `cc9820f5b84f01bbbb5c8e3f54e70ee13cb5e9eb` (fixtures tree `40fe48c`), checked by `test_spec_pin` (16 passed) | +| Environment | Linux, Python 3.13.18, `pip install -e '.[dev,notebook,zarr]'` in a fresh venv (package version 0.5.1, normalization 0.4.0, numpy 2.5.4, tifffile 2026.11.8, ipywidgets 8.1.9, zarr 3.4.2, pytest 9.2.0); run as root on a case-sensitive file system | +| Commands | `python tools/check.py` (Ruff, Ruff format, pytest over `tests/`, code rules); `python -m pytest -q -p no:cacheprovider -rsxX --junitxml=… tests/test_spec_*.py` | +| `tools/check.py` | Ruff clean, Ruff format 432 files already formatted, pytest **5274 passed, 4 skipped** (not run, below), 0 failed, code rules passed (exit status 0) | +| Spec modules (`tests/test_spec_*.py`) | **1732 passed, 0 skipped, 0 xfailed, 0 failed**, from all 24 modules of `tests/spec_gate.py` `EXPECTED_MODULES` (JUnit: no `skipped`, `failure` or `error` element in any case) | +| Install | `pip install "morphobricks-preprocessing[notebook] @ git+…@d023ff9…"` in a fresh venv reports 0.5.1 (normalization 0.4.0), its package files equal this branch's `src/morphobricks_preprocessing` (`diff -r`), and `PreprocessingNotebook()` builds `welcome()`, `steps()` and `results_step()` | +| Shipped notebook | Its three code cells, unmodified, in a real kernel (`nbclient`) in a fresh venv: cell 3 first stops with one line naming cell 1; cell 1 installs `RELEASE_REF` from GitHub and prints **Ready** (0.5.1, normalization 0.4.0); cell 2 prints its one-line note; cell 3 displays the app. With `google.colab` faked: cell 3 before cell 2 prints the one Drive line and opens the app; after cell 2 it opens without it. Cell 1 on a Python 3.10 kernel stops with the one "Python 3.11 or newer" line (on `a4154ad`). Re-run by the review on `9f61cb9` in a fresh venv (cell 3 alone, cell 1 twice, cell 2 outside Colab, cell 3; faked Colab: cell 1 adds the `zarr` extra, cell 3 without Drive prints the one line and opens the app), plus an interface that cannot load (one line naming the `ModuleNotFoundError`), a failed Drive mount (one line naming the `ValueError`) and an unexpected error in cell 3 (its full traceback) | + +Test cases per Spec module: catalogue 376, derived design 80, discovery 17, encoding 58, Fov +image type 14, gate 9, lab links 11, no fallback 15, pin 16, publication 20, reader 330, reader +phases 1-3 287, reader refusal paths 8, reconcile 25, records 22, records goldens 76, refval +cross-check 203, resolve request 43, rules table 14, run aliases 15, run description 9, +RunInputs semantic 8, transitions 73, vendored missing 3. New since v0.5.0: +`test_spec_resolve_request` (PP-0), `test_spec_fov_image_type` (PP-1) and +`test_spec_lab_links` (PP-2, the lab-tool-links part of the section 2 "Provenance" row, which +ADR 0001 GS1 deferred from G1 to the Stage 2 gate). The cases removed since v0.5.0 are the +undescribed and legacy ones (PP-C, SWEEP-PP, PP-03; the P30 legacy case per Spec `cc9820f`). + +**Not run (4 environment skips, outside the Spec modules):** + +| Test | Reason | +|---|---| +| `tests/test_cli_log.py:166` | Case-sensitive file system: the case-only collision cannot be built | +| `tests/test_folder_rules.py:199` | Same | +| `tests/test_folder_rules.py:230` | Same | +| `tests/test_discovery.py:665` | Run as root: file modes are ignored, so an unreadable folder cannot be made | + +These four behaviours are **not verified by this run**. + +**Still the owner's:** the Colab and JupyterLab browser check (`docs/colab-release-check.md`), +before tagging. P24, P27 and P31 stay brick outcomes for the Stage 2 gate. + ## Gate G1 record: release candidate v0.5.0 (10 November 2026) | | | diff --git a/docs/plan.md b/docs/plan.md index b8d8cb2..87de8c6 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -1162,6 +1162,10 @@ Raised during review; to be decided before the milestones that need them: report), 2. Mount Google Drive (Colab only; elsewhere a one-line note), 3. Run preprocessing (the app). Cell 3 run before cell 1, or on Colab without Drive, stops and names the cell to run; no cell runs another.* + *Amended (v0.5.1, owner): the code cells are Colab forms with the code hidden; a cell that + cannot go on stops with one line and no traceback (a stop caused by another error names + that error on the same line; any other error keeps its traceback); cell 3 on Colab without + Drive opens the app after one line saying that only files on the runtime are visible.* 59. How are images with axis errors saved as named-axis copies (design v0.4 §5, PP-11)? *Decided (owner, 2 November 2026; implemented in PP-11) as design v0.4 §5 says: images only, as named-axis OME-TIFF (BigTIFF when large); copies keep the file name and folder diff --git a/docs/user_guide.md b/docs/user_guide.md index b2ee688..bca5baf 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -21,8 +21,10 @@ open it and run its three code cells in order: 2. **Mount Google Drive** connects Google Drive on Google Colab. In JupyterLab or LabConstrictor it prints a one-line note and does nothing. 3. **Run preprocessing** displays the complete preprocessing widget app. Run before cell 1, - it stops and says to run cell 1; on Colab without Drive connected, it says to run cell 2. - It never runs an earlier cell for you. + it stops with one line saying to run cell 1. On Colab without Drive mounted it opens the + app after one line: "Google Drive is not mounted: only files on this Colab runtime are + visible, and they are deleted when the runtime stops." It never runs an earlier cell for + you. Cell 1 chooses one source: diff --git a/notebooks/MorphoBricks_Preprocessing.ipynb b/notebooks/MorphoBricks_Preprocessing.ipynb index eedc65b..121a69a 100644 --- a/notebooks/MorphoBricks_Preprocessing.ipynb +++ b/notebooks/MorphoBricks_Preprocessing.ipynb @@ -6,30 +6,25 @@ "source": [ "# 🔬 MorphoBricks preprocessing\n", "\n", - "Prepare microscopy TIFF or OME-Zarr experiments for the other MorphoBricks bricks. The notebook never changes the source images; results are written to a separate output folder.\n", + "Prepares a microscopy experiment of TIFF or OME-Zarr images for the other MorphoBricks bricks. You describe the experiment once; the notebook then computes one intensity histogram per image plane. Your images are never changed: everything is written to a separate output folder.\n", "\n", "### What it does\n", "\n", - "- **Checks the experiment folder and metadata** and reports problems before pixels are processed.\n", - "- **Confirms the experimental design**: folder labels, biological-repeat pairing and controls.\n", - "- **Confirms image interpretation**: axis roles and fluorescence/RGB image type. Scientific metadata are never silently guessed.\n", - "- **Computes per-plane intensity histograms** used by downstream MorphoBricks.\n", - "- **Runs intensity QC** and, on request, compares groups after preprocessing.\n", + "The app has seven steps: **Images → Axes → Channels → Masks → Design → Check and save → Run**. It scans your folders, shows an example picture, asks what the axes, channels, masks and conditions are, saves your answers as the experiment description and runs. Nothing is guessed: what the app proposes is recorded only when you click.\n", "\n", "### Before you start\n", "\n", - "- Put the experiment in one folder containing TIFF (.tif/.tiff) or OME-Zarr images.\n", - "- Optional masks can be supplied from a separate masks folder.\n", - "- **Google Colab:** keep the experiment and output on Google Drive, under `/content/drive/MyDrive/`.\n", + "- One experiment folder of TIFF (.tif/.tiff) or OME-Zarr images, with folders for the conditions and biological repeats. Masks, if any, in a separate folder.\n", + "- **Google Colab:** keep the experiment and the output folder on Google Drive, under **/content/drive/MyDrive/**.\n", "- **JupyterLab / LabConstrictor:** use normal local paths.\n", "\n", "### How to run\n", "\n", - "Run the three code cells below in order, from top to bottom (click a cell and press **Shift + Enter**, or its ▶ button):\n", + "Run the three code cells in order (click a cell and press **Shift + Enter**, or its ▶ button):\n", "\n", "1. **Install** prepares the environment.\n", "2. **Mount Google Drive** connects your Drive on Google Colab; elsewhere it does nothing.\n", - "3. **Run preprocessing** opens the complete preprocessing app.\n", + "3. **Run preprocessing** opens the app.\n", "\n", "After a restart of the runtime or kernel, run the three cells again: cell 1 then installs nothing that is already there." ] @@ -51,6 +46,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "cellView": "form", "collapsed": false, "jupyter": { "source_hidden": true @@ -65,19 +61,31 @@ "import platform\n", "import subprocess\n", "import sys\n", - "import tomllib\n", + "\n", + "\n", + "class CellStop(RuntimeError):\n", + " \"\"\"Stop this cell with its message on one line, without a traceback (Colab and Jupyter).\"\"\"\n", + "\n", + " def _render_traceback_(self) -> list:\n", + " return [str(self)]\n", + "\n", + "\n", + "# Checked first, before anything that needs Python 3.11 (tomllib), so 3.10 gets this one line.\n", + "if sys.version_info < (3, 11):\n", + " raise CellStop(\"Select a Python 3.11 or newer kernel, then run this cell again.\")\n", + "\n", "from importlib import import_module, invalidate_caches\n", "from importlib.metadata import PackageNotFoundError, version\n", "from importlib.util import find_spec\n", "from pathlib import Path\n", "\n", "# LabConstrictor version identifiers.\n", - "current_version = \"0.5.0\"\n", + "current_version = \"0.5.1\"\n", "notebook_name = \"MorphoBricks_Preprocessing\"\n", "\n", "# A downloaded/Colab notebook installs this immutable source revision.\n", - "RELEASE_TAG = \"v0.5.0\"\n", - "RELEASE_REF = \"fd7afd339bcaab7579c806a28a65bb7ecebe940c\"\n", + "RELEASE_TAG = \"v0.5.1\"\n", + "RELEASE_REF = \"d023ff91289a2d2204e1ae56f7dee114d1a665e4\"\n", "REPOSITORY = \"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/CellMigrationLab/morphobricks-preprocessing.git\"\n", "EXTRAS = \"notebook\"\n", "\n", @@ -100,6 +108,8 @@ "\n", "def _checkout_version(checkout: Path) -> str | None:\n", " \"\"\"Return the nearby checkout package version, or None when it cannot be read.\"\"\"\n", + " import tomllib # Python 3.11 or newer, checked at the top of this cell\n", + "\n", " try:\n", " data = tomllib.loads((checkout / \"pyproject.toml\").read_text(encoding=\"utf-8\"))\n", " except (OSError, tomllib.TOMLDecodeError):\n", @@ -146,7 +156,7 @@ " \"it has uncommitted changes\" if dirty else \"its tree state could not be read\"\n", " )\n", " if differences and not DEVELOPMENT_OVERRIDE:\n", - " raise RuntimeError(\n", + " raise CellStop(\n", " f\"The local checkout {checkout} is not the release this notebook pins: \"\n", " + \"; \".join(differences)\n", " + f\". Check out {RELEASE_REF} with a clean tree, or set DEVELOPMENT_OVERRIDE = True \"\n", @@ -207,9 +217,6 @@ "\n", " Running it again installs nothing that is already there.\n", " \"\"\"\n", - " if sys.version_info < (3, 11):\n", - " raise RuntimeError(\"Select a Python 3.11 or newer kernel, then run this cell again.\")\n", - "\n", " current = Path.cwd().resolve()\n", " nearby = next(\n", " (\n", @@ -229,7 +236,7 @@ " source = checkout / \"src\"\n", " loaded = sys.modules.get(\"morphobricks_preprocessing\")\n", " if loaded is not None and not Path(loaded.__file__).resolve().is_relative_to(source):\n", - " raise RuntimeError(\n", + " raise CellStop(\n", " \"This kernel already loaded another MorphoBricks preprocessing copy. \"\n", " \"Restart the kernel, then run this cell again.\"\n", " )\n", @@ -275,7 +282,7 @@ " check=True,\n", " )\n", " except (subprocess.CalledProcessError, OSError) as exc:\n", - " raise RuntimeError(\n", + " raise CellStop(\n", " \"Notebook setup could not install its requirements. Check the installation \"\n", " \"output and internet access, then run this cell again.\"\n", " ) from exc\n", @@ -283,7 +290,7 @@ "\n", " remaining = _missing_requirements()\n", " if remaining:\n", - " raise RuntimeError(\n", + " raise CellStop(\n", " \"Installation finished, but this kernel still cannot import \"\n", " + \", \".join(remaining)\n", " + \". Restart the kernel and run this cell again.\"\n", @@ -292,8 +299,11 @@ " try:\n", " import_module(\"morphobricks_preprocessing.notebook_ui\")\n", " except ImportError as exc:\n", - " raise RuntimeError(\n", - " \"MorphoBricks preprocessing was found, but its notebook interface could not load.\"\n", + " # The cause stays on the one line: a stop must never hide what failed.\n", + " cause = \" \".join(f\"{type(exc).__name__}: {exc}\".split())\n", + " raise CellStop(\n", + " \"MorphoBricks preprocessing was found, but its notebook interface could not load \"\n", + " f\"({cause}).\"\n", " ) from exc\n", "\n", " package_version = version(\"morphobricks-preprocessing\")\n", @@ -332,6 +342,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "cellView": "form", "collapsed": false, "jupyter": { "source_hidden": true @@ -345,6 +356,14 @@ "# @title 2. Mount Google Drive\n", "import sys\n", "\n", + "\n", + "class CellStop(RuntimeError):\n", + " \"\"\"Stop this cell with its message on one line, without a traceback (Colab and Jupyter).\"\"\"\n", + "\n", + " def _render_traceback_(self) -> list:\n", + " return [str(self)]\n", + "\n", + "\n", "if \"google.colab\" in sys.modules:\n", " from google.colab import drive\n", "\n", @@ -352,8 +371,11 @@ " try:\n", " drive.mount(\"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/content/drive\")\n", " except Exception as exc:\n", - " raise RuntimeError(\n", - " \"Google Drive could not be connected. Allow Drive access, then run this cell again.\"\n", + " # The cause stays on the one line: a stop must never hide what failed.\n", + " cause = \" \".join(f\"{type(exc).__name__}: {exc}\".split())\n", + " raise CellStop(\n", + " f\"Google Drive could not be connected ({cause}). Allow Drive access, then run \"\n", + " \"this cell again.\"\n", " ) from exc\n", " print(\"Google Drive is connected: your files are in /content/drive/MyDrive.\", flush=True)\n", "else:\n", @@ -368,7 +390,7 @@ "\n", "## **3. Run preprocessing**\n", "\n", - "Opens the preprocessing app. Run cells 1 and 2 first: without them this cell stops and says which cell to run.\n", + "Opens the preprocessing app. Run cell 1 first: without it this cell stops with one line naming it. On Google Colab without Drive mounted (cell 2) the app still opens, and a line says that only files on this Colab runtime are visible.\n", "\n", "The app has seven steps, one on screen at a time; move with **Next** or click a step in the strip at the top:\n", "\n", @@ -387,6 +409,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "cellView": "form", "collapsed": false, "jupyter": { "source_hidden": true @@ -402,26 +425,37 @@ "import sys\n", "from pathlib import Path\n", "\n", + "\n", + "class CellStop(RuntimeError):\n", + " \"\"\"Stop this cell with its message on one line, without a traceback (Colab and Jupyter).\"\"\"\n", + "\n", + " def _render_traceback_(self) -> list:\n", + " return [str(self)]\n", + "\n", + "\n", "if not globals().get(\"SETUP_RECORD\"):\n", - " raise RuntimeError(\n", + " raise CellStop(\n", " \"MorphoBricks preprocessing is not installed in this session yet: run cell 1 \"\n", " \"(Install MorphoBricks preprocessing) first, then run this cell again.\"\n", " )\n", "if \"google.colab\" in sys.modules:\n", " if not Path(\"/content/drive/MyDrive\").is_dir():\n", - " raise RuntimeError(\n", - " \"Google Drive is not connected, so the experiment and output folders on Drive \"\n", - " \"cannot be reached: run cell 2 (Mount Google Drive) first, then run this cell again.\"\n", + " print(\n", + " \"Google Drive is not mounted: only files on this Colab runtime are visible, \"\n", + " \"and they are deleted when the runtime stops.\",\n", + " flush=True,\n", " )\n", "\n", "try:\n", " interface = importlib.import_module(\"morphobricks_preprocessing.notebook_ui\")\n", "except ImportError as exc:\n", - " raise RuntimeError(\n", - " \"MorphoBricks preprocessing was found, but its notebook interface could not load. \"\n", - " \"Run cell 1 again.\"\n", + " # The cause stays on the one line: a stop must never hide what failed.\n", + " cause = \" \".join(f\"{type(exc).__name__}: {exc}\".split())\n", + " raise CellStop(\n", + " \"MorphoBricks preprocessing was found, but its notebook interface could not load \"\n", + " f\"({cause}). Run cell 1 again.\"\n", " ) from exc\n", - "notebook = interface.PreprocessingNotebook(describe=True)\n", + "notebook = interface.PreprocessingNotebook()\n", "\n", "import ipywidgets as widgets\n", "from IPython.display import display\n", diff --git a/notebooks/notebook_latest_versions.yaml b/notebooks/notebook_latest_versions.yaml index 5c6fb1b..9232b33 100644 --- a/notebooks/notebook_latest_versions.yaml +++ b/notebooks/notebook_latest_versions.yaml @@ -1,3 +1,3 @@ # Latest version of each notebook, read by LabConstrictor's version check by notebook_name. # Equal to the notebook's current_version and its RELEASE_TAG without the "v" (README, "Notebook"). -"MorphoBricks_Preprocessing": "0.5.0" +"MorphoBricks_Preprocessing": "0.5.1" diff --git a/pyproject.toml b/pyproject.toml index c6b5f14..07b4084 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "morphobricks-preprocessing" -version = "0.5.0" +version = "0.5.1" description = "MorphoBricks preprocessing brick: per-plane intensity histograms and normalization methods for microscopy TIFFs." readme = "README.md" license = "MIT" diff --git a/tests/test_notebook_file.py b/tests/test_notebook_file.py index 56644f5..3d95d52 100644 --- a/tests/test_notebook_file.py +++ b/tests/test_notebook_file.py @@ -74,6 +74,8 @@ def test_notebook_is_an_introduction_and_three_explained_code_cells(): assert code["outputs"] == [] assert code["execution_count"] is None assert code["metadata"].get("jupyter", {}).get("source_hidden") is True + # A Colab form with the code hidden (AGENTS.md section 7, as ZeroCostDL4Mic). + assert code["metadata"].get("cellView") == "form" for cell in cells: if cell["cell_type"] == "markdown": assert "\\n" not in "".join(cell["source"]) @@ -118,6 +120,8 @@ def test_notebook_uses_stable_page_api_not_a_new_app_method(): RELEASE_CALLS = { # v0.5.0: the described run is ``describe=True``; the default is the undescribed v0.4 run. "fd7afd339bcaab7579c806a28a65bb7ecebe940c": "PreprocessingNotebook(describe=True)", + # v0.5.1: every run is described; there is no describe argument. + "d023ff91289a2d2204e1ae56f7dee114d1a665e4": "PreprocessingNotebook()", } CHECKOUT_CALL = "PreprocessingNotebook()" """The call of this checkout's code (every run described, no ``describe`` argument).""" @@ -199,11 +203,12 @@ def test_version_metadata_matches_labconstrictor_index(): assert index == {NOTEBOOK.stem: _setting("current_version")} -def _kernel_notebook(nbformat, *, roles=(INSTALL, DRIVE, RUN), colab_drive=None): +def _kernel_notebook(nbformat, *, roles=(INSTALL, DRIVE, RUN), colab_drive=None, override=True): """Return the notebook's code cells ``roles`` as a notebook ready for this checkout's kernel. With ``colab_drive``, a first cell fakes ``google.colab`` whose ``drive.mount`` makes a - ``MyDrive`` folder, and the cells' ``/content/drive`` is ``colab_drive``. + ``MyDrive`` folder, and the cells' ``/content/drive`` is ``colab_drive``. ``override=False`` + leaves cell 1 as shipped, so it refuses this checkout (not the ``RELEASE_REF`` commit). """ notebook = nbformat.read(NOTEBOOK, as_version=4) code = [cell for cell in notebook.cells if cell.cell_type == "code"] @@ -211,9 +216,10 @@ def _kernel_notebook(nbformat, *, roles=(INSTALL, DRIVE, RUN), colab_drive=None) # The kernel runs in this checkout, whose commit is not RELEASE_REF (#103): the one edit a # developer makes to use a checkout is the explicit override. assert install.source.count("DEVELOPMENT_OVERRIDE = False") == 1 - install.source = install.source.replace( - "DEVELOPMENT_OVERRIDE = False", "DEVELOPMENT_OVERRIDE = True" - ) + if override: + install.source = install.source.replace( + "DEVELOPMENT_OVERRIDE = False", "DEVELOPMENT_OVERRIDE = True" + ) # While RELEASE_REF is behind this checkout, the cell calls the released API: the kernel, # running this checkout, is given this checkout's call (the release PR makes them one). assert run.source.count(release_call()) == 1 @@ -306,6 +312,84 @@ def test_the_run_cell_alone_in_a_fresh_kernel_names_cell_1(): _execute(notebook, allow_errors=True) (error,) = _errors(run) - assert error.ename == "RuntimeError" # not an ImportError of the package + assert error.ename == "CellStop" # the cell's own stop, not an ImportError of the package assert "run cell 1 (Install MorphoBricks preprocessing) first" in error.evalue + # One line, without a traceback (owner, v0.5.1): what Colab and JupyterLab show. + assert error.traceback == [error.evalue] assert not _shows_a_widget(run) + + +def test_a_refusal_of_cell_1_in_a_kernel_is_one_line_without_a_traceback(): + nbformat = _kernel_modules() + # As shipped, cell 1 refuses this checkout: its commit is not RELEASE_REF (#103). + notebook, (install,) = _kernel_notebook(nbformat, roles=(INSTALL,), override=False) + _execute(notebook, allow_errors=True) + + (error,) = _errors(install) + assert error.ename == "CellStop" + assert "is not the release this notebook pins" in error.evalue + assert error.traceback == [error.evalue] + assert "\n" not in error.evalue + + +def test_on_a_faked_colab_without_drive_cell_3_opens_the_app_after_one_line(tmp_path): + nbformat = _kernel_modules() + drive_root = tmp_path / "content/drive" + notebook, (install, run) = _kernel_notebook( + nbformat, roles=(INSTALL, RUN), colab_drive=drive_root + ) + _execute(notebook) + + assert not [*_errors(install), *_errors(run)] + assert _text(run).splitlines() == [ + "Google Drive is not mounted: only files on this Colab runtime are visible, " + "and they are deleted when the runtime stops." + ] + assert not drive_root.exists() # nothing mounted for the user + assert _shows_a_widget(run) + + +def _cell_after(nbformat, notebook, source): + """Put a cell running ``source`` in front of the notebook's last cell.""" + notebook.cells.insert(len(notebook.cells) - 1, nbformat.v4.new_code_cell(source)) + + +def test_an_unexpected_error_in_a_cell_keeps_its_traceback(): + # Only the cells' own stops are one line: a real error still shows where it happened. + nbformat = _kernel_modules() + notebook, (install, run) = _kernel_notebook(nbformat, roles=(INSTALL, RUN)) + _cell_after( + nbformat, + notebook, + "import morphobricks_preprocessing.notebook_ui as interface\n" + "def broken(*arguments, **options):\n" + " raise KeyError('a real defect')\n" + "interface.PreprocessingNotebook = broken\n", + ) + _execute(notebook, allow_errors=True) + + assert not _errors(install) + (error,) = _errors(run) + assert error.ename == "KeyError" + assert len(error.traceback) > 1 + assert any("broken" in line for line in error.traceback) + + +def test_the_stop_for_an_interface_that_cannot_load_names_the_import_error(): + # The one-line stop keeps the cause on its line: it never hides what failed. + nbformat = _kernel_modules() + notebook, (install, run) = _kernel_notebook(nbformat, roles=(INSTALL, RUN)) + _cell_after( + nbformat, + notebook, + "import sys\nsys.modules['morphobricks_preprocessing.notebook_ui'] = None\n", + ) + _execute(notebook, allow_errors=True) + + assert not _errors(install) + (error,) = _errors(run) + assert error.ename == "CellStop" + assert "could not load (ModuleNotFoundError: import of " in error.evalue + assert "morphobricks_preprocessing.notebook_ui" in error.evalue + assert error.evalue.endswith("Run cell 1 again.") + assert error.traceback == [error.evalue] diff --git a/tests/test_notebook_setup.py b/tests/test_notebook_setup.py index 27b3071..87f0ea2 100644 --- a/tests/test_notebook_setup.py +++ b/tests/test_notebook_setup.py @@ -105,8 +105,8 @@ def setup_environment(tmp_path, monkeypatch): drive=tmp_path / "content/drive", namespace={}, versions={ - "morphobricks-preprocessing": "0.5.0", - "morphobricks-normalization": "0.3.4", + "morphobricks-preprocessing": "0.5.1", + "morphobricks-normalization": "0.4.0", "ipywidgets": "7.7.1", }, ) @@ -131,7 +131,7 @@ def install(command, *, check): assert check is True state.installs.append(command) state.available.update({*REQUIRED, ZARR}) - state.versions["morphobricks-preprocessing"] = "0.5.0" + state.versions["morphobricks-preprocessing"] = "0.5.1" monkeypatch.setattr(subprocess, "run", install) @@ -143,7 +143,7 @@ def import_interface(name): import ipywidgets as widgets class FakeNotebook: - def __init__(self, **options): # describe=True, the shipped notebook's run + def __init__(self, **options): # no options: every run is described (v0.5.1) self.options = options state.apps.append(options) @@ -163,7 +163,7 @@ def checkout_at(path): package.mkdir(parents=True) (package / "__init__.py").write_text("", encoding="utf-8") (path / "pyproject.toml").write_text( - '[project]\nname = "morphobricks-preprocessing"\nversion = "0.5.0"\n', + '[project]\nname = "morphobricks-preprocessing"\nversion = "0.5.1"\n', encoding="utf-8", ) return path @@ -179,7 +179,7 @@ def test_setup_reuses_the_matching_installed_release(setup_environment, capsys): "morphobricks_preprocessing.notebook_ui", # cell 3 shows it ] out = capsys.readouterr().out - assert "Ready: preprocessing 0.5.0" in out + assert "Ready: preprocessing 0.5.1" in out assert "ipywidgets 7.7.1" in out @@ -193,8 +193,8 @@ def test_setup_installs_missing_package_from_the_exact_commit(setup_environment, target = command[5] assert target.startswith("morphobricks-preprocessing[notebook] @ git+") assert target.endswith("@" + namespace["RELEASE_REF"]) - assert namespace["RELEASE_REF"] == "fd7afd339bcaab7579c806a28a65bb7ecebe940c" - assert "pinned v0.5.0 candidate" in capsys.readouterr().out + assert namespace["RELEASE_REF"] == "d023ff91289a2d2204e1ae56f7dee114d1a665e4" + assert "pinned v0.5.1 candidate" in capsys.readouterr().out def test_setup_replaces_a_mismatched_installed_release(setup_environment): @@ -285,7 +285,7 @@ def test_an_explicit_development_override_uses_the_checkout_and_says_so( assert sys.path[0] == str(checkout / "src") assert f"commit {OTHER_COMMIT}, uncommitted changes" in out assert "DEVELOPMENT OVERRIDE is on" in out - assert "DEVELOPMENT OVERRIDE on" in out.split("Notebook 0.5.0")[1] + assert "DEVELOPMENT OVERRIDE on" in out.split("Notebook 0.5.1")[1] record = namespace["SETUP_RECORD"] assert record["checkout_commit"] == OTHER_COMMIT assert record["checkout_has_uncommitted_changes"] is True @@ -380,7 +380,9 @@ def refuse(path): state = setup_environment state.start(INSTALL) imported = list(state.imports) - with pytest.raises(RuntimeError, match="Google Drive could not be connected"): + with pytest.raises( + RuntimeError, match=r"Google Drive could not be connected \(ValueError: mount failed\)\." + ): state.start(DRIVE, fresh=False) assert state.imports == imported assert state.apps == [] @@ -425,7 +427,7 @@ def test_cell_1_installs_without_mounting_drive_or_showing_the_app( assert state.apps == [] assert namespace["SETUP_RECORD"]["release_ref"] == RELEASE_REF out = capsys.readouterr().out - assert "Ready: preprocessing 0.5.0; normalization 0.3.4; ipywidgets 7.7.1." in out + assert "Ready: preprocessing 0.5.1; normalization 0.4.0; ipywidgets 7.7.1." in out assert "Next: run cell 2 (Mount Google Drive), then cell 3 (Run preprocessing)." in out @@ -441,8 +443,8 @@ def test_running_cell_1_again_installs_nothing_more(setup_environment, capsys): assert len(state.installs) == 1 assert "Installing notebook requirements" in first assert "Installing notebook requirements" not in again - assert "Ready: preprocessing 0.5.0" in again - assert state.namespace["SETUP_RECORD"]["preprocessing_version"] == "0.5.0" + assert "Ready: preprocessing 0.5.1" in again + assert state.namespace["SETUP_RECORD"]["preprocessing_version"] == "0.5.1" assert record["release_ref"] == state.namespace["SETUP_RECORD"]["release_ref"] @@ -486,18 +488,80 @@ def test_cell_3_after_a_failed_cell_1_still_names_cell_1(setup_environment, monk assert state.apps == [] -def test_cell_3_on_colab_without_drive_names_cell_2(setup_environment, monkeypatch): +NO_DRIVE = ( + "Google Drive is not mounted: only files on this Colab runtime are visible, " + "and they are deleted when the runtime stops." +) +"""Cell 3's one line on Colab without Drive (owner, v0.5.1: the app opens anyway).""" + + +def test_cell_3_on_colab_without_drive_opens_the_app_after_one_line( + setup_environment, monkeypatch, capsys +): calls = colab(monkeypatch) state = setup_environment state.start(INSTALL) - imported = list(state.imports) - with pytest.raises(RuntimeError, match=r"Google Drive is not connected.*run cell 2"): - state.start(RUN, fresh=False) + capsys.readouterr() + state.start(RUN, fresh=False) + printed = capsys.readouterr().out.splitlines() + assert printed[0] == NO_DRIVE # one line, then the app (``display`` prints it here) + assert [line for line in printed if "Drive" in line] == [NO_DRIVE] assert calls == [] # never mounted for the user - assert state.imports == imported - assert state.apps == [] - state.start(DRIVE, RUN, fresh=False) assert state.apps == [APP_OPTIONS] + # With Drive mounted the line is not printed. + state.start(DRIVE, fresh=False) + capsys.readouterr() + state.start(RUN, fresh=False) + assert NO_DRIVE not in capsys.readouterr().out + assert state.apps == [APP_OPTIONS, APP_OPTIONS] + + +def test_cell_3_outside_colab_never_prints_the_drive_line(setup_environment, capsys): + setup_environment.start(INSTALL, RUN) + assert NO_DRIVE not in capsys.readouterr().out + + +# A cell that cannot go on stops with one line and no traceback (owner, v0.5.1): it raises the +# cell's own CellStop, a RuntimeError whose ``_render_traceback_`` is its message alone (IPython, +# and so Colab and JupyterLab, shows that instead of the traceback; tests/test_notebook_file.py +# runs it in a real kernel). + + +def _cell_tree(number): + return ast.parse(cell_source(number)) + + +@pytest.mark.parametrize("number", [INSTALL, DRIVE, RUN]) +def test_every_stop_of_a_cell_is_its_one_line_cell_stop(number): + tree = _cell_tree(number) + (stop,) = [node for node in tree.body if isinstance(node, ast.ClassDef)] + assert stop.name == "CellStop" + assert [ast.unparse(base) for base in stop.bases] == ["RuntimeError"] + namespace = {} + exec(compile(ast.Module(body=[stop], type_ignores=[]), "cell", "exec"), namespace) + error = namespace["CellStop"]("one clear message") + assert error._render_traceback_() == ["one clear message"] + raised = [node.exc for node in ast.walk(tree) if isinstance(node, ast.Raise)] + assert raised + for exc in raised: + assert isinstance(exc, ast.Call) and ast.unparse(exc.func) == "CellStop", ast.unparse(exc) + + +def test_cell_1_checks_the_python_version_before_importing_tomllib(): + # tomllib is Python 3.11: on 3.10 an import before the check would be a ModuleNotFoundError + # in place of the "Python 3.11 or newer" line. + tree = _cell_tree(INSTALL) + check = next( + position + for position, node in enumerate(tree.body) + if isinstance(node, ast.If) and "sys.version_info" in ast.unparse(node.test) + ) + for position, node in enumerate(tree.body[:check]): + assert isinstance(node, (ast.Import, ast.ImportFrom, ast.ClassDef, ast.Expr)), position + if isinstance(node, (ast.Import, ast.ImportFrom)): + assert {alias.name for alias in node.names} <= {"platform", "subprocess", "sys"} + for node in ast.walk(ast.Module(body=tree.body[:check], type_ignores=[])): + assert not (isinstance(node, ast.Import) and "tomllib" in ast.unparse(node)) def test_cell_3_outside_colab_needs_no_drive(setup_environment): @@ -505,3 +569,30 @@ def test_cell_3_outside_colab_needs_no_drive(setup_environment): state.start(INSTALL, RUN) # cell 2 skipped: nothing to mount outside Colab assert state.apps == [APP_OPTIONS] assert "app" in state.namespace + + +@pytest.mark.parametrize("number", [INSTALL, RUN]) +def test_the_interface_stop_names_its_import_error(setup_environment, monkeypatch, number): + # A stop is one line, but it never hides what failed: the ImportError is on that line. + state = setup_environment + if number == RUN: + state.start(INSTALL) + imported = importlib.import_module # the fixture's double + + def broken(name): + if name == "morphobricks_preprocessing.notebook_ui": + raise ImportError("cannot import name 'gone' from 'morphobricks_preprocessing.x'") + return imported(name) + + monkeypatch.setattr(importlib, "import_module", broken) + with pytest.raises(RuntimeError) as stopped: + state.start(number, fresh=number == INSTALL) + stop = stopped.value + assert type(stop).__name__ == "CellStop" + assert ( + "its notebook interface could not load (ImportError: cannot import name 'gone' from " + "'morphobricks_preprocessing.x')." in str(stop) + ) + assert stop._render_traceback_() == [str(stop)] + assert isinstance(stop.__cause__, ImportError) + assert state.apps == []