[guijacquemet]

PP-8d makes notebooks/MorphoBricks_Preprocessing.ipynb run on Google Colab against a full dataset on Google Drive, following LabConstrictor's notebook portability guide, code-hiding guide and the version-check convention of its notebook upload guide. The reference was fiberfeatures' notebooks and tests/test_notebooks.py (at 362f1ba). Based on main 501c2dc, then main d35db1e (PP-8b, roadmap) and, in review, main a4e30a7 (PP-8c, #117) merged in. No rebase. Version stays 0.3.4.

Summary

Notebook (notebooks/MorphoBricks_Preprocessing.ipynb)

• Every code cell starts with # @title Run to …. They were # @markdown ### … before. All code cells stay hidden (see the decisions below).
• Step 0, Colab only, inside if "google.colab" in sys.modules::
  • it mounts Google Drive with drive.mount("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/content/drive") and prints a message first;
  • a refused mount stops with a clear error;
  • it calls output.enable_custom_widget_manager().
• Step 0, locally: it prints Python, the platform and the working folder.
• No magics. pip runs through subprocess with the kernel's Python.
• The GitHub installation is pinned to RELEASE_TAG = "v0.4.0rc2", one constant, through git+{REPOSITORY}@{RELEASE_TAG}. It was main before.
• Step 0 prints the source it uses: the local checkout, "the installed package, version X" or "release v0.4.0rc2 from GitHub". It also prints notebook_name current_version.
• current_version = "0.4.0rc2" (was 0.0.1) and notebook_name = "MorphoBricks_Preprocessing".
• On Colab, Step 0 installs the notebook and zarr extras (review, owner decision); locally notebook, unchanged. On Colab zarr is a checked requirement, so an installed package without it gets the extra.
• ipywidgets 7, which Colab preinstalls, is detected:
  • if only its distribution is old, the notebook extra is reinstalled;
  • if it is already loaded in the kernel, Step 0 asks for a restart ("Runtime › Restart session").
• Colab users / JupyterLab users callouts at Steps 0 and 1:
  • data and output on Drive under /content/drive/MyDrive/…;
  • the output folder kept outside the experiment;
  • data kept at the same place on Drive every session (no copying to the runtime's disk; review);
  • one session per output folder on Drive (review);
  • the runtime is wiped on disconnect;
  • how to copy a path from the Files panel.
• "Common problems" gains output_locked after a disconnect, with the remove_stale_lock cell.
• Adjacent fix in the same file: the Step 3 text and the "After running" item still described the TIFF→OME-Zarr conversion card, which v0.4.0 removed (plan Q46).
• The kernelspec display name changed from "morpho_pre" to "Python 3".

New: notebooks/notebook_latest_versions.yaml. The index LabConstrictor's version check reads, keyed by notebook_name, as in fiberfeatures.

notebook_style.no_dialog_status(dialog, target) (new, ui layer)

• When Browse… cannot open a dialog, as on Colab, the warning now also says how to copy a path on Colab.
• notebook_ui._browse and _browse_csv use it. notebook_ui.py changes only in those two calls, and lost 8 lines.

Tests

• New tests/test_notebook_file.py. The notebook-file tests moved there from test_notebook_ui.py. New tests:
  • every code cell is titled # @title and hidden with the standard metadata;
  • no magics or get_ipython;
  • google.colab imports and /content appear only inside the single top-level Colab guard, checked by AST line spans;
  • the guard mounts Drive and enables the widget manager;
  • RELEASE_TAG matches v\d+.\d+.\d+(rcN)? and is the only git+ install;
  • current_version equals the tag without "v", the yaml index and the tag named in the Step 0 callout;
  • notebook_name equals the file name;
  • headless execution of the whole notebook in a real kernel (nbclient, cwd notebooks/): no error outputs, the local environment report, every code cell shows a widget. It skips with a reason when nbclient or ipykernel is missing. Both are added to the dev extra, so CI runs it.
• tests/test_notebook_setup.py:
  • Colab mount and widget manager (faked google.colab);
  • a refused mount;
  • the local report;
  • the pinned URL and the printed tag and version;
  • the ipywidgets 7 upgrade and restart cases;
  • (review) the zarr extra on Colab only.
• tests/test_notebook_ui.py: the Colab hint in both Browse fallbacks, and a quoted Colab "Copy path" path for clean_path.

Docs

• README: a new section, "Notebook: Google Colab and JupyterLab".
• User guide §1: a new "Google Colab" section, and the pin.
• Plan:
  • question 58, the decision record (renumbered in review after main's PP-8c questions 52–57; every reference updated), with the owner's answers to the questions below;
  • a new "Release v0.4.0 (PP-13) checklist" in §10, which did not exist yet. It holds the pyproject version, the notebook pin (four places, checked by a test) and the Colab hand check;
  • a milestone 5 note;
  • (review) a PP-10 note: a button that removes the lock of a stopped run.
• AGENTS.md: the "Notebook target" row (was "Local Jupyter (not Colab-first)"), the §6 and §7 notebook rules and the repo map.
• development.md: one paragraph.
• There is no notebook CHANGELOG in this repository.

Decisions

• Hidden code. All code cells keep their source hidden. This is an app-like widget notebook in which no code is edited, the case LabConstrictor's code-hiding guide describes. fiberfeatures differs because its settings cells are edited. The test enforces "every code cell titled and hidden".
• Version and pin. current_version is the release tag without "v". This gives one bump per release, and the test ties the two together. The pin moves in the release PR before the tag is pushed, so a tagged notebook installs its own tag. Until v0.4.0rc2 exists, a fresh Colab install from this notebook fails at pip: this is expected, and the owner tags rc2 after PP-8c and this PR.
• Custom widget manager. It is enabled on Colab. Every widget is a core ipywidgets 8 control (HTML, Button, Text, Dropdown, Accordion, Image, IntProgress and so on). No third-party widget or ipympl is used. The package needs ipywidgets ≥ 8, and Colab preinstalls 7. LabConstrictor's own version-check cell makes the same call on Colab.
• Browse…. Browse… is a tkinter dialog on the kernel's computer. On Colab there is no display, so Tk() raises TclError and the fallback message appears. The path boxes take any kernel path. clean_path keeps /content/drive/MyDrive/… unchanged, which is tested.

Notes for review: Google Drive (code read, run semantics unchanged)

• Lock (run/lock.py, O_CREAT|O_EXCL). It is exclusive within one runtime's FUSE mount. Drive syncs files between machines with a delay, so a second Colab session, or a PC syncing the same Drive folder, can take the "same" lock at the same time. Owner: documented only. The user guide, README and notebook say plainly that on Drive the lock cannot guard an output folder across machines or sessions, so use one session per output folder.
• Stale lock after a disconnect. The lock file stays on Drive, and the next check reports output_locked. The notebook has no button for it, so the guide and "Common problems" give a remove_stale_lock(...) cell (a button comes in PP-10). A new runtime has another host name, so remove_stale_lock removes it once asked.
• Fingerprints (size + st_mtime_ns). Drive reports times at about 1 ms resolution. If Drive reports a different time later, for example after a resync, the image is read again: that is safe. Reuse after a size-and-time-preserving overwrite is the general risk, not specific to Drive.
• RowKey contains the absolute read path. Reading the images from another place recomputes every row, so the docs no longer suggest copying to the runtime's disk (owner): data and output stay on Drive.
• Atomic writes (os.replace of a temporary file, staged publishing with backups). They should work on Drive FUSE, but I could not test this. It is on the hand-check list.

I could not run Google Colab. Everything above was verified only by reading the code and by local tests: the notebook was executed in a local kernel and with a faked google.colab.

Colab hand check (owner, before tagging v0.4.0rc2)

1. On a fresh runtime, open the notebook from the branch or tag and run Step 0.
   • The Drive authorisation window appears and the mount succeeds.
   • Step 0 prints "Using release v0.4.0rc2 from GitHub" (after the tag exists) and installs morphobricks-preprocessing[notebook,zarr].
   • The green ready card shows the expected package version.
   • No "Restart session" request appears. If one does, restart and run Step 0 again, then note it.
2. Run Steps 1–4. Every widget renders and reacts: buttons, dropdowns, accordions, preview image, sliders, progress bar and matplotlib plots. If anything is blank, try without enable_custom_widget_manager() and report it.
3. Step 1:
   • Browse… shows the "could not be opened … Copy path" message.
   • A pasted /content/drive/MyDrive/… path checks fine.
   • The default output folder lands on Drive next to the experiment.
4. Step 3 on the full dataset:
   • The run finishes.
   • On Drive, the output has plane_histograms.csv, run_manifest.json, experiment_design.csv and the report.
   • There are no "(1)" duplicate files and no leftover .run-* / .tmp files.
   • There is no lock file after the run.
5. Rerun Step 3 in the same session. Expect: everything unchanged, nothing read.
6. Disconnect the runtime (Runtime › Disconnect and delete runtime) and open a new one. Run Steps 0–3 again on the same Drive folders. Expect: rows reused (unchanged); this shows the Drive mtimes are stable across mounts.
7. Optional: stop a run mid-way (interrupt or disconnect). Check that the next check reports output_locked and that the remove_stale_lock cell from "Common problems" clears it.
8. Optional: an experiment with OME-Zarr images on Drive checks and runs (the zarr extra is installed).

Owner decisions (2 November 2026, applied in review)

1. Lock on Drive: document only. One session per output folder; the lock cannot guard across machines or sessions on Drive. Said plainly in the user guide, README and the Step 1 Colab callout.
2. Stale-lock button: later, in PP-10 (plan §10, PP-10 notes). The documented code cell stays for now.
3. Copy to local disk: dropped from callouts, the user guide, README and the hand-check list. Data and output stay on Drive.
4. OME-Zarr on Colab: Step 0 installs [notebook,zarr] on Colab (locally unchanged). The manual pip line is gone. Tests and docs are updated.
5. RELEASE_TAG = "v0.4.0rc2": the owner publishes v0.4.0rc2 after both PRs merge.

Review fixes

• 4013f01: merge main (PP-8c). Plan question renumbered 52 → 58, with all references (AGENTS ×2, plan §10 ×2, development.md, test docstring).
• 823a7ca: Colab installs the zarr extra, with 3 new tests; the Colab callouts updated. Also fixes a test-isolation bug. tests/test_notebook_setup.py failed 15 of 17 when run on its own, because Step 0's platform.platform() runs uname through the faked subprocess.run. It passed only after another test had filled platform's cache. The fixture now fills the cache first.
• 40376f5: docs for the owner decisions (user guide, README, AGENTS, plan Q58 and the PP-10 note).

Tests

• After review (40376f5, with PP-8c merged), .[dev,notebook,zarr], Python 3.11:
  • pytest: 1588 passed, 1 skipped;
  • ruff check and ruff format --check: clean;
  • tools/check.py --rules: exit 0.
• Before review, after the merge of main: pytest with .[dev,notebook,zarr] (+ nbclient, ipykernel), Python 3.11: 1532 passed, 1 skipped.
• The skip is test_discovery.py:665 ("running with permissions that ignore file modes": the sandbox runs as root). The kernel execution test ran and did not skip.
• CI on f766c1b: all green (lint; tests on Ubuntu, macOS and Windows × 3.11/3.13). No notebook test skipped anywhere; the kernel execution test ran on all six jobs.

§9 figures (measured)

• New notebook_style.no_dialog_status: complexity 1, 2 parameters, 1 statement (11 lines with the call), fully annotated, NumPy docstring. notebook_style.py: 386 → 417 lines (≤ 600), layer ui.
• notebook_ui.py: 3051 → 3043 lines. The baseline shrank, and the _browse_csv entry (11 lines without a docstring) left it.
• tests/test_notebook_ui.py: 1604 → 1540 lines, baseline shrunk.
• New tests/test_notebook_file.py: 199 lines. tests/test_notebook_setup.py: 215 → 317 lines after review.
• Baseline: 3 changes against main, all shrinking (one entry removed, two lowered). No entry grew. # noqa total 88 on main and on head; no new exemptions.
• No new modules or layers, and no duplicated helper. The notebook cell's own functions are annotated and docstringed, and ruff lints the notebook.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc

[guijacquemet]

Independent review: merged (merge commit 44971bc)

Review fixes

• 4013f01, merge main (PP-8c, PP-8c: fixes from running on real data (file modes, absent-axis sizes, masks inputs, stopped-run text, read arguments, design command) #117). The conflict was in plan §12. Main's questions 52–57 are kept, and this PR's question is renumbered 58, with every reference updated: AGENTS ×2, plan §10 ×2, development.md and the test docstring. Main's own Q52 references (file permissions) are untouched.
• 823a7ca, owner decision: on Colab, Step 0 installs [notebook,zarr].
  • Locally it still installs [notebook].
  • On Colab, zarr is a checked requirement, so an installed package without it also gets the extra.
  • 3 new tests: the GitHub release with [notebook,zarr] on Colab; zarr added to an installed package on Colab; locally, zarr is neither required nor installed.
  • The Colab callouts no longer advise copying to the runtime's disk. They say plainly: one session per output folder.
• 823a7ca, test-isolation bug. tests/test_notebook_setup.py failed 15 of 17 when run on its own.
  • Cause: Step 0 calls platform.platform(), which runs uname through subprocess the first time, and the fixture fakes subprocess.run for pip.
  • It passed in the full suite only because another test had filled platform's cache first.
  • Fix: the fixture fills that cache before faking pip.
• 40376f5, docs for the owner decisions.
  • The lock on Drive is documented only: one session per output folder, because the lock cannot guard across machines or sessions on Drive.
  • The copy-to-local advice is removed from the callouts, user guide, README and hand-check list (old item 8 is now an optional OME-Zarr check).
  • The manual zarr pip line is gone.
  • Plan Q58 records the decisions, and plan §10 gains a PP-10 note for a remove-the-lock button.

Verified

• Every code cell starts with # @title Run to … and is hidden.
• Colab-only code (the google.colab import, drive.mount("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/content/drive"), enable_custom_widget_manager(), EXTRAS = "notebook,zarr") and every /content path are only inside the single top-level guard (AST test). There are no magics and no get_ipython.
• The ipywidgets 7 path cannot loop: the distribution is upgraded through the package's ipywidgets>=8 requirement and re-checked. A loaded 7 raises "Restart session"; after the restart nothing is reinstalled, because the requirements are met.
• The pinned git+{REPOSITORY}@{RELEASE_TAG} is the only git+ install. v0.4.0rc2 matches the tag pattern. current_version 0.4.0rc2, notebook_name, notebook_latest_versions.yaml and the Step 0 callout tag agree (tested).
• Embedding API stays compatible:
  • MorphoRelate (src/morphorelate/notebook_steps.py) imports notebook_style as ui and PreprocessingNotebook.
  • This PR only adds notebook_style.no_dialog_status, and changes two private Browse fallbacks.
• Headless kernel test really runs in CI.
  • Locally: 10 of 10 in test_notebook_file.py, with no skip.
  • CI on 40376f5, Ubuntu 3.13: 1589 passed, 0 skipped, which includes test_notebook_runs_in_a_kernel.

Gate (own worktree and venv, .[dev,notebook,zarr], Python 3.11, at 40376f5 with PP-8c merged):

• ruff check and ruff format --check: clean.
• tools/check.py --rules: exit 0.
• pytest: 1588 passed, 1 skipped. The skip is test_discovery.py:665, root, as expected.
• CI: all 7 jobs green.

§9 figures

Measure	Value	Where
Largest touched non-baselined source file	417 lines	notebook_style.py
notebook_ui.py (baselined)	3051 → 3043 lines
New function	complexity 1, 2 parameters, 1 statement	no_dialog_status
Largest test files touched	317 lines	test_notebook_setup.py
	199 lines	test_notebook_file.py

• The baseline only shrinks: the _browse_csv entry was removed, and two line maxima were lowered.
• # noqa total: 88 on main and 88 on head. No new exemptions; no new modules or layers.

Branch: it was auto-deleted on merge and restored at 40376f5.

Not verified: a real Colab run. That is the hand-check list in the PR description, for the owner before tagging v0.4.0rc2.

Follow-up (not blocking): spec §9 calls PreprocessingNotebook/notebook_style a tested embedding API, but there is no dedicated contract test of the names MorphoRelate uses.

---

Generated by Claude Code