PP-8d: Colab and LabConstrictor readiness of the notebook

AGENTS.md

@@ -152,6 +152,7 @@ src/morphobricks_preprocessing/
                     show_output and show_design for step 1) so other bricks can embed them
                     (plan §12 Q26, Q31)
 notebooks/MorphoBricks_Preprocessing.ipynb   thin notebook: every code cell calls notebook_ui only
+notebooks/notebook_latest_versions.yaml      the notebook's latest version, for LabConstrictor's version check
 tests/              pytest with simulated TIFFs (tests/tiff_tree.py, tests/simulation.py); the
                     schema-version-2 fixture (tests/v2_fixture.py, tests/data)
 experiments/binning_study/      the study behind the binning decisions (not part of the package)
@@ -514,7 +515,7 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
 | RGB `S` axis (Q3) | Prefilled as C, confirmed like any role |
 | Significant bits (Q4) | Manifest only, information only; `bit_depth` and binning unchanged |
 | Corrupt files (Q6) | Explicit exclusion list, recorded in manifest, rows removed |
-| Notebook target | Local Jupyter (not Colab-first) |
+| Notebook target | Google Colab and JupyterLab from one file, following LabConstrictor's notebook portability guide (plan Q58; was local Jupyter only): `# @title` cells, Colab-only code (Drive mount, custom widget manager) inside `if "google.colab" in sys.modules:`, no magics, GitHub install pinned to `RELEASE_TAG` |
 | CLI | `check <folder>` (summary, label tree, issues, suggested `--axes`) and `run <folder> --axes ROLES` with `--output`, `--exclude`, `--bins`, `--histogram-range`, `--robust-quantiles`, `--memory-limit 2GiB`, `--missing-as-singleton`, `--mode restart|full`, `--no-masks`, `--no-exclude`, `--remove-stale-lock`, `--log`, and the conversion options above. Axis roles are always stated explicitly, never taken silently from metadata. Exit 1 on errors |
 | Large planes (Q15) | Planes over the memory limit are read in blocks automatically, announced by the warning `planes_read_in_blocks` and `read_in_blocks` in the manifest. Block size derived from the limit (whole strips/tiles/chunks, uncompressed pieces and chunks also by rows). Compressed single-strip/tile planes keep the `plane_over_memory_limit` error. Two-pass histograms identical to whole planes. (Up to v0.3, OME-Zarr copies were written block by block; conversion was removed in v0.4.0.) Axis preview reduced (every n-th pixel) with a caption saying so |
 | Normalization brick (P1) | The modules `dtypes`, `histogram`, `records`, `csv_io`, `pooling` and `normalization` moved unchanged to `morphobricks_normalization` (normalization plan N1, N2, N6) and are imported from there. Up to v0.3 their public names stayed importable from `morphobricks_preprocessing` and the old submodules, with a `DeprecationWarning`; those stand-ins (`_moved.py`) were removed in v0.4.0 (plan Q46). The package's own code never uses the old paths. Outputs unchanged (CSV schema 2); the manifest's `versions` adds `morphobricks-normalization`; Step 0 checks for it; contract §3–§7 text moved to the normalization contract (§1.2, now §6.2 for the CSV) |
@@ -599,10 +600,14 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
   tests skip without zarr; CI's Python 3.11 jobs pin zarr 2.18 and 3.13 jobs get zarr 3.
 - **The notebook file is tested too:**
   - valid nbformat 4, with no outputs;
-  - code cells hidden and titled `# @markdown ### Run to …`;
+  - code cells hidden and titled `# @title Run to …`;
+  - Colab-only code and `/content` paths only inside the Colab guard; no magics;
+  - the GitHub install pinned to `RELEASE_TAG`, equal to `current_version` and the
+    version index;
   - Step 0 bootstraps the environment; later code cells only call `notebook_ui`;
   - the buttons it names exist;
-  - its cells execute.
+  - its cells execute, by `exec` and in a real kernel (`nbclient`, `tests/test_notebook_file.py`;
+    Step 0's alternatives in `tests/test_notebook_setup.py`).
 
 ---
 
@@ -627,12 +632,22 @@ plainer version "not interactive enough". Descriptive, guided and forgiving is t
 - **Code cells:**
   - Hidden, with metadata
     `{"cellView": "form", "collapsed": false, "jupyter": {"source_hidden": true}, "jupyterlabHideCode": {"locked": true}}`.
-  - The first line is a Colab-style title: `# @markdown ### Run to …`.
+  - The first line is a Colab title: `# @title Run to …` (LabConstrictor portability
+    guide; every code cell, all hidden, plan Q58). Longer explanations go in Markdown cells.
   - Outputs are stripped.
   - Step 0 is a standalone standard-library bootstrap: nearby checkout, installed package,
-    or GitHub installation, with missing requirements installed using the kernel's Python.
+    or GitHub installation of the release `RELEASE_TAG`, with missing requirements
+    installed using the kernel's Python (`subprocess`, never `!pip`); on Colab with the
+    `zarr` extra too. Colab-only code
+    (Drive mount, `output.enable_custom_widget_manager()`) stays inside
+    `if "google.colab" in sys.modules:`; no `/content` path outside it. `current_version`
+    and `notebook_name` are LabConstrictor's names; at each release the pin moves
+    (plan §10, release checklist).
   - Later cells are one call each, e.g. `notebook.folder_step()`.
 - **Step 0** starts the notebook and shows a green "ready" card with the package version.
+- **Runtime callouts:** Steps 0 and 1 carry "**Colab users:**" / "**JupyterLab users:**"
+  blockquotes (Drive, output folder outside the experiment, one session per output folder
+  on Drive, runtime wiped on disconnect, Browse… only locally).
 - **Closing cell:** "✅ After running" checklist and "Common problems" that quote the real
   message texts (check they still match the code).
 - **Text:** use bold, not inline code, in the markdown (fenced text blocks such as the

README.md

@@ -82,6 +82,36 @@ The notebook's Step 0 can set this up automatically: it uses a nearby checkout o
 installed package, and installs missing requirements into the active kernel. A standalone
 download installs from GitHub if needed (internet access and Git required).
 
+### Notebook: Google Colab and JupyterLab
+
+[`notebooks/MorphoBricks_Preprocessing.ipynb`](notebooks/MorphoBricks_Preprocessing.ipynb)
+runs unchanged in Google Colab and in JupyterLab (also as a LabConstrictor app). It follows
+LabConstrictor's
+[notebook portability guide](https://github.com/CellMigrationLab/LabConstrictor/blob/main/.tools/docs/notebook_portability.md):
+
+1. Every code cell starts with `# @title …` (Colab's cell header) and is hidden: the notebook is
+   driven by widgets, so no code is edited. Explanations are in Markdown cells.
+2. Colab-only setup runs in Step 0 inside `if "google.colab" in sys.modules:`: it mounts
+   Google Drive at `/content/drive` and enables Colab's custom widget manager (the notebook
+   needs ipywidgets 8). Locally Step 0 prints the Python version, platform and working folder.
+   No shell or line magics are used; pip runs with the kernel's Python.
+3. When the package is missing (always on a fresh Colab runtime), Step 0 installs the release
+   named by `RELEASE_TAG` (currently `v0.4.0rc2`), never the moving `main` branch, and prints
+   which source and version it uses. A checkout around the notebook, or an installed package,
+   is used first.
+4. `current_version` and `notebook_name` in Step 0 are the names LabConstrictor's version check
+   reads; [`notebooks/notebook_latest_versions.yaml`](notebooks/notebook_latest_versions.yaml)
+   lists the latest version. At each release, `RELEASE_TAG`, `current_version`, the index and
+   the tag named in the Step 0 Colab callout move to the new tag before it is tagged
+   ([plan](docs/plan.md) §10, release checklist); `tests/test_notebook_file.py` checks that
+   they agree, that the rules above hold, and runs the notebook in a local kernel. The Colab
+   run itself is checked by hand.
+5. On Colab, Step 0 installs the `notebook` and `zarr` extras; locally it installs `notebook`.
+6. Steps 0 and 1 have *Colab users* / *JupyterLab users* callouts. On Colab, keep the data and
+   the output folder on Google Drive; the [user guide](docs/user_guide.md) §1 explains why, and
+   the limits of Google Drive: on Drive the lock cannot guard an output folder across machines
+   or sessions, so use one session at a time per output folder.
+
 From a terminal:
 
 ```bash

docs/development.md

@@ -25,6 +25,9 @@ functions the tests use.
 The notebook's Step 0 contains a standalone bootstrap using only the standard library,
 so it can locate a checkout or install missing requirements before importing the package.
 Its environment alternatives are tested without installing packages during the test suite.
+Colab-only setup sits inside `if "google.colab" in sys.modules:` and the GitHub installation
+is pinned to the release in `RELEASE_TAG` (plan §12 question 58);
+`tests/test_notebook_file.py` checks these rules and runs the notebook in a real kernel.
 
 ## Style
 

docs/plan.md

@@ -299,6 +299,8 @@ API, CLI). The notebook and CLI only call the API.
   *Setup feedback: standalone Step 0 detects the environment, prefers a nearby checkout,
   reuses an installed package, or installs missing requirements into the active kernel.*
   *Step-1 feedback: centre the folder-label table's headers and contents consistently.*
+  *v0.4.0 (PP-8d, question 58): the notebook runs on Google Colab and in JupyterLab following
+  LabConstrictor's portability guide; its GitHub installation is pinned to a release tag.*
   *Notebook guidance feedback: separate "What next?" boxes show the next action in larger
   text, following the current confirmations. Step-1 message details are collapsed with
   persistent orange warning and red error counts. Step-3 conversion settings use an
@@ -381,6 +383,20 @@ API, CLI). The notebook and CLI only call the API.
   35.*
 - **Release v0.3.1:** `position_id` names an image with the other image extensions bricks
   read (MorphoBricks D85, PR #33). *To be tagged, see question 36.*
+- **PP-10 (notebook cards) notes:** a button on the `output_locked` card that removes the
+  lock of a stopped run (owner, 2 November 2026, PP-8d review: Colab disconnects leave
+  locks behind). Same rule as `--remove-stale-lock`: only on that request, never the lock
+  of a process still running on the same computer. Until then the notebook's "Common
+  problems" gives a `remove_stale_lock(...)` code cell. *To do in PP-10.*
+- **Release v0.4.0 (PP-13) checklist**, for every tag of v0.4.0 including release
+  candidates, in the release PR before the tag is pushed:
+  - set `version` in `pyproject.toml`;
+  - set the notebook's pin (question 58): `RELEASE_TAG` in Step 0 to the new tag,
+    `current_version` to the tag without "v", the same version in
+    `notebooks/notebook_latest_versions.yaml`, and the tag named in the Step 0 Colab callout
+    (`tests/test_notebook_file.py` checks that the four agree);
+  - run the notebook by hand on Google Colab with the PR #118 checklist (the tests run it
+    only in a local kernel).
 
 ## 11. Tests and acceptance criteria
 
@@ -1235,3 +1251,32 @@ Raised during review; to be decided before the milestones that need them:
     `--remove-stale-lock`, as `check`, `run` and `masks` do (the `output_locked` message
     tells users to give it to the command). Same rule: the lock is removed only on that
     request, and never one of a process still running on the same computer.*
+58. How does the notebook run on Google Colab, and which release does it install (PP-8d)?
+    *Decided (2 November 2026, owner's PP-8d brief; supersedes the "Local Jupyter (not
+    Colab-first)" target): one notebook file for Google Colab and JupyterLab, following
+    LabConstrictor's notebook portability guide. Every code cell starts with `# @title`, and
+    explanations are in Markdown cells. Every code cell keeps its source hidden: the notebook
+    is an app-like widget notebook in which nothing is edited (LabConstrictor's code-hiding
+    guide), so this differs from fiberfeatures, whose settings cells stay visible. Colab-only
+    code runs inside `if "google.colab" in sys.modules:` in Step 0: Google Drive is mounted
+    at `/content/drive` and Colab's custom widget manager is enabled (the package needs
+    ipywidgets 8; Colab ships 7). Locally Step 0 prints the Python version, platform and
+    working folder. No shell or line magics; pip runs through `subprocess` with the kernel's
+    Python. When the package is missing, Step 0 installs the release named by `RELEASE_TAG`
+    (v0.4.0rc2 first), never an unpinned `main`; a nearby checkout and an installed package
+    are still used first, and Step 0 prints which one and its version. `current_version`
+    equals the tag without "v" and `notebook_name` the file name, as LabConstrictor's
+    version check reads them, with `notebooks/notebook_latest_versions.yaml` as the index;
+    the pin is moved at every release (§10, release checklist). Steps 0 and 1 have "Colab
+    users" / "JupyterLab users" callouts (data and results on Drive, output folder outside
+    the experiment, one session per output folder, a runtime wiped on disconnect). The run's
+    semantics are unchanged on Google Drive; its limits there are in the user guide §1.*
+    *Decided in review (2 November 2026, owner, PR #118 questions): (1) the lock on Google
+    Drive is documented only: one session per output folder, because the lock cannot guard
+    an output folder across machines or sessions on Drive; (2) a button that removes the
+    lock of a stopped run comes in PP-10 (§10, PP-10 notes); until then a documented code
+    cell; (3) no advice to copy the dataset to the runtime's own disk, because reuse
+    depends on the absolute path of the images: data and output stay on Drive; (4) on
+    Colab, Step 0 installs the `notebook` and `zarr` extras (locally `notebook`, unchanged),
+    so OME-Zarr images can be read there; (5) `RELEASE_TAG` stays `v0.4.0rc2`; the owner
+    tags it after PP-8c and PP-8d are merged.*

docs/user_guide.md

@@ -20,12 +20,14 @@ run **Step 0**. It detects local Jupyter or Google Colab and checks these option
    subfolders, use that checkout's source code, including local edits.
 2. **Installed package:** otherwise, use the package already available to this kernel.
 3. **GitHub installation:** if the package is missing, install it with the notebook
-   dependencies from the project's GitHub repository.
+   dependencies from the project's GitHub repository, at the release the notebook names
+   (`RELEASE_TAG` in Step 0, for example `v0.4.0rc2`), never the changing `main` branch.
 
 Missing notebook dependencies are installed into the same Python environment as the
 kernel. A local checkout is installed in editable mode when it needs dependencies;
 an installed package's notebook extra is used when its UI dependencies are missing.
-Setup prints the selected source and Python executable. It does not upgrade an already
+Setup prints the selected source with its version or release, the notebook's version and the
+Python executable. It does not upgrade an already
 usable installation. Installation needs internet access, and the GitHub option needs Git.
 If another copy of the package is already loaded, restart the kernel to switch checkouts.
 
@@ -40,9 +42,59 @@ pip install jupyterlab
 To read OME-Zarr images, install the `zarr` extra too:
 `pip install "morphobricks-preprocessing[notebook,zarr] @ git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git"`.
 
-Start JupyterLab with `jupyter lab` and open the notebook. Google Colab is detected by the
-setup cell, but native **Browse…** dialogs are only available with a local desktop: paste
-paths on the runtime filesystem when using a hosted notebook.
+Start JupyterLab with `jupyter lab` and open the notebook. **Browse…** opens your computer's
+folder dialog; where no dialog can open (Google Colab, a remote server), it says so and you
+paste the path instead.
+
+### Google Colab
+
+Open the notebook in Google Colab (**File › Open notebook › GitHub**, or upload the file) and
+run **Step 0**. On Colab it:
+
+- connects your Google Drive: allow access in the window that opens. Your Drive is then the
+  folder `/content/drive/MyDrive`;
+- installs MorphoBricks preprocessing at the release the notebook names, with its notebook
+  dependencies and the `zarr` extra (to read OME-Zarr images), which takes a minute or two
+  (ipywidgets 8 replaces Colab's ipywidgets 7; if a message asks you to, choose
+  **Runtime › Restart session** and run Step 0 again);
+- enables Colab's widget support for these widgets.
+
+**Where to keep the data.** A Colab runtime is a temporary computer: everything on it except
+Google Drive is wiped when it disconnects (after some idle time, or after several hours).
+
+- Put the experiment folder on Google Drive and paste its path in step 1, for example
+  `/content/drive/MyDrive/experiment_2026_10`. **Browse…** cannot open a dialog on Colab: open
+  the **Files** panel in the left bar, right-click the folder, choose **Copy path** and paste.
+- Keep the output folder on Google Drive too, outside the experiment folder. The default,
+  next to the experiment folder, is right when the experiment is on Drive.
+- Keep both there for every session. The output records where the images were read (their
+  full path) and each image's size and modification time; a later run reuses the rows only
+  when the images are read from the same place. Do not copy the experiment to the runtime's
+  own disk to read it faster: the copy is lost when the runtime disconnects, and reading it
+  from another place recomputes every row.
+
+**Limits of Google Drive** (the run behaves the same; these are things to know):
+
+- **One session per output folder.** On Google Drive the lock cannot guard an output folder
+  across machines or sessions. It only stops a second run in the same Colab runtime. Google
+  Drive copies files between computers with a delay, so a run started at the same time from
+  another Colab session, or from a computer that syncs the same Drive folder, is not
+  stopped. Use one session at a time per output folder on Drive.
+- **A lock left by a disconnected runtime.** If the runtime disconnects during a check or a
+  run, its lock stays in the output folder and the next check says "Another run is using
+  this output folder". When you are sure no other session uses the folder, remove it in a
+  new code cell, then check the folder again:
+
+  ```python
+  from morphobricks_preprocessing.run import remove_stale_lock
+
+  remove_stale_lock("/content/drive/MyDrive/experiment_2026_10_MorphoBricks_Preprocessing")
+  ```
+
+- **Unchanged images read again.** An image counts as unchanged when its size and
+  modification time are the same as at the last run. If Google Drive reports a file's
+  time differently later (for example after the file was synced again from another
+  computer), the image is read again. That costs time but never reuses wrong rows.
 
 ## 2. Organising the experiment folder