Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -733,7 +733,7 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
| RGB `S` axis (Q3) | Prefilled as C, confirmed like any role |
| Significant bits (Q4) | Manifest only, information only; `bit_depth` and binning unchanged |
| Corrupt files (Q6) | Explicit exclusion list, recorded in manifest, rows removed |
| Notebook target | Google Colab and JupyterLab/LabConstrictor from one file, following LabConstrictor's notebook portability guide: one `# @title` runnable cell, explicit `display()` of the existing notebook pages, Colab-only Drive mount inside `if "google.colab" in sys.modules:`, no magics, core ipywidgets 7.7/8 compatibility, no forced widget restart/custom manager, remote install pinned to immutable `RELEASE_REF` |
| Notebook target | Google Colab and JupyterLab/LabConstrictor from one file, following LabConstrictor's notebook portability guide: three `# @title` code cells in order (1. Install, 2. Mount Google Drive, 3. Run preprocessing; owner, 10 November 2026), explicit `display()` of the existing notebook pages, Colab-only Drive mount inside `if "google.colab" in sys.modules:`, no magics, core ipywidgets 7.7/8 compatibility, no forced widget restart/custom manager, remote install pinned to immutable `RELEASE_REF` |
| CLI | `check <folder>` (summary, label tree, issues, suggested `--axes`) and `run <folder> --axes ROLES` with `--output`, `--exclude`, `--bins`, `--histogram-range`, `--robust-quantiles`, `--memory-limit 2GiB`, `--missing-as-singleton` (also on `check`, F10), `--mode restart|full`, `--no-masks`, `--no-exclude`, `--remove-stale-lock` and `--log`; also `masks`, `calibrate`, `design` and `convert` (the TIFF conversion options of v0.3, `--convert-tiff` and `--chunk-size`, were removed in v0.4.0). Axis roles are always stated explicitly, never taken silently from metadata (amended 3 November 2026, plan §12 question 73: when every image names all its axes, `run` uses those roles without `--axes` and prints them as the images name them; plain stacks still need `--axes`). Exit 1 on errors |
| Large planes (Q15) | Planes over the memory limit are read in blocks automatically, announced by the warning `planes_read_in_blocks` and `read_in_blocks` in the manifest. Block size derived from the limit (whole strips/tiles/chunks, uncompressed pieces and chunks also by rows). Compressed single-strip/tile planes keep the `plane_over_memory_limit` error. Two-pass histograms identical to whole planes. (Up to v0.3, OME-Zarr copies were written block by block; conversion was removed in v0.4.0.) Axis preview reduced (every n-th pixel) with a caption saying so |
| Normalization brick (P1) | The modules `dtypes`, `histogram`, `records`, `csv_io`, `pooling` and `normalization` moved unchanged to `morphobricks_normalization` (normalization plan N1, N2, N6) and are imported from there. Up to v0.3 their public names stayed importable from `morphobricks_preprocessing` and the old submodules, with a `DeprecationWarning`; those stand-ins (`_moved.py`) were removed in v0.4.0 (plan Q46). The package's own code never uses the old paths. Outputs unchanged (CSV schema 2); the manifest's `versions` adds `morphobricks-normalization`; Step 0 checks for it; contract §3–§7 text moved to the normalization contract (§1.2, now §6.2 for the CSV) |
Expand Down Expand Up @@ -818,14 +818,16 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
tests skip without zarr; CI's Python 3.11 jobs pin zarr 2.18 and 3.13 jobs get zarr 3.
- **The notebook file is tested too:**
- valid nbformat 4, with no outputs;
- code cells hidden and titled `# @title Run to …`;
- code cells hidden and titled `# @title <N>. …` (1. Install, 2. Mount Google Drive, 3. Run preprocessing);
- Colab-only code and `/content` paths only inside the Colab guard; no magics;
- the GitHub install pinned to `RELEASE_TAG`, equal to `current_version` and the
version index;
- The single runnable cell bootstraps the environment and then displays only widgets built by `notebook_ui`;
- exactly three code cells, in order: 1 installs and reports versions, 2 mounts Google Drive
on Colab (a one-line note elsewhere), 3 displays only widgets built by `notebook_ui` and,
run out of order, names the cell to run first (`tests/test_notebook_setup.py`);
- the buttons it names exist;
- its cells execute, by `exec` and in a real kernel (`nbclient`, `tests/test_notebook_file.py`;
Step 0's alternatives in `tests/test_notebook_setup.py`).
cell 1's alternatives and the cells out of order in `tests/test_notebook_setup.py`).

---

Expand All @@ -851,14 +853,18 @@ plainer version "not interactive enough". Descriptive, guided and forgiving is t
- **Code cells:**
- Hidden, with metadata
`{"cellView": "form", "collapsed": false, "jupyter": {"source_hidden": true}, "jupyterlabHideCode": {"locked": true}}`.
- The first line is a Colab title: `# @title Run to …` (LabConstrictor portability
guide; every code cell, all hidden, plan Q58). Longer explanations go in Markdown cells.
- The first line is a Colab title: `# @title <N>. …`, numbered as the headings (LabConstrictor
portability guide; every code cell, all hidden, plan Q58). Longer explanations go in Markdown cells.
- Outputs are stripped.
- Step 0 is a standalone standard-library bootstrap: nearby checkout, installed package,
- Three code cells (owner, 10 November 2026, as ZeroCostDL4Mic and NucleiSky): **1. Install**,
**2. Mount Google Drive**, **3. Run preprocessing**, each under a Markdown heading that
explains it. Cell 3 checks that cell 1 ran (`SETUP_RECORD`) and, on Colab, that Drive is
mounted, and otherwise stops naming the cell to run; no cell runs another.
- Cell 1 is a standalone standard-library bootstrap: nearby checkout, installed package,
or GitHub installation of the release `RELEASE_TAG`, with missing requirements
installed using the kernel's Python (`subprocess`, never `!pip`); on Colab with the
`zarr` extra too. Colab-only code
(Drive mount) stays inside `if "google.colab" in sys.modules:`; no `/content` path
(cell 2's Drive mount) stays inside `if "google.colab" in sys.modules:`; no `/content` path
outside it, and no custom widget manager is enabled. `current_version`
and `notebook_name` are LabConstrictor's names; at each release the pin moves
(plan §10, release checklist).
Expand Down
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,25 @@ v0.5.1: no run before v0.5.1 is read.)

## Unreleased (v0.5.1)

### The notebook has three code cells: install, mount Google Drive, run

Owner request of 10 November 2026, as in the lab's ZeroCostDL4Mic and NucleiSky notebooks. The
single code cell of `notebooks/MorphoBricks_Preprocessing.ipynb` is split into three, each under
a Markdown heading that explains it; what the cells do together is unchanged (`RELEASE_REF`,
the install target and `PreprocessingNotebook(describe=True)` stay as they were).

- **1. Install MorphoBricks preprocessing**: the pinned install with its extras, the version
report and `SETUP_RECORD`, as before; it no longer mounts Drive. Running it again installs
nothing that is already there; no restart is requested.
- **2. Mount Google Drive**: mounts Drive on Colab (the same refusal when the mount fails); in
JupyterLab or LabConstrictor it prints a one-line note and does nothing.
- **3. Run preprocessing**: shows the app. Run before cell 1 it stops with "run cell 1 (Install
MorphoBricks preprocessing) first", not an `ImportError`; on Colab without Drive at
`/content/drive/MyDrive` it stops naming cell 2. It never runs an earlier cell.
- Tests: exactly these three cells in order and each one's part (`test_notebook_file.py`), the
cells out of order and cell 1 run twice (`test_notebook_setup.py`), and the notebook in a
real kernel with and without a faked `google.colab`, and cell 3 alone.

### PP-C: every run is described; undescribed runs and their compatibility code removed

No backward compatibility before the first pipeline release (owner policy of 10 November 2026,
Expand Down
12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ pip install -e ".[dev]"

The optional `notebook` extra installs the widget dependencies for the notebook
(`pip install "morphobricks-preprocessing[notebook] @ git+https://github.com/CellMigrationLab/morphobricks-preprocessing.git@v0.5.0"`).
The notebook's single runnable cell can set this up automatically: it uses a compatible nearby
The notebook's first code cell (**1. Install**) can set this up automatically: it uses a compatible nearby
checkout or installed package, and installs missing requirements into the active kernel. A standalone
download installs from GitHub if needed (internet access and Git required).

Expand All @@ -81,9 +81,13 @@ is one portable widget app for Google Colab, JupyterLab and LabConstrictor. It f
LabConstrictor's
[notebook portability guide](https://github.com/CellMigrationLab/LabConstrictor/blob/main/.tools/docs/notebook_portability.md):

1. The notebook is intentionally small: one short Markdown introduction and one `# @title …`
runnable cell. That cell prepares the environment and explicitly displays the complete
preprocessing UI; users never edit code, so LabConstrictor may hide it.
1. The notebook is intentionally small: a short Markdown introduction and three `# @title …`
code cells, each under a heading that explains it, as in the lab's ZeroCostDL4Mic and
NucleiSky notebooks: **1. Install MorphoBricks preprocessing** (prepares the environment and
reports versions), **2. Mount Google Drive** (Colab only; elsewhere a one-line note) and
**3. Run preprocessing** (explicitly displays the complete preprocessing UI). Cell 3 run out
of order says which cell to run first and never runs it for the user. Users never edit
code, so LabConstrictor may hide it.
2. Colab-only setup stays inside `if "google.colab" in sys.modules:` and mounts Google Drive.
The UI uses only core ipywidgets supported by both ipywidgets 7.7 and 8; it does not force an
ipywidgets upgrade/restart or enable Colab's custom-widget manager. Locally the same notebook
Expand Down
31 changes: 21 additions & 10 deletions docs/colab-release-check.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,15 @@ named-axis copies). Items were updated for later changes: PP-10a replaced the st
cell with a button, and PP-12 replaced the `.run-*` backup folders with `.publish` and
`.staging`.

**The notebook's pin.** The shipped notebook installs the package at its `RELEASE_REF`, not
main's code, so its cell calls the API of the code at that commit. Between releases main's code
**The notebook's pin.** The shipped notebook installs the package at its `RELEASE_REF` (set in
cell 1), not main's code, so its cell 3 calls the API of the code at that commit. Between releases main's code
moves on while the cell keeps the released call: after v0.5.1's PP-C, main has no `describe`
argument, while `RELEASE_REF` is still v0.5.0's `fd7afd3`, whose cell calls
`PreprocessingNotebook(describe=True)`. The release PR moves `RELEASE_REF` and the cell's call
`PreprocessingNotebook(describe=True)`. The release PR moves `RELEASE_REF` and cell 3's call
in the same commit and records the new pair in `RELEASE_CALLS` of `tests/test_notebook_file.py`,
which fails for a `RELEASE_REF` without one.

- [ ] `RELEASE_REF` is the release commit, and the cell's `PreprocessingNotebook(...)` call is
- [ ] `RELEASE_REF` is the release commit, and cell 3's `PreprocessingNotebook(...)` call is
that commit's API (v0.5.1: `PreprocessingNotebook()`).

Use a dataset on Google Drive with a masks folder, and a second small experiment of
Expand All @@ -29,17 +29,28 @@ Report a failure as an issue before the final tag.

## A. Fresh runtime and app rendering

1. Open the notebook on a fresh Google Colab runtime and run its single code cell.
- [ ] Drive authorisation appears and the mount succeeds.
The notebook has three code cells, run in order: **1. Install MorphoBricks preprocessing**,
**2. Mount Google Drive** and **3. Run preprocessing**.

1. Open the notebook on a fresh Google Colab runtime and run cell 3 first.
- [ ] It stops at once with "run cell 1 (Install MorphoBricks preprocessing) first", not an
`ImportError`, and installs nothing.
2. Run cell 1 (install).
- [ ] Setup reports the expected notebook, preprocessing, normalization and ipywidgets
versions and the pinned source revision.
- [ ] No ipywidgets upgrade or runtime restart is requested.
2. In the same cell output:
versions and the pinned source revision, then names cells 2 and 3.
- [ ] No ipywidgets upgrade or runtime restart is requested, and Drive is not mounted.
- [ ] Run cell 1 again: nothing is installed again and the same versions are reported.
3. Run cell 3 before cell 2.
- [ ] It stops with "Google Drive is not connected … run cell 2 (Mount Google Drive)".
4. Run cell 2 (Drive).
- [ ] Drive authorisation appears, the mount succeeds and the cell says Drive is connected.
5. Run cell 3 (the app).
- [ ] The complete preprocessing app renders inline; Colab does not open the Code snippets
pane instead of the widget.
- [ ] Buttons, dropdowns, accordions, preview image, sliders, progress bars and matplotlib
plots render and react.
3. Repeat the same single-cell smoke test in JupyterLab/LabConstrictor.
6. Repeat cells 1 to 3 in JupyterLab/LabConstrictor.
- [ ] Cell 2 prints its one-line "Not Google Colab" note and does nothing else.
- [ ] The same app renders and uses local paths without executing Colab-only setup.

## B. Step 1 (PR #118)
Expand Down
5 changes: 5 additions & 0 deletions docs/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -1157,6 +1157,11 @@ Raised during review; to be decided before the milestones that need them:
Colab, Step 0 installs the `notebook` and `zarr` extras (locally `notebook`, unchanged),
so OME-Zarr images can be read there; (5) `RELEASE_TAG` stays `v0.4.0rc2`; the owner
tags it after PP-8c and PP-8d are merged.*
*Amended (10 November 2026, owner): the notebook has three code cells, as the lab's
ZeroCostDL4Mic and NucleiSky notebooks: 1. Install (the pinned install and the version
report), 2. Mount Google Drive (Colab only; elsewhere a one-line note), 3. Run
preprocessing (the app). Cell 3 run before cell 1, or on Colab without Drive, stops and
names the cell to run; no cell runs another.*
59. How are images with axis errors saved as named-axis copies (design v0.4 §5, PP-11)?
*Decided (owner, 2 November 2026; implemented in PP-11) as design v0.4 §5 says: images
only, as named-axis OME-TIFF (BigTIFF when large); copies keep the file name and folder
Expand Down
26 changes: 17 additions & 9 deletions docs/user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,23 @@ its own output folder, which is never inside the experiment folder.

You need Python 3.11 or newer and JupyterLab (or Google Colab). Download
[`notebooks/MorphoBricks_Preprocessing.ipynb`](../notebooks/MorphoBricks_Preprocessing.ipynb),
open it and run its single code cell. That cell prepares the environment and displays the
complete preprocessing widget app.
open it and run its three code cells in order:

The setup chooses one source:
1. **Install MorphoBricks preprocessing** prepares the environment and prints the versions.
Running it again installs nothing that is already there; no restart is needed.
2. **Mount Google Drive** connects Google Drive on Google Colab. In JupyterLab or
LabConstrictor it prints a one-line note and does nothing.
3. **Run preprocessing** displays the complete preprocessing widget app. Run before cell 1,
it stops and says to run cell 1; on Colab without Drive connected, it says to run cell 2.
It never runs an earlier cell for you.

Cell 1 chooses one source:

1. **Nearby checkout:** when the kernel is running in this repository or one of its subfolders,
use that checkout's source code. The checkout must be the commit the notebook pins
(`RELEASE_REF`) with no uncommitted changes, and setup prints its commit and tree state. Any
other commit or edited tree is refused; developers set `DEVELOPMENT_OVERRIDE = True` at the
top of the cell to use it, and the printed header then says so.
top of cell 1 to use it, and the printed header then says so.
2. **Matching installed package:** otherwise, reuse an installed preprocessing package only
when its version matches the notebook.
3. **Pinned GitHub source:** otherwise install the notebook's immutable `RELEASE_REF`, never
Expand Down Expand Up @@ -53,12 +60,13 @@ paste the path instead.
### Google Colab

Open the notebook in Google Colab (**File › Open notebook › GitHub**, or upload the file) and
run the notebook's single code cell. On Colab it:
run the notebook's three code cells in order. On Colab:

- connects your Google Drive: allow access in the window that opens. Your Drive is then the
folder `/content/drive/MyDrive`;
- installs the notebook's pinned preprocessing revision, with its notebook dependencies and
the `zarr` extra when needed;
- cell 1 installs the notebook's pinned preprocessing revision, with its notebook
dependencies and the `zarr` extra when needed;
- cell 2 connects your Google Drive: allow access in the window that opens. Your Drive is then
the folder `/content/drive/MyDrive`. Cell 3 refuses to open the app until Drive is
connected, so the data and the output are never silently put on the runtime's own disk;
- uses Colab's native core ipywidgets support. The notebook accepts ipywidgets 7.7 or 8 and
does not force a widget upgrade, runtime restart or custom-widget manager.

Expand Down
Loading
Loading