Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions docs/CONFORMANCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Conformance data: `labconstrictor_tools/conformance.json`

One machine-readable file, shipped inside the `labconstrictor-tools` wheel (`[tool.setuptools.package-data]`), that holds the rules every host must agree on, as data. It implements part of the proposal in [HOST_PARITY.md](HOST_PARITY.md#2-proposed-conformance-file) (roadmap phase 0.3, "87a") and carries the fixtures of [SPEC_WRONG_SCIENCE.md](SPEC_WRONG_SCIENCE.md) under the key `wrong_science`. Hosts keep their own constants and code; their tests read this file and compare. A change of a rule is a change of this file and a failing test in every host that has not followed.

**Only decided rules are in it.** A rule the spec marks `(default, not yet confirmed)` has `"status": "default"`; a rule nobody decided is not in the file (see "Gaps" below). The file never holds a value only one host needs.

## Layout

| Key | Content | Run by |
|---|---|---|
| `conformance` | format version (1) | the loader |
| `limits` | named numbers: `results_kept`, `run_records_kept`, `form_integer_max` (9007199254740991), `unbounded_float` (1e15), `choices_debounce_ms`, `choices_busy_retry_ms`, `worker_close_timeout_s`, `lookup_in_flight_max`, `lookup_pending_max`, `lookup_timeout_s`; each `{value, status: decided\|default, spec, ...}` | every host: its constant must equal `value` |
| `host_internal_codes` | the four new host-side codes (`frame_unsupported_transform`, `selection_mixed_planes`, `selection_plane_mismatch`, `target_unavailable`) | hosts |
| `messages` | sentence templates with `{name}` placeholders (Python `str.format`; other languages replace `{name}`) | hosts compare the text they produce |
| `compat` | the compatibility rules of [PROTOCOL.md](PROTOCOL.md) as vectors | Tools (`compat.check_schema`); Fiji and QuPath apply the same rules in their language |
| `wrong_science.coordinate` | base image `ASYM`, the rectangle and its mask, the placement formula, cases CF-01..CF-14 | hosts (and an arithmetic check in Tools) |
| `wrong_science.calibration` | unit table (um per unit), micro-sign spellings, the damaged-sign rule, anisotropy and default-dpi rules, the `_calibration` request shape, cases CAL-01..CAL-18 and transitions CAL-T1..T9 | hosts (and an arithmetic check in Tools) |
| `wrong_science.retention` | marker names and fields; `fixtures_tools` RET-01..09, 12, 13 (structured: setup, action, expect) and `fixtures_hosts` HRET-01..04, RET-10, 11, 14 (prose) | Tools runs `fixtures_tools` against `retention.prune` and the CLI; every host runs `fixtures_hosts` with the `prune` subprocess replaced by a recorder |
| `wrong_science.run_identity`, `async_lookups` | RUN-01..06, ASY-01..08 (prose sequences and expectations) | hosts |
| `wrong_science.number_limits` | NUM-01..05 | hosts (Napari, Fiji dialog and macro, QuPath, notebook) |

A host that cannot run a case reports it as **unsupported**, never as passed (SPEC_WRONG_SCIENCE 1.5, 3.7).

## The loader

`labconstrictor_tools.conformance` (standard library only):

```python
from labconstrictor_tools import conformance
data = conformance.load() # validated on every first load; ConformanceError lists every problem
conformance.path() # the file, for hosts that are not Python
conformance.validate(some_dict) # the schema: a list of problems, empty when valid
```
`python -m labconstrictor_tools.conformance --path` prints the file's location (inside the app's or the host's Python environment); `--validate` checks it. The schema checks structure, not host behaviour: required keys, numeric `value`, `status`, identifier names, `{name}` placeholders only, unique ids with the right prefix per section, `compat` vectors in order, the four-vertex rectangle, a positive um factor per unit. A malformed edit therefore fails every host test at once; it cannot silently skip a rule.

## How each host's test loads it

| Host | Test | Loading | Checks |
|---|---|---|---|
| Tools (CLI, `command.py`, `convert.py`, notebook) | `tests/test_conformance.py` | `conformance.load()` | limits against the code, `compat` vectors against `check_schema`, the retention `fixtures_tools` against `retention.prune` and the CLI, arithmetic of the coordinate and unit data |
| Napari | `tests/test_conformance.py` (a plain script, no Qt needed for most of it) | `from labconstrictor_tools import conformance; conformance.load()` (Tools is a dependency of the plugin) | constants against `limits` (`_INT_RANGE`, `_FLOAT_RANGE` against `form_integer_max`, `unbounded_float`), `ResultPresenter` and refusal sentences against `messages`, `host_internal_codes`, CF placement through a fake viewer (`viewer.add_points` receives the Napari data coordinate), region mask through `selection_mask`, CAL and NUM cases through the schema and form code, HRET fixtures with a recorder instead of `prune` |
| Fiji | a case type `conformance` in `tests/run_cases.py`; Groovy side `tests/conformance_check.groovy` | the Python driver runs `python -m labconstrictor_tools.conformance --path` with the app's interpreter and passes the result in the environment variable `LC_CONFORMANCE`; Groovy: `new JsonSlurper().parse(new File(System.getenv("LC_CONFORMANCE")))` | the harness parses the script text without its last line with `new GroovyShell().parse(...)` and calls the helper functions (`polygonRoi`, `selectionMask`, `macroNumber`, ...) and reads the `@Field` constants against `limits` |
| QuPath | `tests/gui_test_conformance_body.groovy`, run by `tests/run_gui_test.sh` (or `tests/headless_core.groovy` where no GUI is needed) | `LcJson.parse(new File(System.getenv("LC_CONFORMANCE")))`, same environment variable | the static members (`LcConst.*`, `LcDialog.*`, `LcArea.clamped`, `LcRegistry.load`) against `limits` and `messages`; CF placement, CAL and NUM cases |

Rules for a host test: it fails when a key it knows is missing (no silent skip); it reports `unsupported` for a case it cannot run; it compares its **own** behaviour (drawn coordinates, request contents, files on disk), not only that no error occurred (SPEC_WRONG_SCIENCE 1.5).

## Gaps: what the spec does not yet decide (so the file does not contain it)

1. **Keys of the HOST_PARITY sketch that are not in SPEC_WRONG_SCIENCE**: `max_shapes`, `table_rows_shown`, `region_max_objects`, `max_export_bytes`, `max_display_bytes`, `cancel_grace_s`, `worker_orphan_grace_s`, `worker_idle_s`, the log limits, `error_codes`, `notice_codes`, `worker_exit_hints`, `number_text`, `quoting` and `command_vectors`, `results_text`, `trust_vectors`, and the wording of the HOST_PARITY `messages` that are not in the wrong-science spec. They are proposals of the parity audit, not decisions; `number_text` also diverges from the Tools CLI today (TOOLS-7), and the Windows quoting of a trailing backslash is marked "TO DECIDE". `tests/quote_vectors.json` remains the shared quoting source until these are decided. `host_error` (QuPath's internal code) is not listed in `host_internal_codes` because the spec names only the four new ones.
2. **Axis order of `rect`** (section 1.5): the spec does not say; the mask pixels show it is `[y, x]` (Napari order), while shapes in the protocol are `[x, y]`. CF-09 and CF-14 give `server_wh` as `[width, height]` and `export_shape` as `[rows, columns]` (mixed conventions inside one fixture list).
3. **CF-07, 08, 10, 11, 12, 13** and all of RUN, ASY, HRET and RET-10/11/14 are sequences with prose expectations; the spec defines no machine-executable vocabulary for them, so only Tools' own fixtures (RET-01..09, 12, 13) are structured, and the others are interpreted by each host's harness.
4. **Length units** (2.1.4): the spec lists `um`, `micron(s)`, `micrometer/-metre`, `nm`, `mm`, `cm`, `m` and the Angstrom spellings, but CAL-03 uses the long form `millimeter`. The file lists `millimeter` and `millimetre` (needed by CAL-03) and no other long form (`nanometer`, `centimeter`, `meter`); case sensitivity, plural forms and surrounding spaces are not specified.
5. **Number limits** (6): the refusal sentence is given for integers only (`'L' must be <= 9007199254740991`, the `>=` form from HOST_PARITY I06); the float sentence and the message when a GUI box cannot hold the number are not specified. The spec writes the keys as `limits.form_integer_max = 9007199254740991`; the file nests the number as `limits.form_integer_max.value` (the layout of the HOST_PARITY sketch) so that it can carry `status` and `spec`.
6. **Tools divergence found while writing the file**: the notebook form refuses an integer beyond 2**53-1 with `'L' is N, beyond the +-9007199254740991 the form can hold exactly: ...`, not with the `'L' must be <= 9007199254740991` template of 6.2 (NUM-02, NUM-03). To fix with the notebook change; the file follows the spec.
7. **Lookup limits** (5.4: one in flight, 8 pending, 60 s) and the other working decisions of SPEC 8 are `status: default`, not yet confirmed.
8. **Napari N-D, QuPath downsample d != 1, TimeIntervalOf and ZStepOf** fixtures wait for Wave 2 (spec header); CF-14 is data, but no host exports at d != 1 yet.
9. **Run-identity and async fixtures have no pass criteria for timing** (no clock or barrier vocabulary): each host's harness decides how to build the "deterministic barriers" of 5.7.
2 changes: 2 additions & 0 deletions docs/HOST_PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,8 @@ Columns: id, rule, reference, Napari, Fiji, QuPath, CLI, notebook, status, note.

## 2. Proposed conformance file

**Status (phase 0.3): the file exists** with the rules already decided in [SPEC_WRONG_SCIENCE.md](SPEC_WRONG_SCIENCE.md) (limits, host-internal codes, compatibility vectors, the `wrong_science` fixtures) and a validating loader, `labconstrictor_tools.conformance`; see [CONFORMANCE.md](CONFORMANCE.md), which also lists the keys of the sketch below that are not decided yet and therefore not in the file. The sketch is kept as the proposal for them.

**One machine-readable file in this repository: `labconstrictor_tools/conformance.json`** (inside the package, so that every environment that already has `labconstrictor-tools` installed has it; add `[tool.setuptools.package-data] labconstrictor_tools = ["conformance.json"]` to `pyproject.toml`; `tests/conformance.json` would not reach the Napari, Fiji and QuPath test environments). It holds the values and test vectors the rows above show to be copied by hand into four code bases: named limits, error codes, message templates, number grammar, quoting, command vectors, coordinate vectors, result sentences, trust cases. Hosts keep their own constants; their tests read this file and compare. A change of a rule becomes a change of this file and a failing test in every host that has not followed.

Two keys carry the migration: `value` is the reference and `today` is what each host has now, so the first run of the new tests prints exactly the `DIVERGES` rows of section 1 and nothing else.
Expand Down
2 changes: 1 addition & 1 deletion docs/PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Feature ids (the implemented ones are in `FEATURES`; a planned id appears there
| `retention_prune` | the CLI command `labconstrictor-tools prune [--root results\|runs\|all] [--dry-run] [--json]` and `labconstrictor_tools.retention`: only folders carrying a valid `.lc-owner.json` and a `.lc-done.json` are ever removed ([SPEC_WRONG_SCIENCE.md](SPEC_WRONG_SCIENCE.md) section 3; hosts only mark a run finished, then start `prune`) | in the first release after 0.2.0rc1 (not in 0.2.0rc1) |

### Compatibility rules for hosts
A host that needs a minimum states it as `(min_tools_version, required_features)` and applies the rules below to every schema it loads, with `labconstrictor_tools.compat.check_schema` (Python hosts: Napari, notebooks) or the same rules written in its own language (Fiji, QuPath; the data is in `conformance.json`, [HOST_PARITY.md](HOST_PARITY.md#2-proposed-conformance-file), `compat` section). The result is `ok`, `warn` or `refuse`, with one sentence for the person:
A host that needs a minimum states it as `(min_tools_version, required_features)` and applies the rules below to every schema it loads, with `labconstrictor_tools.compat.check_schema` (Python hosts: Napari, notebooks) or the same rules written in its own language (Fiji, QuPath; the data is in `conformance.json`, [CONFORMANCE.md](CONFORMANCE.md), `compat` section). The result is `ok`, `warn` or `refuse`, with one sentence for the person:
1. `protocol` not supported by the host: `refuse` (as today).
2. A required feature missing from `features` (or no `features` list at all): `refuse`, "needs <feature>; update the app's labconstrictor-tools".
3. `tools_version` missing (unknown/old) while a minimum is set: `refuse`; with no minimum set: `warn`, "Tools version unknown".
Expand Down
1 change: 1 addition & 0 deletions docs/REGRESSION_LEDGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ How to add one: [TESTING.md](TESTING.md#the-ledger).
| 2026-10-08 | F15: an empty, complex, object or text array is written as an image result (or crashes with KeyError) | `portable_dtype` and `_write_one` do not refuse what no host can open | this PR | `test_error_codes.py::ImagesNoHostCanOpen.test_complex_object_and_text_arrays_are_unsupported_dtype`, `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`make_array/*`) |
| 2026-10-08 | F16: hints on the wrong kind of parameter, `Min(5), Max(1)`, output markers on inputs and duplicate output names are accepted and ignored | `introspection` validates only some misuses | this PR | `test_error_codes.py::HintMisuse.test_bounds_on_text_and_an_empty_range`, `test_error_codes.py::HintMisuse.test_duplicate_output_names`, `test_roundtrip_hints.py::InvalidCombinations.test_every_invalid_combination_is_refused_with_a_message_that_names_the_culprit` |
| 2026-10-12 | An output declared `Name("../other_run/result")`, an absolute name or `Name("cells/area")` wrote outside the result folder (or failed because the directory did not exist) | `introspection._outputs` checked only duplicate names; the writers joined `name + suffix` onto the job folder unchecked | [#104](https://github.com/CellMigrationLab/LabConstrictor-Tools/pull/104): unsafe names are a `DeclarationError` (`introspection._check_output_name`); `convert._job_file` refuses any path that is not a direct child of the job folder | `test_output_names.py::Declaration.test_unsafe_names_are_a_declaration_error_for_every_output_type`, `test_output_names.py::Writers.test_every_writer_refuses_a_name_that_leaves_the_job_folder` |
| 2026-10-12 | Found writing the cross-host data: the 12 Oct spec rules (number limits, coordinate frame, calibration, retention) lived only as prose and as copies in four code bases, so one host could drift without any test noticing | No shared, validated, packaged source of the rules: the proposed `conformance.json` did not exist and PROTOCOL.md already pointed at it | this PR: `labconstrictor_tools/conformance.json` (package data) and `conformance.load()` / `validate()`; data checked against the code and against plain-Python oracles | `test_conformance.py::Schema.test_each_kind_of_damage_is_named`, `test_conformance.py::AgainstTheCode.test_limits_match_the_code_and_the_spec`, `test_conformance.py::AgainstTheCode.test_the_compat_vectors_give_the_same_answer_as_the_reference`, `test_conformance.py::CoordinateData.test_the_placement_formula_for_every_downsample_variant`, `test_conformance.py::Packaging.test_the_built_wheel_contains_the_file_and_the_loader_reads_it_from_there` (and the sabotage rows `conformance:*`) |
| 2026-10-08 | F20: on ARM (Apple silicon) a uint64 image holding 2**64-1 is stored as float32 2**64 instead of being refused | `portable_dtype` checks exactness by casting the float32 back to uint64, an out-of-range cast that is undefined: x86 gives 0 (refused), ARM saturates (accepted) | this PR | `test_error_codes.py::Float32Exactness.test_uint64_beyond_float32_is_refused_with_unsupported_dtype_on_every_cpu` (numpy warns about the out-of-range cast on x86, the test turns that into a failure), `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_image/uint64/beyond-float32`; found by the macOS job of CI) |
| 2026-10-12 | A host sending a misspelt, renamed or removed parameter (`sigam` for `sigma`) got COMPLETE with the default value of the right one | `convert.load_inputs` looped over the declared inputs and never looked at extra keys (only the command line checked names) | [#106](https://github.com/CellMigrationLab/LabConstrictor-Tools/pull/106) | `test_roundtrip_unknown_inputs.py::UnknownInputs.test_a_misspelt_renamed_or_extra_input_is_refused_and_the_tool_does_not_run` |
| 2026-10-12 | A worker that wrote megabytes to stderr without a newline (one dumped array, no line ends) made the host hold all of it, and the tail kept more than its 1 MB limit | `_Tail.append` never evicted the only line left, and `for line in stderr` buffered a whole line before the tail saw it | [#103](https://github.com/CellMigrationLab/LabConstrictor-Tools/pull/103) | `test_roundtrip_failures.py::Noise.test_one_endless_stderr_line_is_bounded_while_it_is_still_open`, `test_roundtrip_failures.py::Noise.test_a_single_line_longer_than_the_limit_keeps_its_end_only`, `test_roundtrip_failures.py::Noise.test_the_end_of_an_oversized_line_is_in_the_crash_details` |
Expand Down
2 changes: 1 addition & 1 deletion docs/SPEC_WRONG_SCIENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Status: **for review; the retention rules of section 3 are implemented in Tools (`labconstrictor_tools.retention`, `labconstrictor-tools prune`, feature `retention_prune`), nothing else is.** Maintainer decisions are recorded in section 8.

**v1: 2D. N-D (T/Z stacks) and the QuPath resolution (downsample) choice follow in Wave 2.** The frame (1.1.2) and the placement formula (1.1.3) already have their final shape, so Wave 2 adds fixtures, not rules. Wave 2 fixtures still needed: `TYX` and `ZYX` results with `t`/`z` columns, a 2D result on a `TYX` layer (Napari), a per-axis `_calibration` with z and t, and QuPath export at downsample 2 and 4 on a real pyramid (CF-14 is the data, the GUI harness is the missing part). Same conventions as [SPEC_WAVE2.md](SPEC_WAVE2.md) and [HOST_PARITY.md](HOST_PARITY.md): additive within `protocol: 1`, rules are host-neutral, a rule is "done" when its fixtures pass in every host that claims the feature. The fixtures below are data, meant to live in `labconstrictor_tools/conformance.json` under the key `wrong_science` and to be reused by the conformance suite (Tools #87).
**v1: 2D. N-D (T/Z stacks) and the QuPath resolution (downsample) choice follow in Wave 2.** The frame (1.1.2) and the placement formula (1.1.3) already have their final shape, so Wave 2 adds fixtures, not rules. Wave 2 fixtures still needed: `TYX` and `ZYX` results with `t`/`z` columns, a 2D result on a `TYX` layer (Napari), a per-axis `_calibration` with z and t, and QuPath export at downsample 2 and 4 on a real pyramid (CF-14 is the data, the GUI harness is the missing part). Same conventions as [SPEC_WAVE2.md](SPEC_WAVE2.md) and [HOST_PARITY.md](HOST_PARITY.md): additive within `protocol: 1`, rules are host-neutral, a rule is "done" when its fixtures pass in every host that claims the feature. The fixtures below are data and live in `labconstrictor_tools/conformance.json` under the key `wrong_science` (see [CONFORMANCE.md](CONFORMANCE.md)), to be reused by the conformance suite (Tools #87).

These rules exist because a tool run can succeed and still be scientifically wrong: a mask on the wrong pixels, a result drawn away from its image, a pixel size from another image, a researcher's folder deleted. Each host fixes its own code, but all of them implement **the same rule**, so a fix in one host cannot leave another one behind.

Expand Down
Loading
Loading