Expand experimental metadata to condition annotations, own controls and explicit contrasts #172

Goal

Make preprocessing the single source of truth for experimental identity and comparison metadata that later MorphoBricks can consume without re-asking the user how conditions relate to one another.

The current experiment_design.csv schema 1 records biological-repeat pairing and a global boolean control. That is too weak for real experiments and is already weaker than the design MorphoCoverage supports.

This issue should therefore be treated as a broader experimental metadata v2 design, not only as “allow several controls”.

What MorphoCoverage already teaches us

Reviewed against MorphoCoverage main at 64ff307d90fe5eb62a72630f35b85bd7fe6040a4.

MorphoCoverage deliberately separates several concepts that preprocessing currently mixes together:

1. Acquisition / sampling identity

MorphoCoverage's shared metadata contract treats these as different levels:

• condition
• biological_repeat
• optional technical_repeat
• position_id / image_file for FOV identity

It also recognises optional condition metadata such as target, explicitly not as a grouping column.

This should remain separate from comparison definitions.

2. One condition's primary/own reference

MorphoCoverage allows one control for all conditions, but with several controls it resolves one own control/reference per condition.

That is a useful invariant for preprocessing too.

Examples:

condition        own_control
DrugA            DMSO
DrugB            DMSO
KO1              WT
KO2              WT
VEC_SiCol6a1     VEC_SiNC
THBS5_SiCol6a1   THBS5_SiNC

A condition may take part in additional comparisons, but its primary reference remains unambiguous.

3. Treatment/target keys

MorphoCoverage can associate the same perturbation under different control backgrounds:

control        treatment_key   condition
VEC_SiNC       SiCol6a1        VEC_SiCol6a1
THBS5_SiNC     SiCol6a1        THBS5_SiCol6a1

That lets it derive an across-background comparison such as:

THBS5_SiCol6a1 / THBS5_SiNC
vs
VEC_SiCol6a1 / VEC_SiNC

This is richer and more useful than a generic “comparison set”.

4. Controls can themselves have a parent/reference

MorphoCoverage supports a control's parent:

VEC_SiNC   -> VEC
THBS5_SiNC -> THBS5

This expresses hierarchical experimental designs without pretending every control is globally equivalent.

5. Optional master control

MorphoCoverage can name a master control used for raw comparisons between control backgrounds.

This is a separate concept from each condition's own control.

6. Explicit contrasts

MorphoCoverage has a strict contrast table with the columns:

kind
condition
reference
condition_control
reference_control
family
label

and currently distinguishes:

• own_control
• across_normalised
• vs_master_raw

A condition can therefore appear in several comparisons even though it has only one primary/own control.

7. Experiment / family membership

MorphoCoverage separately supports experiment: a condition -> experiment mapping used to say which conditions belong to one statistical/correction family.

This should not be conflated with the biological-repeat group or the control mapping.

8. Positive control is a different concept

MorphoCoverage treats a positive control used to test representation/detection adequacy separately from the negative/reference control used for condition contrasts.

Preprocessing should preserve that distinction if we record positive controls.

Proposed preprocessing ownership

Preprocessing should own experimental facts and declared relationships.

It should not own MorphoCoverage analysis choices such as:

• readout/features;
• normalization method;
• feature scales;
• intensity-feature selection;
• statistical tests;
• stopping rules;
• FDR method.

Those remain downstream analysis configuration.

The preprocessing metadata should instead answer:

What was acquired, which samples/repeats belong together, what does each condition represent, and what comparisons/references did the experimenter intend?

Suggested file model

Do not force all of this into the current one-row-per-biological-repeat experiment_design.csv.

A simple layered model is easier to validate and reuse.

A. experiment_design.csv — sampling structure

Keep this focused on acquisition/repeat structure:

condition
biological_repeat
repeat_group
image_count
technical_repeat_count
confirmed
confirmed_on
schema_version

This remains one row per condition x biological-repeat folder.

The current schema-1 control column can be migrated for backward compatibility, but comparison semantics should move out of this table in v2.

B. condition_metadata.csv — one row per condition

Suggested canonical fields:

condition
experiment
target
own_control
treatment_key
positive_control
master_control
schema_version

Not every field must be populated.

Interpretation:

• condition: canonical condition identity.
• experiment: optional sub-experiment / analysis-family identity.
• target: optional biological target, matching MorphoCoverage's shared optional target concept.
• own_control: this condition's primary reference. For an ordinary control, it may be itself or its declared parent according to the final chosen convention.
• treatment_key: optional shared perturbation identity across backgrounds, e.g. SiCol6a1.
• positive_control: explicit positive-control role, kept separate from own_control.
• master_control: optional flag/role identifying the master reference for cross-background raw comparisons.

Whether positive_control / master_control are booleans or roles in a separate table can be decided during implementation; the important point is that they are not the same thing as own_control.

Allow additional condition descriptors only under a clearly defined extension rule; do not silently interpret arbitrary columns as grouping or comparison semantics.

C. contrasts.csv — optional explicit advanced comparisons

For advanced experiments, use a table compatible with MorphoCoverage's existing contrast vocabulary:

kind
condition
reference
condition_control
reference_control
family
label

This avoids making downstream bricks translate between two incompatible contrast schemas.

The simple UI should generate this table from condition_metadata.csv when possible. Expert users can inspect/edit explicit contrasts.

Why this is better than generic comparison sets

The earlier proposal used:

comparison_set   condition   role

That is not enough to answer several important questions:

• Which reference is this condition's own control?
• Are two conditions the same perturbation in different backgrounds?
• Is a control itself compared with a parent condition?
• Which control is the master reference?
• Is a comparison raw or normalized to two different controls?
• Which statistical/experimental family should the comparison belong to?

MorphoCoverage has already solved those semantics. Preprocessing should record enough metadata to reproduce them.

Notebook design

Keep the UI progressive so simple experiments remain simple.

1. Biological repeats

As today:

• pair biological repeats across conditions;
• confirm comparability;
• record technical-repeat/image counts.

2. Conditions

Show one row/card per condition.

Allow optional annotation:

• biological target;
• experiment/sub-experiment;
• primary/own control;
• treatment key;
• positive-control role.

For a single-control experiment:

Reference condition: DMSO
[Apply DMSO as the reference for all other conditions]

should be enough.

For several controls:

DMSO     controls: DrugA, DrugB
WT       controls: KO1, KO2

The user should not have to edit a raw mapping.

3. Advanced relationships

Only reveal when needed:

• control -> parent links;
• master control;
• treatment keys shared across backgrounds;
• explicit additional contrasts.

Show a compact preview:

Primary comparisons
DrugA vs DMSO
DrugB vs DMSO
KO1 vs WT
KO2 vs WT

Across backgrounds
SiCol6a1 effect:
THBS5_SiCol6a1 / THBS5_SiNC
vs VEC_SiCol6a1 / VEC_SiNC

Nothing is inferred silently.

Cross-brick contract

Preprocessing should expose one strict reader that returns immutable experimental metadata.

Conceptually:

design.conditions
design.repeat_groups
design.condition_metadata
design.own_reference(condition)
design.experiment_of(condition)
design.target_of(condition)
design.treatment_key_of(condition)
design.comparisons
design.contrasts
design.positive_controls
design.master_control

Exact API names are not prescribed here.

MorphoCoverage should eventually be able to construct its Setup.control, Setup.experiment, and optional contrast table directly from this metadata rather than asking the user to redeclare the same experimental design.

Other bricks should use the same reader rather than parse the CSV files independently.

Validation invariants

At minimum:

Sampling

• every condition/repeat row names a real discovered condition;
• repeat-group rules stay as strict as schema 1;
• biological-repeat pairing is independent of comparison metadata.

Conditions

• exactly one metadata row per condition;
• every named condition exists;
• targets/treatment keys/experiment names are strict typed text identities;
• no NA-like, whitespace-only or silently coerced identifiers.

Own references

• a condition has at most one primary/own control;
• referenced controls exist;
• no invalid self-reference under the chosen convention;
• no reference cycles if controls may themselves have parents;
• a condition may lack a control entirely.

Treatment keys

• a treatment key may link comparable perturbations across different backgrounds;
• the same key under one control cannot ambiguously name two conditions unless explicitly designed to support replicate reagents;
• treatment keys never imply that two conditions are identical.

Contrasts

• every condition/reference exists;
• contrast labels are unique;
• own_control is unique per condition;
• across_normalised requires both side controls;
• kind is explicit, never inferred from labels;
• family is explicit or deterministically generated.

Experiments/families

• every condition belongs to at most one declared experiment/family unless we explicitly design multi-membership;
• experiment membership is distinct from repeat grouping and control membership.

Provenance and reruns

The run manifest should hash/pin the exact experimental metadata consumed:

• repeat/sampling design;
• condition metadata;
• explicit contrast table when present.

Changing only experimental metadata should not reread image pixels. It should produce a new manifest/provenance generation and make downstream bricks use the new declared design.

A downstream brick must refuse a metadata file whose SHA differs from the one recorded by the run, as it already does for experiment_design.csv.

Migration from schema 1

Migration must be deterministic.

For the current boolean control:

• zero controls -> no own-control mapping;
• exactly one control -> it may become the default own control of all other conditions;
• several control=True conditions -> do not guess which conditions belong to which control; require user confirmation in the notebook.

Preserve the original schema-1 file until the v2 design is confirmed and written atomically.

Acceptance

• Experimental metadata distinguishes sampling structure from condition/comparison structure.
• Notebook supports one global control with minimal interaction.
• Notebook supports several condition-specific own controls.
• Controls may have parent/reference relationships when needed.
• A master control can be declared independently of own controls.
• Conditions may have optional target, treatment_key and experiment metadata.
• Positive controls are represented separately from negative/reference controls.
• Advanced explicit contrasts use a schema compatible with MorphoCoverage's contrast table.
• A condition can participate in several explicit contrasts while retaining one primary control.
• Biological-repeat grouping remains independent of comparison semantics.
• Strict shared reader exposes the design to every MorphoBrick.
• MorphoCoverage can consume preprocessing-owned metadata without redeclaring the same controls/experiments/contrasts.
• Invalid or ambiguous relationships are refused rather than guessed.
• Metadata-only edits do not recompute image histograms.
• The run manifest pins the exact metadata files consumed.