Cypher integration tests for autonomous wrapper (1-cypher -> 3-cypher tiers) #178

Context

The link R package (NewGraphEnvironment/link, v0.38.0) ships an autonomous CLI for multi-host WSG dispatch:

bash data-raw/wsgs_run_pipeline.sh \
  --wsgs=A,B,C \
  --config=<bundle> \
  --schema=<schema> \
  --no-cyphers \
  --with-mapping-code \
  --force

In v0.38.0 the M4+M1 baseline (Phase 1 of #172's acceptance) was validated end-to-end: 16-WSG default-bundle dispatch lands 16/16 in fresh_default.streams on M4, ~20 min wall, exit 0, zero operator prompts. See v0.38.0 NEWS entry + planning/archive/2026-05-issue-172-provincial-run-autonomy-renames/README.md.

The cypher tiers from #172's acceptance (Phase 2 single-cypher, Phase 3 full distributed) were deferred — the M4+M1 baseline needed to land repeatably before adding cypher complexity. With v0.38.0 shipped, those tiers can now ramp up.

Problem

The autonomous CLI's cypher passthrough (--cy-workspaces=<csv>) was implemented but never validated against real cypher droplets after the rename + autonomy refactor. Specifically:

1. Single-cypher invocation (--cy-workspaces=job1) — wsgs_dispatch.sh spins job1 via rtj's cypher_up.sh, dispatches the M4 + M1 + cy1 buckets in parallel, pulls cy1's RDS via scp, consolidates the cy1 source into M4's PG via the renamed schema_consolidate.R. Trap-EXIT burns the cypher.
2. Three-cypher distributed (--cy-workspaces=job1,job2,job3) — same flow times 3. Validates LPT bucket allocation across 5 hosts with the host_speeds-weighted projection from prior _per_wsg_times.csv.
3. Step 0 pre-clean (state_clean.sh --schemas=<schema>) needs to cleanly drop the target schema on each cypher droplet via ssh cypher@<ip> 'docker exec ...'.

Goals

Layer cypher integration on top of the M4+M1 baseline. Spin one at a time, validate, then add another. After each integration, burn the cypher to confirm spin/run/burn cycle is tight and idempotent.

Acceptance — three tiers

Tier 1: 1-cypher integration (job1)

• Spin + smoke. From cold: bash data-raw/wsgs_run_pipeline.sh --wsgs=<13-WSG-study-area-set> --config=bcfishpass --schema=fresh --cy-workspaces=job1 --with-mapping-code. Verify:
  • Pre-flight passes for M4 + M1 + cy1.
  • Step 0 pre-clean drops fresh on all 3 hosts.
  • Step 1+2 snapshot completes on all 3.
  • Step 3 cypher_up.sh job1 succeeds; tofu output -raw droplet_ip resolves.
  • Step 4 cypher_prep.sh runs (git+install+snapshot+DDL fix on the cypher).
  • Step 5 archive runs_archive.sh on all 3 hosts.
  • Step 7 dispatch — LPT splits 13 WSGs across m4 + m1 + cy1 with host_speeds-weighted plan.
  • Step 9 consolidate pulls cy1's docker-side fresh schema into M4 via the M1 + cy1 sources list.
  • Step 10 trap-EXIT burns cy1.
• Burn verification. After exit, doctl compute droplet list shows no cypher droplets; tofu state list for job1 workspace is empty.
• Idempotent re-spin. Re-run the same command. Cy1 spins fresh, runs again, burns clean. Validates the spin/burn cycle is tight.

Tier 2: 2-cypher integration (job1 + job2)

• Add --cy-workspaces=job1,job2. Same flow; LPT distributes 13 WSGs across m4 + m1 + cy1 + cy2.
• Validate balancing — projected finish times for the 4 hosts should be within ~10% of each other (LPT's projected_finish output).
• Both cyphers burn at exit.

Tier 3: 3-cypher integration (job1 + job2 + job3) — full distributed

• --cy-workspaces=job1,job2,job3. LPT distributes 13 WSGs across 5 hosts.
• Smoke step (Step 6) is no longer auto-skipped (--wsgs= is set but cyphers ARE used → smoke harness can do per-host validation). Verify smoke fires.
• Per-cypher RDS pull via scp cypher@<ip>:... works for all 3.
• Consolidate handles 4-source list (m1 + cy1 + cy2 + cy3).
• All 3 cyphers burn at exit.

Cross-tier acceptance

• Different scenarios on different runs. Validate that state_clean.sh --schemas=fresh_default cleanly switches the same cypher set between --config=bcfishpass and --config=default --schema=fresh_default between runs. Operator should be able to alternate without operator-side state-management.
• Custom schema names. Validate --schema=fresh_demo (arbitrary, not bundle's default) works end-to-end.
• Wrapper for spin/run/burn cycle. If muscle memory of cypher commands isn't tight enough, file a follow-up for a thin cypher_cycle.sh wrapper around the cypher_up + run + burn sequence. Not required for this issue's acceptance — included as a hook.

Test WSG set: study-area union

13 WSGs — the union of two upcoming reporting study areas:

• Skeena 2024: BULK, KISP, KLUM, MORR, ZYMO (from fish_passage_skeena_2024_reporting/0400-results.Rmd)
• Fraser 2025: LCHL, NECR, FRAN, MORK, UFRA, WILL, TABR, LSAL (from fish_passage_fraser_2025_reporting/index.Rmd params)

Use this set for Tier 1-3 validation. Mix-and-match a subset for incremental tests.

References

• v0.38.0 NEWS entry (NewGraphEnvironment/link NEWS.md) — describes the autonomy CLI surface.
• v0.38.0 acceptance artifact: planning/archive/2026-05-issue-172-provincial-run-autonomy-renames/README.md — M4+M1 baseline at 16/16 WSGs, 20m wall.
• Rapper helper script for cypher cycle (rtj): ~/Projects/repo/rtj/scripts/cypher/cypher_up.sh, cypher_run.sh, cypher_down.sh.
• wsgs_dispatch.sh cypher-side wrap template: lines ~474-491 (generates per-cypher wrap script for SSH tunnel + Rscript invocation).
• state_clean.sh --schemas= scoped mode (added in v0.38.0): data-raw/state_clean.sh.

Out of scope

• LPT-fallback empty-bucket edge case when N_WSGs ≤ N_hosts without timing CSVs — separate issue (pre-existing, not a cypher-integration regression).
• M4 PG over-tuning / fresh#199 — separate.
• Tunnel autossh / bcfp tunnel drops during long provincial runs cause silent per-WSG errors #167 — separate.
• S3-based consolidate / S3-based consolidate: route pg_dumps through s3://newgraph/ instead of serial scp #170 — separate.