Repository navigation
Description
Original author: Cedric Vidal (@cedricvidal)
Summary
We recently migrated the 16 runs referenced in growth-ecosystems/scope-doc#51 from prod (https://msscope.azurewebsites.net) to integration (https://msscope-int.azurewebsites.net). The migration was done as a sequence of manual CLI calls + ad-hoc workarounds. We should capture this as a reusable standalone shell script under scripts/ so future cross-environment run migrations are one command.
Motivation
The migration decomposed into ~70% existing CLI primitives stitched together, plus two genuinely manual pieces (criteria-drift reconciliation and an upload size-cap workaround). A script removes the manual transcription/ordering risk and makes the process repeatable.
Proposed scope
A standalone bash script (e.g. scripts/migrate-runs.sh) that uses the existing pnpm cli commands and the REST API. Suggested interface:
scripts/migrate-runs.sh --from <SRC_URL> --to <DST_URL> -i <id...> [--submission-id <id>] [--dry-run]
Steps the script should perform:
- Preflight: verify both environments reachable; confirm each source run is
status: done(onlydoneruns are importable); warn on runs that already exist in the destination (import returns 409). - Criteria reproduction (exact): collect the transitive closure of all criteria referenced by the runs from the source, then in the destination:
- create any missing criteria (via
criteria import/POST /api/v1/criteria— upsert/$setOnInsert, so it will not overwrite existing), and - reconcile drifted criteria — criteria that exist in the destination but differ in
prompt/dependsOnmust be force-updated viaPUT /api/v1/criteria/:idso the destination matches the source exactly. (This is the key gap:criteria import/seedare insert-only and silently skip existing criteria. NotePUTaccepts an emptydependsOn: [], which the CLI's variadic--depends-oncan't express.) - verify parity (prompt + dependsOn) for the full closure before proceeding.
- create any missing criteria (via
- Download the runs from source (
run download-batch -i <ids...>). - Upload to destination (
run upload-batch, falling back to per-runrun upload). Import preserves the request_idand per-attemptrun._id. - Verify: each migrated run returns 200 /
status: doneand itsturns+scenario.criteriacounts match the source.
Known constraints to handle / document
- Upload size cap: integration's portal nginx (
apps/portal/nginx.conf,/api/reverse proxy) has noclient_max_body_size, so it defaults to nginx's 1 MB limit. Batch uploads >~1 MB fail (HTTP 413/502). Tracked separately ingrowth-ecosystems/scope-core-infra#92(raise to 25m). The script should fall back to per-run upload and surface a clear error when a run exceeds the cap, rather than crashing. Once infra#92 lands this becomes a non-issue — do not bake the artifact-stripping hack (dropping.har/ workspace snapshots to fit) into the script permanently; at most offer it as an explicit opt-in flag with loud warnings, since it reduces fidelity. - CLI quirk:
run upload-batchcrashes with "Unexpected end of JSON input" when a proxy returns a non-JSON error body (e.g. 413/502). The script should detect HTTP status explicitly (e.g. viacurl -w) for clean diagnostics. - Auth: GET and upload POST are currently unauthenticated in both environments (CLI sends no auth header). Keep the script env-agnostic via
--from/--toURLs.
Acceptance criteria
-
scripts/migrate-runs.shexists, is executable, follows the conventions of the other scripts inscripts/(set -euo pipefail, header comment block,--help). - Reproduces referenced criteria in the destination exactly (creates missing + reconciles drifted), with a pre-flight parity check.
- Migrates runs by ID and/or
--submission-id, preserving IDs. -
--dry-runprints the plan without mutating the destination. - Gracefully handles the upload size cap (per-run fallback + clear error), without permanently encoding the artifact-stripping workaround.
- Post-migration verification of
donestatus + turns/criteria parity. - Optional: a
pnpmalias inpackage.json(e.g.migrate:runs).
Reference
The full manual procedure, the 16 IDs, the criteria diff, and the size-cap findings from the #51 migration are documented and can seed the implementation.
Related: growth-ecosystems/scope-core-infra#92 (upload size limit).
Activity