Where does the trend live: github-action-benchmark vs committed JSONL, alert-don't-gate #197

Activity

1. blooop

   added

   wayfinder:research

   on Aug 14, 2026
2. blooop

   added a parent issue on Aug 14, 2026

   • Launch benchmarking: time-to-entrypoint as a tracked per-stage trend #193
3. blooop

   marked this as blocking on Aug 14, 2026

   • The CI trend workflow: warm + cold-recreate on main, publish, alert #198
4. blooop

   self-assigned this

   on Aug 14, 2026
5. blooop commented on Aug 14, 2026

   

   blooop

   on Aug 14, 2026

   OwnerAuthor

   More actions

   Resolution

   Use benchmark-action/github-action-benchmark with tool: customSmallerIsBetter, auto-push: true to a gh-pages branch, alert-threshold: '150%' + comment-on-alert: true + fail-on-alert: false — in its own workflow, not a job in ci.yml. It is a genuinely well-trodden, currently-maintained action (v1.22.1, 2026-05-06; repo pushed 2026-07-23; node24; 1.2k stars; not archived), it fits the house style of tag-pinned marketplace actions, and — the decisive point — its store is a committed JSON file, so picking it does not foreclose the hand-rolled option later.

   The one honest gap is the baseline, and it is not the gap the ticket guessed. Details below.

   ---

   1. Does the JSON shape carry 6 stages × 2 shapes legibly?

   Yes, as a flat list of named cases inside one suite. customSmallerIsBetter takes an array of {name, unit, value} with optional range and extra:

   [
     {"name": "warm / total",             "unit": "s", "value": 2.176,  "range": "± 0.04", "extra": "median of 5 | devpod v0.26.1 | ubuntu-latest"},
     {"name": "warm / wf-handoff",        "unit": "s", "value": 0.011,  "range": "± 0.00", "extra": "median of 5"},
     {"name": "warm / host-prep",         "unit": "s", "value": 0.190,  "range": "± 0.02", "extra": "median of 5"},
     {"name": "warm / devpod-up",         "unit": "s", "value": 0.505,  "range": "± 0.03", "extra": "median of 5"},
     {"name": "warm / tools",             "unit": "s", "value": 0.150,  "range": "± 0.02", "extra": "median of 5"},
     {"name": "warm / attach-exec",       "unit": "s", "value": 1.730,  "range": "± 0.09", "extra": "median of 5"},
     {"name": "cold-recreate / total",    "unit": "s", "value": 15.899, "range": "± 1.94", "extra": "median of 5 | image + clone cache warm"},
     {"name": "cold-recreate / devpod-up","unit": "s", "value": 10.600, "range": "± 1.20", "extra": "median of 5"}
   ]

   Facts that matter:

   • name (the action input) is the suite; the name inside each JSON object is the case. Use one suite (name: devlaunch launch stages) and 12 cases. The action's own README is explicit that the suite name "must be identical among all benchmarks".
   • Every case gets its own Chart.js chart on one generated page — 12 charts, no nesting, no per-shape grouping and no overlay of warm vs cold on one axis. Chart order is the order the cases appear in the JSON, which we control, so emitting all warm / * then all cold-recreate / * puts each shape's charts adjacent. Shape-prefixed names (cold-recreate / devpod-up) are the whole mechanism for expressing the 2-shape dimension — the format has no second axis.
   • range is worth using. Put the per-commit stddev there; it renders as spread on the chart, which makes the ±15% cold-recreate noise visible instead of implicit. extra is a free-text tooltip — good home for n=, devpod version, runner label.
   • window.BENCHMARK_DATA on the page holds everything, so a custom view can be built later without changing what we write.

   2. How the alert lands, and who owns the rolling baseline

   Nobody owns a rolling baseline — the action compares against the single immediately-previous entry, per case name. From src/write.ts:

   const prev = prevSuite.benches.find((b) => b.name === current.name);
   if (prev === undefined) { continue; }   // new/renamed case: silently no alert

   prevBench is the last stored entry for that suite — previous commit, not a median-of-history, not best-known, not a window. There is no input to change this. That is the real finding, and it lands directly on this map's noise problem: #193 measured cold-recreate singles spreading 15.2–20.0s (max/min = 1.32) on a quiet host, and shared runners are worse. Prev-vs-current on single samples would false-alarm whenever the previous commit happened to draw a fast run.

   The fix is in the harness, not the store: each committed point must already be a median of N repeats — which is exactly what scripts/bench_launch.py already computes. Median-of-5 collapses most of that spread, leaving real headroom under a 150% threshold. Start at one global alert-threshold: '150%'; #193 already parks per-stage thresholds as tune-after-experience, and that stays right.

   Delivery: on a push event there is no PR, so the action takes the leaveCommitComment path — a commit comment on the pushed merge commit, containing a markdown table of every case with current vs previous and the ratio. Not an issue, not a PR comment.

   • Needs github-token: ${{ secrets.GITHUB_TOKEN }} and workflow permissions: contents: write — one permission covers both the commit comment and the gh-pages push. (Repo default is read-only and ci.yml declares contents: read, so this must be declared; publish.yml already sets contents: write at job level, so it is not a new pattern here.)
   • Commit comments notify almost nobody. Set alert-comment-cc-users: '@blooop' — the @-mention in the body is what actually generates a notification. Without it the alert is functionally invisible.
   • fail-on-alert: false is the default and is the whole point. fail-threshold only applies when fail-on-alert is true, so leaving it unset is safe.
   • summary-always: true is a free win: the comparison table lands in the job summary of every run, so "did it move" is answerable without opening the chart or waiting for a threshold breach.

   3. gh-pages storage on this public repo

   Checked live: devlaunch has no gh-pages branch and GitHub Pages is not enabled (repos/blooop/devlaunch/pages → 404). Nothing to fight. Sphinx docs exist in docs/ but no workflow deploys them, so there is no future collision either — and if docs are ever deployed via actions/deploy-pages (artifact source, not branch source) they still would not touch this branch.

   What gets committed, on first run: the action creates gh-pages, writes dev/bench/data.js (a JS file assigning the full history to window.BENCHMARK_DATA) and generates dev/bench/index.html if absent, then pushes with GITHUB_TOKEN. It only ever touches benchmark-data-dir-path, so it coexists with anything else on the branch.

   • One manual repo-settings step: Settings → Pages → source gh-pages / root, or the chart is only readable as raw data.js. Free on a public repo. This is the single thing that cannot be done from the workflow file.
   • Chart URL will be https://blooop.github.io/devlaunch/dev/bench/.
   • Pushes made with GITHUB_TOKEN do not trigger workflows, so there is no feedback loop with the push: branches: [main] trigger.
   • Concurrent pushes: on a remote-rejected push the action does git reset --hard HEAD~1 and retries, up to 10 times — no rebase. It works, but a concurrency group is cheaper and makes it moot.
   • max-items-in-chart is unset by default = unbounded growth of data.js. Set it (500) so the file stays small and the chart stays readable; oldest entries are dropped.

   4. The rename sharp edge — real, but not a discriminator

   Rename warm / host-prep → warm / prepare-host and: the old case stops receiving points and its chart freezes with all its history intact; the new case starts an empty series; and per the prev === undefined branch above it is silently excluded from alerting until it has two points. Nothing is lost, but the trend line is cut in two.

   Recovery is cheap and total: data.js is a JSON literal with the names verbatim, so one sed -i 's#"warm / host-prep"#"warm / prepare-host"#g' dev/bench/data.js on gh-pages plus a commit rewrites the whole history and the series rejoins. Call it two minutes.

   It is the same two minutes under the committed-JSONL alternative — any store keyed by name has this property — so rename cost does not choose between the options. The actual mitigation is upstream: freeze the stage vocabulary before the first trend run. #193 already makes that a named ticket, and the vocabulary is also the contract with the wayfinder companion map, so it should be treated as a versioned interface and renamed only with a deliberate migration, not casually.

   5. Concrete workflow shape

   A separate workflow file, deliberately not a job in ci.yml: ci.yml's gate aggregator is the single required check, and house convention is that a new ci.yml job joins gate.needs in the same PR — which would turn a slow, noisy wall-clock measurement into a merge gate, i.e. exactly what #193 decision 1 grilled out.

   name: Launch benchmark
   
   # Trend-only. This workflow must never gate a merge (map #193, decision 1:
   # alert-don't-gate — runner variance makes a hard wall-clock gate either too
   # loose to fire or flaky). It lives outside ci.yml on purpose: a job in ci.yml
   # would join `gate`'s needs and become a required check.
   on:
     push:
       branches: [main]
     workflow_dispatch:
   
   concurrency:
     # Serialise. The action commits to gh-pages; two concurrent runs race on that
     # push. It does retry (reset --hard + up to 10 attempts), but not racing is cheaper.
     group: launch-benchmark
     cancel-in-progress: false
   
   permissions:
     contents: write   # push gh-pages AND post the alert commit comment
   
   jobs:
     bench:
       runs-on: ubuntu-latest
       timeout-minutes: 30
       steps:
         - uses: actions/checkout@v6
         - uses: prefix-dev/setup-pixi@v0.9.6
           with:
             cache: true
             frozen: true
   
         # Emits ONE customSmallerIsBetter array: ~6 ownership-boundary stages x
         # {warm, cold-recreate}, each value a MEDIAN of N repeats. Medians are
         # load-bearing: the action's baseline is the previous commit's single
         # point, so de-noising has to happen here or the alert is a coin flip.
         - name: Bench launch shapes
           run: pixi run bench-ci --repeats 5 --out bench.json --raw bench-raw.jsonl
   
         - uses: benchmark-action/github-action-benchmark@v1.22.1
           with:
             name: devlaunch launch stages
             tool: customSmallerIsBetter
             output-file-path: bench.json
             github-token: ${{ secrets.GITHUB_TOKEN }}
             auto-push: true
             gh-pages-branch: gh-pages
             benchmark-data-dir-path: dev/bench
             alert-threshold: '150%'
             comment-on-alert: true
             fail-on-alert: false           # the point of the whole map
             alert-comment-cc-users: '@blooop'   # commit comments notify nobody without this
             summary-always: true           # comparison table on every run, free
             max-items-in-chart: 500        # else data.js grows without bound
   
         # Raw per-run samples, for digging after an alert. The trend carries only
         # medians; artifacts carry the spread that produced them.
         - uses: actions/upload-artifact@v7
           if: always()
           with:
             name: bench-raw
             path: bench-raw.jsonl

   Pinning: @v1.22.1 exact, matching the prefix-dev/* precedent for actions that do something consequential; @v1 would also be house-consistent. Dependabot's monthly github-actions update covers either.

   The only new repo surface is the pixi run bench-ci task that emits that JSON — which is #198's job, and this settles its output contract.

   6. What was rejected, and why

   (a) Committed JSONL on an orphan branch + a static chart page. The honest case for it is real: you own the baseline, so you could alert on a rolling median of the last 10 commits rather than the previous commit — a strictly better fit for a signal this noisy. Rejected because it costs four hand-rolled moving parts (append, push-with-retry, a chart page, and compare-and-comment via gh api) to reimplement one tag-pinned action, against a standing preference for the simplest option when the cost is low and a repo that leans on well-trodden actions everywhere else.

   The decisive point is that this is not a one-way door. dev/bench/data.js is the committed JSON store the alternative was asking for — one file, plain JSON, on a branch, appended per commit. If prev-commit alerting proves too noisy in practice, a rolling-baseline alert step can read that same file and post its own comment, with alert-threshold raised out of the way, and no data migrates. Choosing the action buys the chart and the plumbing now and keeps the JSONL option open.

   (b) CI artifacts only. Fails the destination — #193 wants a developer to read a chart, and artifacts have no reader, expire (90 days default), and are not joinable across commits without writing a downloader. Rejected as the trend store, but kept in a supporting role: the workflow above uploads the raw per-run samples, which is where the spread behind each median lives when someone is investigating an alert.

   (c) The external-data-json-path hybrid (action manages a JSON file we check out and commit ourselves). Tempting on paper, but the chart page is only generated on the gh-pages path — this mode gives the store without the chart, i.e. the JSONL option minus its one advantage. Rejected.

   Follow-on for the map (not edited here)

   • The Pages setting (source gh-pages / root) is a one-time manual repo-settings step; the first workflow run creates the branch.
   • The CI trend workflow: warm + cold-recreate on main, publish, alert #198's harness must emit medians, not single samples, and should emit range (stddev) and extra (n, devpod version, runner) — the alert quality depends on it.
   • Stage names are a versioned interface shared with the wayfinder companion map; renames need the data.js sed, so freeze the vocabulary before the first run.
6. blooop

   closed this as completedon Aug 14, 2026