[META] Fastest possible FastLED examples CI rebuild — profile + benchmark

[zackees]

Goal

Drive the wall-clock time to compile the full FastLED examples/ set on CI as low as possible, for two distinct regimes:

1. Cold — first time a runner sees this cache key. No fbuild cache, no toolchain on disk, no pip cache. This is what every new board / new OS / first PR after a cache bust pays.
2. Warm (group rebuild) — same graph re-built with a populated fbuild cache. This is the common path on master after #2319 landed in the FastLED repo.

We want both numbers low, and we want to understand where the time goes — not just trust that it feels fast.

Non-goals

• Not trying to speed up a single example compile. The unit under test is the whole example list for one board.
• Not building hardware-specific fast paths. Improvements must come from fbuild / cache / CI orchestration, not from dropping examples.

Approach

A dedicated orphan branch in this repo — bench/fastled-examples — containing only:

• .github/workflows/benchmark.yml — a single self-contained workflow (workflow_dispatch) that:
  • installs fbuild via the local setup action,
  • clones FastLED at a pinned SHA,
  • compiles all examples for a chosen board (uno by default, overridable),
  • prints per-phase wall-clock timings to the job summary,
  • uploads raw timing data as an artifact.
• README.md documenting the branch purpose.

Two invocations per change = one number for each regime:

• Run 1 with cache-version bumped / fresh key → cold.
• Run 2 immediately after → warm.

Phases we want to measure

• actions/setup-python
• pip install fbuild
• actions/cache restore (fbuild cache)
• git clone FastLED (+ LFS if relevant)
• fbuild daemon startup
• toolchain download / materialization (first compile)
• per-example compile time (or at least aggregate compile phase)
• actions/cache save
• job teardown

Each phase gets its own ::group:: + date +%s.%N delta so we can eyeball it in logs and parse it from artifacts.

Tracking

• Land orphan branch bench/fastled-examples with initial workflow + README
• First cold run — record baseline timings here
• First warm run — record baseline timings here
• Identify largest phase; open sub-issue + attempt optimization
• Iterate until cold and warm both stop improving

Baselines and per-iteration results will be posted as comments on this issue.

Dimensions to vary later

• Board: uno, esp32dev, esp32s3, teensy41 (different toolchain sizes)
• OS: ubuntu-latest vs ubuntu-24.04 vs self-hosted
• Parallelism: --parallel vs --no-parallel
• fbuild daemon: cold-spawn vs pre-spawned
• pip: uv pip vs pip vs pre-built wheel cache

Related

• PR ci: use FastLED/fbuild/setup action to warm per-board fbuild cache FastLED#2319 — wired the fbuild cache into build_template.yml.
• This issue is the follow-on: now that cache exists, how fast can we actually go.

[zackees]

Iteration 0 — infrastructure landed, first cold run queued

• Orphan branch pushed: bench/fastled-examples
• Benchmark workflow uses push trigger (since workflow_dispatch needs the workflow on main, which defeats the orphan-branch point). Parameters live in bench.config — edit + commit + push to re-run.
• First cold run (board=uno, ref=master, examples=all): https://github.com/FastLED/fbuild/actions/runs/24611956113 (currently queued).

Plan for baseline

1. Finish this cold run → record phase timings in next comment.
2. Push an empty commit with CACHE_BUST= cleared → warm run. Record.
3. Identify the largest phase. Attack it. Re-measure.

Phases currently timed

• checkout_fbuild — shallow clone fbuild for the setup action
• fbuild_cache_hit — cache hit flag
• clone_fastled — shallow clone FastLED at ref
• uv_sync — ./install / uv sync for FastLED's .venv
• compile — ./compile --no-interactive --no-parallel uno all
• job_total — full job wall-clock

actions/setup-python + pip install fbuild are inside the composite setup action; their wall-clock shows up in the Actions UI under that step rather than in phase-timings.tsv. I'll break those out explicitly after we have a baseline.

[zackees]

Iteration 0 — cold baseline

Run: https://github.com/FastLED/fbuild/actions/runs/24616066368
Commit: 009c1da
Config: BOARD=uno, EXAMPLES=all, FASTLED_REF=master, CACHE_BUST=cold-1776539797
Cache hit: false (cold, as intended)

Total wall-clock: 198s (3m 18s)

phase	seconds	% of total
Set up job (GH runner init)	3	1.5%
Checkout benchmark branch	1	0.5%
Load bench.config	0	—
pip install fbuild	4	2.0%
restore fbuild cache (miss)	1	0.5%
clone FastLED	2	1.0%
install uv	2	1.0%
uv sync FastLED venv	27	13.6%
compile examples	148	74.7%
save fbuild cache	3	1.5%
Post cleanup	7	3.5%

Observations

1. Compile dominates at 75%. ./compile --no-interactive --no-parallel uno all is the target of optimization. --no-parallel is the obvious lever — toggling it is the first experiment.
2. uv sync is 27s. Second-largest phase. Worth investigating whether the FastLED .venv can be cached at the benchmark level.
3. pip install fbuild = 4s is already cheap. Not a target.
4. Queue wait: ~2h 33m on FastLED-org ubuntu-latest — runner availability is the real outer bottleneck, but that's org-wide, not something this benchmark can fix.

Next

• Warm run queued: https://github.com/FastLED/fbuild/actions/runs/24618517601
• After warm numbers land, compare and pick iteration 1 target. Top candidates: (a) --parallel for compile, (b) cache the FastLED .venv to eliminate uv sync.

[zackees]

Iteration 0 — warm run (same cache key as cold)

Run: https://github.com/FastLED/fbuild/actions/runs/24618517601
Commit: 99736cd (empty commit, same CACHE_BUST=cold-1776539797)
Cache hit: false — see "Root cause" below.

Total: 180s (3m 0s). Cold vs warm delta: -18s (-9.1%). Compile step itself only dropped 148 → 143s (~3%).

phase	cold (s)	warm (s)	delta
Set up job	3	2	-1
pip install fbuild	4	2	-2
restore fbuild cache	1	0	-1
clone FastLED	2	3	+1
install uv	2	1	-1
uv sync FastLED venv	27	23	-4
compile examples	148	143	-5
save fbuild cache	3	2	-1
Total	198	180	-18

The 5s compile delta is runner noise — the cache was effectively empty.

Root cause: we cached the wrong directory

Log evidence from the warm run:

Cache not found for input keys: bench-v1-cold-1776539797-Linux-X64-...-uno-master
Running: .../fastled/.venv/bin/fbuild .../fastled/.build/pio/uno build -e uno
   0.00 =   BUILDING uno   =
  20.54 Artifact: .../.build/pio/uno/.fbuild/build/uno/release/firmware.elf
 ...
   0.00 =   BUILDING uno   =   ← next example
   1.50 Artifact: .../firmware.elf

Key findings:

1. fbuild's actual cache lives inside the FastLED checkout — at fastled/.build/pio/uno/.fbuild/ — not at $FBUILD_CACHE_DIR ($RUNNER_TEMP/fbuild-cache). The actions/cache step faithfully saved an empty directory.
2. PlatformIO toolchain lives at ~/.fastled/global_cache/ — also not cached by the current workflow.
3. fbuild toolchain lives at ~/.fbuild/ (per fbuild's setup action docs). Also not cached.
4. Inside a single run, fbuild's daemon is clearly effective — first example takes ~20.5s (toolchain init), subsequent examples ~1.5s each. That's the warm-at-run-level behavior. We want warm-across-runs.

Iteration 1 plan

Expand the cache to the directories that actually hold state across invocations:

• ~/.fbuild/ — fbuild toolchain + daemon state
• ~/.fastled/global_cache/ — PlatformIO toolchain downloads
• fastled/.build/pio/<board>/ — compile outputs + local .fbuild/ cache

Expected impact: compile phase should drop from ~143s to (first-example warm + 82 × near-zero) + cache restore overhead. Target: total < 60s on warm.

Implementation: switch from caching $FBUILD_CACHE_DIR to caching the three paths above. Separate cache keys per board so cache doesn't cross-pollinate between targets.

[zackees]

Iteration 1 — first attempt blocked by upstream fbuild regression

Run: https://github.com/FastLED/fbuild/actions/runs/24619508294 — failed, compile step errored on every example with build failed: io error: Operation not permitted (os error 1).

Root cause: fbuild-2.1.18 (released 2026-04-19T02:46Z — ~50 min before this run queued) is broken. Two bugs: (a) wheel's fbuild console-script is missing the exec bit, (b) every build errors with EPERM at 0.0s. Details filed as #129.

Iteration 0 ran against 2.1.16 and worked fine; uv sync now pulls 2.1.18 transitively.

Iteration 1b — pinning fbuild to 2.1.16

Run: https://github.com/FastLED/fbuild/actions/runs/24620250563 (queued)

Workaround: pin both the top-level pip install fbuild==2.1.16 and the FastLED-venv's fbuild via uv pip install --python .venv/bin/python "fbuild==2.1.16" after uv sync. Also chmod +x defensively.

Once this finishes:

• cache save footprint tells us what's actually worth caching
• warm run should show real savings (not the 5s noise we saw in iter0).

[zackees]

Iteration 1b — still blocked, broader regression confirmed

Run: https://github.com/FastLED/fbuild/actions/runs/24620250563 — failed, identical build failed: io error: Operation not permitted (os error 1) as iter1.

The fbuild pin did take effect — log shows `uv pip install "fbuild==2.1.16"` correctly downgraded `fbuild==2.1.18` → `fbuild==2.1.16` in the FastLED venv, and `.venv/bin/fbuild --version` printed `fbuild 2.1.16`. Compile still fails.

This is not limited to our benchmark

FastLED master's own build_uno CI is also red with the identical signature since 02:33 UTC today — see e.g. https://github.com/FastLED/FastLED/actions/runs/24619689637 (same build failed: io error: Operation not permitted (os error 1), fbuild 2.1.18). Confirmed FastLED master has gone red in the same window as upstream fbuild 2.1.18 landed.

Blocker

• fbuild-2.1.18 broken: wheel script missing exec bit + 'Operation not permitted' on every build #129 (fbuild 2.1.18 broken wheel + EPERM regression) — filed.
• Even downgrading to 2.1.16 in our venv doesn't fix it, which suggests the 2.1.18 breakage isn't purely version-tied. It may be a binary/daemon artifact that persists once 2.1.18 has been installed, or there's a second regression introduced by something that shipped on 2026-04-19 (PyPI cache, runner image, or a dependency). Either way, production is red, not just our benchmark.

Holding pattern

• Pausing iteration loop until either:
  • fbuild 2.1.19 ships with a fix, or
  • FastLED master CI goes green again.
• Best baseline we have: iter0 cold = 198s, iter0 "warm" (no real cache) = 180s, compile alone = 148s (board=uno, examples=all, fbuild 2.1.16, FastLED @ 5b7a1db).
• When upstream recovers, iter1 plan remains: cache ~/.fbuild + ~/.fastled/global_cache + fastled/.build/pio/<board>. Expected warm total well under 60s if fbuild's internal compile-cache is honored across runs.