The ESP32-S3 example sweep is sharded 8 ways in .github/workflows/build_esp32s3.yml. That sharding was a workaround for an fbuild bug, and the bug has since been fixed — so the shape of the gate should be reconsidered rather than left as-is.
The workaround and its stated cause
Sharding landed in be689c1 (#3952, closing #3791). The commit says why, in its own words:
A serial esp32s3 all-example sweep takes several hours because each sketch rebuilds the framework sources. Keep full coverage while each reusable-workflow job remains below its 45-minute bound (#3791).
#3791 records the same event from the other side: every esp32s3 run hung in_progress from 2026-07-31T20:39Z, against a normal runtime of 19m28s, and the first stuck run was on chore/fbuild-2.5.5.
The root cause, and its fix
"Each sketch rebuilds the framework sources" was not inherent to the sweep. It was FastLED/fbuild#1411/#1413: the ESP32 SDK-libs completion check required <mcu>/lib/libfreertos.a, which ESP32-S3 alone does not ship there — its FreeRTOS build is per flash/PSRAM mode, so the archive lives under dio_opi/, qio_qspi/, etc. The check therefore never passed, and every S3 build re-downloaded and re-extracted the 298 MB SDK archive before doing any work.
Measured in that fix: no-op build 161.2s → 0.3s, one-file fresh build 233.0s → 7.1s. It shipped in fbuild 2.5.22, and FastLED has been on 2.5.23 since #4158.
The effect is visible in this repo's own history:
| run |
date |
fbuild |
shard time |
33711963539 |
09-03 |
2.5.21 |
37–38 min |
33820032021 |
09-04 |
2.5.21 |
35–43 min |
33993978241 |
09-05 |
2.5.22 |
13–14 min |
34059602277 |
09-06 |
2.5.23 |
5–14 min |
The per-sketch framework rebuild is gone. Shard 4 of 34059602277 built 12 examples in 4m36s — ~23 s per sketch. If each still rebuilt the framework that would be minutes apiece.
What the current numbers actually say
From run 34059602277 (master, fbuild 2.5.23), and step timings from job 101557646580:
- Fixed per-shard overhead is ~35 s (
Set up job → build start, including a 21 s fbuild install/cache restore). Everything else is real compile work.
- Per-shard durations: 5:15, 6:51, 6:53, 10:40, 10:43, 11:26, 14:03, 14:25.
- Wall clock ≈ 14 min (bounded by the slowest shard). Total ≈ 75 runner-minutes across 8 concurrent runners.
- 79
.ino sketches in examples/.
The part that argues against simply removing the matrix
Duplicated setup is only ~35 s × 8 ≈ 4.7 min — sharding is not costing much in overhead. The ~75 runner-minutes is overwhelmingly compile work that a single job would still have to do, minus whatever framework/library warm-up is currently repeated per shard.
So a fully serial sweep today looks like roughly an hour, not the 19m28s of the pre-regression era (the sweep has grown since) and not the "several hours" of the bug era. That is above the timeout-minutes: 45 bound in build_template.yml — the same bound the sharding commit was written to stay under.
Reading it plainly: the bug that made sharding necessary is fixed, but the sweep is now large enough that a single job may still not fit. What is clearly no longer justified is 8 shards, which were sized against a several-hour serial cost that no longer exists.
Suggested next step
Measure before changing. Dispatch the workflow once with --shard-count 1 (and a raised timeout-minutes) to get a real serial number instead of an extrapolation from per-shard sums. Then:
- if it lands comfortably under 45 min, drop the matrix entirely;
- if not, reduce to 2–3 shards, which the post-fix numbers support and which cuts runner consumption while keeping the gate well inside its bound.
Either way the current 8 is sized for a cost that no longer exists.
Why this matters beyond minutes
This is a PR-gate workflow. #3791's own conclusion was that a CI system which does not flag a job going from 19 minutes to hours lets agents shard around a defect instead of fixing it. The shard count is the last remaining artifact of that episode; leaving it in place keeps the workaround's shape after its cause is gone.
Related: FastLED/fbuild#1411, FastLED/fbuild#1413, #3791, #3952, #4158.
The ESP32-S3 example sweep is sharded 8 ways in
.github/workflows/build_esp32s3.yml. That sharding was a workaround for an fbuild bug, and the bug has since been fixed — so the shape of the gate should be reconsidered rather than left as-is.The workaround and its stated cause
Sharding landed in be689c1 (#3952, closing #3791). The commit says why, in its own words:
#3791 records the same event from the other side: every
esp32s3run hungin_progressfrom 2026-07-31T20:39Z, against a normal runtime of 19m28s, and the first stuck run was onchore/fbuild-2.5.5.The root cause, and its fix
"Each sketch rebuilds the framework sources" was not inherent to the sweep. It was FastLED/fbuild#1411/#1413: the ESP32 SDK-libs completion check required
<mcu>/lib/libfreertos.a, which ESP32-S3 alone does not ship there — its FreeRTOS build is per flash/PSRAM mode, so the archive lives underdio_opi/,qio_qspi/, etc. The check therefore never passed, and every S3 build re-downloaded and re-extracted the 298 MB SDK archive before doing any work.Measured in that fix: no-op build 161.2s → 0.3s, one-file fresh build 233.0s → 7.1s. It shipped in fbuild 2.5.22, and FastLED has been on 2.5.23 since #4158.
The effect is visible in this repo's own history:
33711963539338200320213399397824134059602277The per-sketch framework rebuild is gone. Shard 4 of
34059602277built 12 examples in 4m36s — ~23 s per sketch. If each still rebuilt the framework that would be minutes apiece.What the current numbers actually say
From run
34059602277(master, fbuild 2.5.23), and step timings from job101557646580:Set up job→ build start, including a 21 s fbuild install/cache restore). Everything else is real compile work..inosketches inexamples/.The part that argues against simply removing the matrix
Duplicated setup is only ~35 s × 8 ≈ 4.7 min — sharding is not costing much in overhead. The ~75 runner-minutes is overwhelmingly compile work that a single job would still have to do, minus whatever framework/library warm-up is currently repeated per shard.
So a fully serial sweep today looks like roughly an hour, not the 19m28s of the pre-regression era (the sweep has grown since) and not the "several hours" of the bug era. That is above the
timeout-minutes: 45bound inbuild_template.yml— the same bound the sharding commit was written to stay under.Reading it plainly: the bug that made sharding necessary is fixed, but the sweep is now large enough that a single job may still not fit. What is clearly no longer justified is 8 shards, which were sized against a several-hour serial cost that no longer exists.
Suggested next step
Measure before changing. Dispatch the workflow once with
--shard-count 1(and a raisedtimeout-minutes) to get a real serial number instead of an extrapolation from per-shard sums. Then:Either way the current 8 is sized for a cost that no longer exists.
Why this matters beyond minutes
This is a PR-gate workflow. #3791's own conclusion was that a CI system which does not flag a job going from 19 minutes to hours lets agents shard around a defect instead of fixing it. The shard count is the last remaining artifact of that episode; leaving it in place keeps the workaround's shape after its cause is gone.
Related: FastLED/fbuild#1411, FastLED/fbuild#1413, #3791, #3952, #4158.