Skip to content

compile-many: stage-2 perf regression on big-framework boards (esp32/teensy) #361

Description

@zackees

Summary

compile-many is ~2.5× slower than the legacy serial loop on big-framework boards (esp32-arduino, teensy-arduino), even after the framework-share seed shipped in #337.

Evidence

FastLED CI, esp32s3 board, last 5 runs each side (full FastLED examples set):

Path Median runtime Hits 60-min timeout?
serial loop (master, FASTLED_USE_FBUILD_CI=0) ~24 min no
compile-many (FastLED PR #2672) >60 min yes, twice in a row

uno via compile-many runs in ~2 min vs ~5 min serial — so the seed is doing its job on small-framework boards. The regression is specific to boards where stage-2 work dominates total runtime.

What we ruled out

  • Wrapper timeout: raised 1800 → 3600s, same outcome
  • --framework-jobs: raised 1 → 2, no improvement (there's only one stage-1 framework build to parallelize)
  • Cold zccache: even with a warm zccache from prior PRs, no improvement
  • setup-fbuild cache: confirmed cache hit, binary install is instant

Hypotheses to investigate (in order)

  1. Seed not firing on ESP32: compile_many.rs:665 gates the seed on env_name == stage1_env. Verify this comparison actually matches in the ESP32 case — log on each stage-2 worker whether the seed was applied or skipped.
  2. Stage-2 daemon serialization (compile-many stage-2 workers serialize at ~200ms intervals — what's the shared lock? #341): the SQLite pin/unpin under tokio::Mutex adds ~200ms per worker × ~100 esp32 examples = ~30s, not enough alone to explain 36+ min regression — but may compound with Fix ESP32-S3 PSRAM board configuration in board JSON database #3 below.
  3. Per-sketch link contention: even with the framework .o files seeded, each stage-2 sketch still re-links 250+ framework objects. Two parallel xtensa-esp32-elf-g++ linker invocations on a 2-core GHA runner may fight for IO/RAM. Worth measuring single-worker vs two-worker stage-2 wall-clock.
  4. Hardlink cost on Windows-style mounts: GH Actions Linux runners use ext4 so this shouldn't apply, but worth confirming the seed isn't accidentally falling back to copy.

Asks

Blocker for

FastLED PR #2672 (gate FASTLED_USE_FBUILD_CI default-on) is closed pending this fix. Without it, every FastLED CI run on esp32s3/teensy41 would be 2.5× slower than today.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: p1Important follow-up after p0 foundations

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions