feat: resolve remote example assets at build time (sha256-verified, globally cached)

[zackees]

Summary

Add build-time resolution of remote example assets, so large binary payloads live in FastLED/assets instead of the FastLED source tree, and fbuild fetches them during the build.

Current state

fbuild resolves remote toolchain/platform archives today (ci/boards.py is full of https://github.com/pioarduino/...platform-espressif32.zip style URLs). There is no equivalent for example data assets. Any sketch needing a binary payload must commit it into the FastLED repo.

Concretely, examples/Fx/FxLedmapper32x32/data/ currently carries ~10 MB of binaries (a 6.05 MB video1.rgb and a 4.10 MB .zip that duplicates it). That is the case this issue exists to eliminate.

Proposed

A declarative asset reference resolved before compile, e.g. an assets.json (or a [assets] block) beside the sketch:

{
  "video1.rgb": {
    "url": "https://raw.githubusercontent.com/FastLED/assets/main/examples/FxLedmapper32x32/video1.rgb",
    "sha256": "<digest>",
    "size_bytes": 6051840
  }
}

fbuild resolves each entry into the sketch's data/ dir at build time.

Requirements

• Verify by content digest, not ETag. ETag is a server-chosen opaque validator, not a content hash — measured across four origins it ranges from the exact content MD5 (nodejs.org) to an Azure last-modified timestamp (GitHub Releases, "0x8DEFE5A52510C18"), which changes on re-upload of identical bytes. Integrity must come from sha256 in the manifest. See Add zccache fetch — global content-addressed cache for external tool downloads zackees/zccache#1469 for the full evidence.
• Cache globally, keyed by digest — not per-repo, not per-worktree. Same rationale as Add zccache fetch — global content-addressed cache for external tool downloads zackees/zccache#1469; ideally the same CAS.
• Conditional GET for revalidation — measured 304 in 0.14s / 0 bytes vs 200 in 16.0s / 29.7 MB on a comparable payload.
• Segmented download — raw.githubusercontent.com advertises Accept-Ranges: bytes; measured 3.7x on a 29.7 MB payload (14.2s → 3.9s at 8 connections).
• Offline / air-gapped path — a populated cache must build with no network.
• Deterministic failure — a digest mismatch fails the build loudly; it must never silently fall back to a stale or partial file.

Manifest format

Prefer consuming zackees/manifest.json rather than inventing a format. Its Index → Catalog → Release tiers with OCI-style {url, sha256, size_bytes} descriptors already give integrity-chained federation across repos and CDNs, which is exactly the shape needed once FastLED/assets is served over a CDN.

Related

• FastLED/assets — hosting + manifest (issue linked below)
• FastLED/FastLED — removing the committed binaries
• zackees/fastled-wasm — the WASM preview path must resolve the same URLs into its VFS
• Add zccache fetch — global content-addressed cache for external tool downloads zackees/zccache#1469 — the download/CAS engine this should ideally reuse

[zackees]

Design decision: .lnk sidecar files (supersedes the assets.json sketch above)

Rather than a per-sketch assets.json, a remote asset is declared by a sidecar file named after the asset itself:

data/video1.rgb.lnk      ->  resolves to  data/video1.rgb

This keeps the declaration adjacent to where the file will materialize, makes it obvious which asset is remote, and needs no central registry per sketch.

Format — plain-text key=value, one pair per line

# Remote asset descriptor. Resolved at build time.
url=https://cdn.jsdelivr.net/gh/FastLED/assets@main/examples/FxLedmapper32x32/video1.rgb
url=https://raw.githubusercontent.com/FastLED/assets/main/examples/FxLedmapper32x32/video1.rgb
sha256=164804d94802dd96cd8cd37f7528ca6d30f8fd9396e812fe67ec6164bbdd2282
size_bytes=6051840

Key	Required	Meaning
url	yes	Asset URL. May repeat — first is primary, later entries are fallbacks tried in order.
sha256	no (recommended)	Content digest; verified after download.
size_bytes	no	Expected size; cheap early mismatch check before hashing.

Lines beginning with # are comments. Blank lines ignored. Unknown keys should be ignored rather than fatal, so the format can grow.

Resolver behavior

1. For each <name>.lnk in a sketch's data/, resolve to <name> in the same directory.
2. If <name> already exists and its sha256 matches, do nothing.
3. Otherwise fetch, trying url entries in order.
4. Verify size_bytes (if given), then sha256 (if given). Mismatch is a hard build failure — never fall back to a stale or partial file.
5. Materialized assets should be gitignored; only the .lnk is committed.

Why not ETag

Unchanged from the original issue body: ETag is a server-chosen opaque validator, not a content hash. sha256 in the .lnk is the integrity guarantee. ETag remains useful only for cheap conditional-GET revalidation.

Live example

examples/Fx/FxLedmapper32x32/data/video1.rgb.lnk in FastLED/FastLED is the first real instance — the 6 MB video1.rgb is already removed from that repo and hosted in FastLED/assets. The example cannot build until this resolver exists; the readme documents a manual curl in the meantime.

[zackees]

Both declaration formats are supported (not either/or)

A sketch may use .lnk sidecars, a single assets.json, or both at once. They parse to the same record, so downstream code does not care which was used.

<name>.lnk — one sidecar per asset, named after the file it produces. Best when a sketch has one or two assets, since the declaration sits exactly where the file will appear.

assets.json — one file declaring many assets. Better when a sketch has several.

{
  "assets": {
    "video1.rgb": {
      "urls": ["https://cdn.jsdelivr.net/gh/FastLED/assets@main/.../video1.rgb"],
      "sha256": "164804d9...",
      "size_bytes": 6051840
    }
  }
}

url (singular string) is accepted as shorthand for a one-element urls.

Precedence: when both declare the same filename, the .lnk wins — a sketch can override one entry of a shared assets.json without editing it.

Reference implementation landed in FastLED/FastLED

ci/asset_resolver.py implements both formats and can serve as the spec for fbuild's Rust version:

• parse_lnk(path) / parse_assets_json(path) → RemoteAsset
• discover(data_dir) → merged list, .lnk overriding assets.json
• fetch(asset) → tries each URL in order, falls back on failure
• resolve(data_dir, force=False) → materializes, skipping assets already present and valid

Verification behavior worth mirroring exactly:

• size_bytes checked first (cheap), then sha256
• mismatch raises — never a silent fallback to a stale or partial file
• an existing file with a wrong digest is re-fetched, not trusted
• nothing is written when verification fails

Test coverage

ci/tests/test_asset_resolver.py — 17 tests, all passing, no network (uses file:// URLs). Covers both formats individually, both used together, .lnk precedence, URL fallback, sha256 mismatch, size mismatch, re-fetch of a corrupted existing file, and unknown-key tolerance.

End-to-end, against the live CDN

declared: video1.rgb  urls=2  sha256=164804d94802dd96...  size=6051840
materialized: examples/Fx/FxLedmapper32x32/data/video1.rgb
exists: True size: 6051840

Resolved from cdn.jsdelivr.net, digest and size verified. The materialized payload is gitignored via examples/Fx/FxLedmapper32x32/data/.gitignore — the declaration is committed, the payload never is.

[zackees]

Split out: #1356 covers storage-target descriptors (where an asset lands — littlefs/spiffs/sdcard/firmware) and the misconfiguration diagnostics that go with them, including the yellow-warning case of a .lnk that declares no destination.

Also worth narrowing this issue's scope: fbuild lnk pull already does sha256-verified fetch into a global blob cache, so the build-time resolution asked for here largely exists. What remains is the last mile — buildfs/uploadfs, partition config, and a LittleFS FsImpl in FastLED.

[zackees]

Tracked in the meta issue: FastLED/FastLED#3986 — remote assets: hosting, build-time resolution, and on-device filesystem deployment.

[zackees]

Closing — premise was stale; superseded by #1358

This issue asked for build-time resolution of remote example assets. That already exists in fbuild and did when the issue was filed; I hadn't read the source before writing it.

crates/fbuild-toolchain/src/lnk/mod.rs:1-22 documents the full pipeline:

  .lnk file (in source tree)
        -> [scanner]      walks tree, parses every *.lnk
        -> [resolver]     cache lookup by sha256; miss -> download; verify
        -> [materializer] hardlink/copy cached blob into build/resources/<rel>
        -> downstream build steps (objcopy, embed_files, ...)

And there is already a working consumer: crates/fbuild-build-esp/src/esp32/orchestrator/embed_stage.rs:38-46 resolves .lnk entries named in board_build.embed_files and objcopy-embeds them into firmware.

Verified against the live CLI:

$ fbuild lnk pull examples/Fx/FxLedmapper32x32
ok  .../data/video1.rgb.lnk -> ~/.fbuild/prod/cache/archives/lnk-blobs/.../<sha256>/video1.rgb
lnk pull: 1 ok, 0 failed (of 1)

$ fbuild lnk check examples/Fx/FxLedmapper32x32
lnk check: 1 ok, 0 missing, 0 mismatched (of 1)

sha256-verified fetch into a global content-addressed cache — which is most of what this issue asked for.

What actually remains, and where it moved

Gap	Tracked in
No filesystem-image consumer (buildfs/uploadfs)	#1358
No destination/storage-target concept, and no diagnostics for misconfiguration	#1356
Two incompatible .lnk formats in the wild	#1357
No LittleFS backend in FastLED, so an image would be unreadable	FastLED/FastLED#3984

The .lnk spec discussion posted here is superseded by #1357, which has the fuller picture: the text format has three implementations (C++ runtime, Python scanner, and the committed examples/AudioUrl file), not the single stray file I originally described.

Closing as superseded. Umbrella: FastLED/FastLED#3986.