research: normalize the .lnk format — two incompatible variants exist today

[zackees]

Summary

There are two mutually incompatible .lnk formats in the FastLED ecosystem today. The one file actually committed to FastLED cannot be read by the tool named lnk. This issue is to research and decide a single normalized format plus a migration path.

The split

Format A — plain text (FastLED#2284). Committed today at FastLED/FastLED:examples/AudioUrl/data/track.mp3.lnk on origin/master:

# .lnk asset link file — see FastLED issue #2284.
# The first non-comment line is the URL this asset resolves to.
# Lines starting with '#' and blank lines are ignored.
# Additional lines (metadata like sha256=, fallback=) are reserved for
# future use and are ignored by the v1 parser.
https://www.soundhelix.com/examples/mp3/SoundHelix-Song-1.mp3

Format B — JSON v1 (fbuild crates/fbuild-toolchain/src/lnk/format.rs). What fbuild lnk add emits and fbuild lnk pull consumes:

{
  "sha256": "164804d94802dd96cd8cd37f7528ca6d30f8fd9396e812fe67ec6164bbdd2282",
  "size": 6051840,
  "url": "https://raw.githubusercontent.com/FastLED/assets/main/examples/FxLedmapper32x32/video1.rgb",
  "v": 1
}

format.rs::from_raw parses JSON only and hard-rejects anything else, so Format A fails outright under fbuild lnk pull / check.

Why this matters now

• examples/AudioUrl is on origin/master in Format A. Any fbuild lnk pull over that tree errors on it.
• New work (examples/Fx/FxLedmapper32x32) uses Format B.
• A third variant was independently drafted during design work — key=value with repeated url= for fallbacks — which converged on Format A's reserved sha256= / fallback= keys rather than Format B. That three-way convergence is evidence the format was never actually settled.
• FastLED/fbuild#1356 (storage targets and diagnostics) depends on knowing which format it is annotating. This should probably be decided before that design lands.

Research needed

Prior art is genuinely split, which is why this needs research rather than a coin flip:

• git-LFS uses a text key-value pointer — and it is the closest analogue to this use case:

  version https://git.lfs.github.com/spec/v1
  oid sha256:4d7a214614ab2935c943f9e0ff69d22eadbb8f32b1258daaa5e2ca24d17e2393
  size 12345
  

  Deliberately line-oriented so it diffs cleanly and merges predictably in git.
• DVC uses YAML (.dvc files).
• Bazel uses Starlark declarations, not sidecar files.
• Nix uses expressions, not sidecars.

The git-LFS precedent is a real argument for text, and it is the system whose problem most resembles ours (pointer files checked into git, standing in for large binaries).

Questions to answer:

1. Text or JSON? JSON is trivially machine-parseable and already implemented in fbuild. Text diffs and merges better in git — the property git-LFS optimized for and the reason it rejected structured formats. Which matters more here, given these files live in git and are edited rarely but reviewed often?
2. Should the parser accept both? A tolerant reader (sniff { → JSON, else text) is cheap and eliminates the migration cliff. Does permanent dual support cost more in confusion than it saves in transition?
3. Migration path for examples/AudioUrl. Convert to the winner, or leave and make the parser tolerant? Note it also lacks a sha256 entirely, so converting it requires fetching and hashing the upstream MP3 — which is a third-party URL that could change.
4. What is the versioning story? Format B has v: 1 and format.rs:110-116 hard-rejects v != 1, so a v2 bump breaks every older fbuild on sight. Adding fields to v1 is safe (unknown keys are ignored by both LnkFileRaw and FastLED's ci/asset_resolver.py). Any normalization must not require a version bump, or must accept breaking older clients.
5. Is sha256 required or optional? Format A has none. Format B treats it as present-in-practice. A .lnk without a digest gets fetch but no integrity — should that be an error, a warning, or allowed?
6. Multiple URLs / fallbacks? Format A reserves fallback=. Format B has a single url. FastLED's ci/asset_resolver.py accepts a list for url. CDN-primary-with-raw-fallback is a real need (see feat: structured manifest.json index + CDN serving for FastLED assets assets#1), so the normalized format should have an answer.

Acceptance criteria

• A single normalized format is specified and documented
• examples/AudioUrl/data/track.mp3.lnk parses successfully under fbuild lnk check
• examples/Fx/FxLedmapper32x32/data/video1.rgb.lnk continues to parse
• A stated position on dual-format support (permanent, transitional with an end date, or never)
• FastLED's ci/asset_resolver.py and its tests updated to match
• Migration documented for anyone with existing .lnk files

Related

• investigate: storage-target descriptors and misconfiguration diagnostics for .lnk / assets.json #1356 — storage targets and diagnostics (depends on this)
• feat: resolve remote example assets at build time (sha256-verified, globally cached) #1354 — build-time resolution
• FastLED#2284 — origin of Format A
• ci/asset_resolver.py in FastLED/FastLED — reference parser, 19 tests

[zackees]

Tracked in the meta issue: FastLED/FastLED#3986 — remote assets: hosting, build-time resolution, and on-device filesystem deployment.

[zackees]

Research: the split is wider than this issue describes — and JSON was already deliberated

Investigated against current origin/master of both repos (FastLED 3de8d4f8d, fbuild 49ec69ce). Three corrections to the issue body.

1. The text format has THREE implementations, not one committed file

I originally found only examples/AudioUrl/data/track.mp3.lnk and assumed it was a stray. It is not — it is one artifact of a complete, shipped v1 asset pipeline (FastLED#2284):

Layer	Location	Role
C++ runtime	src/fl/stl/url.h:274 fl::parse_lnk(), :325 parse_lnk_with_metadata()	parses .lnk at runtime
C++ asset API	src/fl/asset/asset.h, asset.cpp.hpp:109	FL_ASSET("data/track.mp3") handles
Python build	ci/compiler/asset_scanner.py:85 _parse_lnk_content()	emits the WASM manifest; docstring says it "Mirrors the C++ fl::parse_lnk_with_metadata() logic"

ci/wasm_build.py:1712-1720 calls scan_sketch_assets() / write_manifest_json() on every WASM build. So the text format is live in the shipping product, not legacy cruft.

Its grammar (url.h:312-325, mirrored in asset_scanner.py:85-126): first non-comment line is the primary URL; subsequent key=value lines carry metadata; recognized keys are sha256 and fallback; unknown keys ignored for forward-compat. LnkMetadata (url.h:252-259) is explicit that v1 "parsed but not enforced... The runtime ignores everything except primary."

2. JSON-vs-text was already deliberated in fbuild, and git-LFS was explicitly rejected

crates/fbuild-toolchain/src/lnk/format.rs:27-31 documents the rationale:

## Why JSON
- human-readable + git-diff-able
- no external tooling needed (vs Git LFS pointer format)
- sha256 is mandatory -> reproducible builds + content-addressable cache
- extensible via `v` field if a v2 ever happens

So my framing of "git-LFS precedent argues for text" is already-considered ground, not a new input. That weakens research question 1 considerably — the fbuild side has a written position.

3. Two schema facts I got wrong

• sha256 is REQUIRED in fbuild's schema, not optional (format.rs:22 — "Required fields: v, url, sha256"). The text format treats it as optional and unenforced. This is the sharpest incompatibility: examples/AudioUrl/data/track.mp3.lnk has no digest at all, so it cannot become a valid fbuild .lnk without someone fetching and hashing a third-party MP3 whose bytes may change.
• There is an extract field (file | zip | tar.gz, format.rs:41-50) with no text-format equivalent. Any normalization has to account for it.

What this means for the decision

The two formats are not competing drafts — they serve different consumers with different guarantees:

• Text: runtime-parseable on an MCU (no JSON parser in the hot path), digest optional and advisory, mirrors declared via fallback=.
• JSON: build-time only, digest mandatory, content-addressable caching, archive extraction.

That reframes the question from "which format wins" to "should these stay two formats with a documented boundary, or does one absorb the other — and at what cost?"

Three real options:

1. Keep both, document the boundary. Text = runtime asset links; JSON = build-time blob pointers. Cheapest, but the shared .lnk extension guarantees continued confusion, and asset.h:13 already misdescribes reality ("At build time, fbuild scans <sketch>/data/ for *.lnk files" — fbuild's parser would reject the text files it is describing).
2. Tolerant reader on both sides. Sniff { -> JSON, else text. ~30 lines each in Rust and Python; C++ runtime would need a JSON path it currently does not have and arguably should not carry.
3. Converge on JSON, migrate the text pipeline. Cleanest end state, but requires a JSON parser in the C++ runtime (fl::parse_lnk is deliberately allocation-light), a digest for the AudioUrl asset, and touching a shipped v1 API.

My recommendation is option 1 plus a rename — distinct extensions for the two roles (e.g. keep .lnk for the runtime asset link, use something distinct for build-time blob pointers), which removes the ambiguity without forcing a JSON parser into the MCU runtime. Recording it as a recommendation, not a decision: this touches a shipped API in both repos and deserves a maintainer call.

Related discovery

src/fl/asset/asset.h:21-22 states: "v1 scope: WASM and host/stub only. ESP32 LittleFS / SD-card resolution is future work and intentionally omitted." That is independent confirmation of the gap tracked in FastLED/FastLED#3984 and #1358 — the asset pipeline was deliberately built without a device-filesystem path.

[zackees]

Parser side landed — #1362 merged

fbuild now reads both formats. from_str_any() sniffs the first non-blank, non-comment character ({ -> JSON, else text) and from_path() routes through it, so lnk pull, lnk check, and the ESP embed stage all gained the text format with no call-site changes. JSON stays canonical — fbuild lnk add is unchanged.

Paired with FastLED/FastLED#3990, which folded FastLED's three Python/C++ parsers into one scanner that accepts both .lnk forms plus assets.json. Both sides of the split now read both formats.

One deliberate asymmetry

sha256 remains mandatory in fbuild even for the text form, where the C++ runtime treats it as optional. fbuild's resolver caches by content digest, so a .lnk without one cannot be content-addressed or verified — relaxing that would give up the guarantee the cache is built on. The error is actionable instead:

text-format .lnk for <url> has no `sha256=` line; fbuild caches by content
digest and cannot verify or content-address without one. Add a
`sha256=<64 hex chars>` line, or regenerate the file with `fbuild lnk add <url>`.

Verified

• text-format .lnk fetched and sha256-verified into the blob cache
• a deliberately wrong digest rejected, .partial never promoted — the integrity path, not just the happy path
• 43 lnk tests pass (7 new), clippy clean

Caveat on how it landed: merged with --admin at the user's explicit instruction. The 98-job CI matrix was still running and got cancelled by the merge (92 cancelled, 2 passed), so CI did not validate this — the evidence above is all local.

Still open on this issue

The parser accepts both, but the underlying question this issue asks — should there be two formats — is unanswered:

1. examples/AudioUrl/data/track.mp3.lnk still has no sha256, so it now gets a clear error instead of a confusing one. Adding a digest means hashing a third-party MP3 whose bytes can change.
2. Long-term direction. My recommendation stands: keep both roles but give them distinct extensions — .lnk for the runtime asset link, something else for build-time blob pointers — rather than forcing a JSON parser into the MCU runtime. That is a maintainer call, not a parser change.
3. asset.h:13 still misdescribes reality ("At build time, fbuild scans <sketch>/data/ for *.lnk files") — truer now than before, but worth a pass.

Leaving open for those.