Repository navigation
feat: stem storage format, MP3 128 kbps by default (#783) - #822
Merged
Merged
Conversation
Settings and the API gain stem_format (wav | mp3) and stem_mp3_kbps (128 | 192 | 256), whitelisted and validated together so one bad value changes nothing. STEMDECK_STEM_FORMAT and STEMDECK_STEM_MP3_KBPS seed the defaults for Docker and Unraid; a saved choice wins, as for separation_quality. The default is one constant in config.py.
…eader A last pipeline stage (stem_storage.py) runs once nothing reads the stems any more. With MP3 selected it encodes each stem with libmp3lame CBR into a temporary file, checks that ffmpeg and libsndfile both decode it to the WAV's exact frame count and that its loudest block lines up at lag 0, and only when every stem passed moves the MP3s into place and deletes the WAVs. Any failure keeps the WAVs and records the song as WAV; a cancel cleans up and raises. The song is never failed. app/core/stem_files.py resolves a stem by the file on disk, WAV first (a WAV song's cached download MP3 never shadows its WAV), and every reader of a finished song goes through it: the stem, region, mixdown, zip, backing, vocal envelope, peaks, take length, loudness, fingerprint, lyrics timing, vocal split (handed a decoded WAV), recovery and the sweep, which would otherwise have deleted an unregistered MP3 song. Each job records stem_format and stem_mp3_kbps. The identify regression tests pin WAV: their stubbed stems are placeholder bytes.
The chunked engine streams PCM out of WAVs by Range request and cannot read an MP3. A song whose stem URLs end in .mp3 opens on the full-decode engine instead (desktop and phone), whose decodeAudioData honours the LAME header, so the stems stay sample-aligned with the click and takes. Over the full-decode RAM ceiling it plays on the streaming path, as a fulldecode-mode song always did. Lane downloads are named after the stored format, which the desktop app saves the bytes under.
…rary A Stem files group with the format select (MP3 preselected on a fresh profile, WAV) and the bitrate select (128, 192, 256 kbps), shown only for MP3. Saved through /api/settings and applied to the next song that finishes, no restart. The search skips a row its section hides. Strings in all ten tables (and the pt-PT group title), with Guide sentences for both rows.
… checks seed.py adds e2e0f00dface, the fixture track stored as MP3 the way the storage stage leaves a song. stem-storage.spec checks the Settings rows (MP3 128 preselected, bitrate hidden for WAV, saved across a reload, found by search), that Chromium's decodeAudioData gives the WAV's samples back from the MP3 (same length, lag 0), and that the song plays on the full-decode engine, downloads as .mp3, runs the click, exports, and plays in Karaoke and Performance. Closes #783
thcp
marked this pull request as ready for review
October 20, 2026 13:32
thcp
added a commit
that referenced
this pull request
Oct 22, 2026
…g-release (#836) Closes #835 ## What changed - **PR to `main`, push to `main`, release:** full suite, as before. - **PR to any other base:** smoke + changed specs + mapped specs. A hub module (former `full` source entries: `main.js`, `state.js`, `mixer.js`, `catalog.js`, `index.html`, `app/api/jobs.py`, `stems.py`, `app/core/**`, ...) or an unmapped file under `static/` or `app/` now adds a fixed **core set** (17 specs) instead of everything. `uv.lock` and `pyproject.toml` add core too. - **Full suite on a non-main PR** only for a change to how the suite runs: `playwright.config.mjs`, `tests/e2e/helpers*`, `serve.sh`, `seed.py`, `package*.json`, the three e2e workflows, the selector, `spec-map.json`, or unmapped code under `tests/e2e/`. - **New `e2e-full.yml`:** full suite on every push to `big-release`, nightly (04:17 UTC) on `big-release`, and `workflow_dispatch` with a `ref` input. - **New reusable `e2e.yml`** (`workflow_call`, inputs `mode`, `base-ref`, `checkout-ref`) holds the steps once. Same pinned SHAs, `permissions: {}`, no secrets. - **Concurrency:** a new push to a PR cancels that PR's older e2e run. Pushes, releases, nightly and manual runs are never cancelled. `timeout-minutes: 30` per shard. ## What did not change - Application code, specs, `uv.lock`, package files. Three shards, `--pass-with-no-tests`. - Check names gain a prefix: `frontend-e2e / frontend-e2e (1)` to `(3)`. A reusable workflow always prefixes the caller job. No branch protection or ruleset requires these names today (checked). ## Real diffs, before and after | Diff | Files | Before | After | |---|---|---|---| | #806 (queue.js, catalog.js) | 5 | FULL 105 | 30 (core), includes queue-progress | | #820 | 7 | FULL 105 | 59 (core) | | #821 | 2 | 47 | 47 | | #823 | 8 | 5 | 5 | | #828 (uv.lock) | 4 | 5 | 22 (core) | | #819, #830, #831 | 7 to 19 | FULL 105 | FULL 105 (edit `spec-map.json`) | | #829 | 19 | FULL 105 | FULL 105 (edits `helpers.mjs`) | | #822 | 33 | FULL 105 | FULL 105 (edits `seed.py`) | | `main...big-release` | 331 | FULL | FULL (PR to main) | `npx playwright test --list <selected>` matches the selector exactly for #806 (30 files, 200 tests) and #820 (59 files, 465 tests). Full suite: 724 tests in 105 files. Core + smoke alone: 159 tests in 22 files. ## Trade-off A non-main PR can pass while a spec outside its core and mapped sets breaks. The push to `big-release` run catches it on the merge commit, minutes later. ## Limits - **The nightly schedule only starts once `e2e-full.yml` reaches `main`.** GitHub runs scheduled workflows from the default branch, and the job checks out `big-release` explicitly for that reason. - Neither concurrency nor the timeout helps a hosted runner GitHub never assigns (queue time does not count toward the timeout). Fix by hand: re-run the job, or `gh run cancel <id>` then `gh run rerun <id> --failed`. - This PR changes `ci.yml`, so its own CI runs the full suite. Expected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #783.
What changed
STEMDECK_STEM_FORMATandSTEMDECK_STEM_MP3_KBPSseed it for Docker and Unraid (template rows added); a saved choice wins, as forseparation_quality. Whitelisted server-side, both values checked before either is kept; a non-object body is ignored (200), as Settings endpoints answer 500 when the body is not an object #797. Read when a job finishes: no restart.DEFAULT_STEM_STORAGEinapp/core/config.py. An existing install that never saved the setting gets MP3 for new songs only; the helper text says so.app/pipeline/stem_storage.py), last in the pipeline, after every stage that reads stems. Separation and all analysis still run on WAV. Per stem: encode to a temp file, then verify: ffmpeg and libsndfile both decode it to the WAV's exact frame count, and its loudest 10 s block lines up at lag 0 (cross-correlation, plus a ratio guard for steady tones). Only when every stem passed: rename into place, then delete the WAVs. Any failure keeps every WAV and records the song as WAV; cancel cleans up and raises; the song never fails over it.app/core/stem_files.py):stem_pathpicks<name>.wav, else<name>.mp3, by what is on disk. WAV first, because a WAV song can hold a cached download<name>.mp3(GET/stems/{name}.mp3) that must never shadow it.decoded_wavhands the vocal split model a decoded WAV. Each job recordsstem_format/stem_mp3_kbps(state, registry, metadata.json, recovery)..mp3opens on the full-decode engine (desktop and phone); the chunked engine only streams WAV. Lane downloads are named.mp3.What did not change
.wavas before.uv.lockunchanged.source.<ext>of an upload is never touched.original.wavinstems/is the complement backing track, a stem, so it is stored like the others.Audit of
.wav(129 lines in 23 backend files on big-release)stem_path(stems, region, mixdown, mode lanes, zip, backing/rest, vocal envelope, peaks, take length, loudness, fingerprint, lyrics timing and stamp, vocal split, recovery, the TTL sweep)/stems/{name}.wavserves either format with its own type and file namestatic/js: 64 matches, almost all.wave*CSS classes. Real WAV assumptions: the chunked engine (WAV-only by design, routed around) and the lane download name (fixed).Two real bugs the audit found and this fixes: the TTL sweep's
_has_stems_on_disklooked only for.wav, so it would have deleted an unregistered MP3 song; and the desktop app saves a lane download under the page's name, which said.wavover MP3 bytes.Sample alignment (measured)
Broadband test signal and a real separated job, round trip WAV -> MP3 -> decode, all three bitrates:
-ss(region exports), 5 offsetsdecodeAudioData(Playwright, seeded MP3 job)No compensation needed: libmp3lame's LAME/Xing header (
-write_xing 1, explicit) carries delay and padding and all three decoders trim by it. Tolerance is 0 samples. Guards:test_mp3_storage_is_sample_aligned_in_ffmpeg_and_libsndfile(128/192/256),test_mixdown_from_mp3_stems_lines_up_with_one_from_wavs,test_a_shifted_output_keeps_the_wavs(576/1105/2000), and the Chromium spec.Not verified: Tauri WebKit on macOS and Linux, Safari (no WebKit project in the suite). The storage stage cannot check the browser's decoder.
One finding: ffmpeg reading an MP3 from a pipe keeps the end padding (+479 samples here; start still aligned). No server reader pipes a stem; all pass a file path.
Real end-to-end (isolated server, 20 s sine upload, Demucs on this Mac)
.mp3, no.wav.wavBoth: every stem 882000 frames, mixdown wav/mp3, region, zip wav/mp3, master loudness and mastered export, EQ export, click + count-in export, backing and rest, vocal envelope, peaks, a take recorded through the API (length = song, lands on its sample), mixdown with the take. Playback, mixer, click, Karaoke and Performance checked in Chromium on the seeded MP3 song.
Tests
tests/test_stem_storage.py(51), covering settings/env precedence and validation, the resolver, every failure mode (no libmp3lame, full disk, corrupt output, short output, shifted output, failed move, stall, cancel mid-encode, unexpected error), atomicity, and the readers above.tests/test_identify_regressions.py(its stubbed stems are placeholder bytes, so MP3 storage could only take its failure path there).stem-storage.spec9/9.npm run test:jsgreen (newstem-format.test.mjs). ruff, ruff format, bandit, i18n coverage clean.Risks
<audio>path: no click, no key change, not sample-locked. The phone decodes in full too.