Skip to content

fix: prevent AVAssetWriter -16364 (InvalidTimestamp) and harden instant/pause recording paths - #2094

Merged
richiemcilroy merged 30 commits into
mainfrom
cursor/fix-writer-invalid-timestamp-809d
Aug 7, 2026
Merged

fix: prevent AVAssetWriter -16364 (InvalidTimestamp) and harden instant/pause recording paths#2094
richiemcilroy merged 30 commits into
mainfrom
cursor/fix-writer-invalid-timestamp-809d

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Aug 6, 2026

Copy link
Copy Markdown
Member

Problem

Two users on 0.5.8 hit the same mid-recording failure:

Task mux-video failed: Video muxer stopped accepting frames at frame 8342:
Failed to encode video frame: WriterFailed/The operation could not be completed
(frame #8339, ts=285.513558066s)

The recording aborts with an error dialog and the display track is unusable (no moov atom is ever written).

Root cause

Uploaded logs contain the full NSError: AVFoundationErrorDomain -11800 with underlying NSOSStatusErrorDomain -16364 — CoreMedia's InvalidTimestamp (duplicate/backwards PTS, reported asynchronously a few appends late). MP4Encoder::queue_video_frame enforced monotonicity on nanosecond-precision Durations while the writer receives microsecond-truncated PTS; two frames inside the same microsecond pass the guard and collapse into duplicate writer PTS. The trigger is a stall-recovery burst (the log shows a ~15s system-wide stall right before failure). Reached by macOS studio recordings with camera active (non-fragmented AVFoundationMp4Muxer), the camera track writer, and camera-only recordings. Not a 0.5.8 regression — the path is byte-identical since 0.4.7x-era code.

Fixes

  • enc-avfoundation: quantize video PTS to whole microseconds before the monotonic tie correction — the guard now operates in the units the writer sees; emitted PTS are strictly increasing integral microseconds with non-overlapping extents.
  • enc-avfoundation: hold the pending frame across pause() instead of flushing with nominal duration, which put the first post-resume frame inside the flushed sample's extent (the sporadic overlapping-extents writer-failure shape). Stop-while-paused still flushes via finish_start; a container-duration assertion proves the muxed timeline is unchanged.
  • enc-avfoundation: shift a held frame's deferred offset when a pause gap is consumed (found by adversarial review): a tie-bumped frame held across a pause carried a stale offset snapshot that, applied on append after resume, overwrote the gap-adjusted timestamp_offset and silently shifted all later timestamps.
  • recording (macOS): disk-exhaustion guard for all AVAssetWriter modes, not just instant. Previously a studio or camera recording filling the disk killed the writer on a failed async write (file unrecoverable); now the encoder thread stops while the writer is alive so finish() preserves the recording.
  • mediafoundation-ffmpeg (Windows): strictly monotonic muxer PTS in stream ticks. A muxer-path audit found the same unit-mismatch class: MF stamps samples in 100ns ticks, the stream time base is ~333× coarser, and nothing guarded the re-quantization — surfacing as dropped packets in the hardware encoder path. One guard at the writer-visible unit closes it for every MF consumer.
  • Diagnosability: QueueFrameError::WriterFailed and all four fatal-message sites now debug-format the NSError, so dialogs and logs carry the code and NSUnderlyingError instead of "The operation could not be completed".

Audits (documented, no action needed)

A full audit of every muxer/encoder path for the unit-mismatch class found all other shipping paths safe: the ffmpeg H264 stack guards in tick space (normalize_input_pts) with warn-and-continue containment; the OOP muxer's guard runs in the same tick unit that crosses the process boundary; all audio encoders funnel through one guarded base; win_segmented{,_camera}.rs contain the defect but are dead code. Error-propagation audit: only the display/screen pipeline is fatal to a recording; mic/camera/system-audio failures degrade.

Test coverage (real encoders, real files, wired into CI)

  • enc-avfoundation (macOS CI, real AVAssetWriter): same-microsecond pair bumped apart, same-microsecond bursts survive, pause/resume with resume-tie keeps extents disjoint + container duration intact, stop-while-paused flushes the held frame. Queue calls use a writer-ready retry helper mirroring production so paravirtualized runners (no hardware VideoToolbox) can't flake the suite.
  • enc-ffmpeg (all platforms): stall-recovery burst with same-microsecond timestamps, exact duplicate, and backwards blip must encode every frame with strictly monotonic PTS and survive the production remux + decode probe.
  • cap-recording lib (all platforms): SharedPauseState excision/no-frame-pause/accumulated-cycles/backwards-resume coverage.
  • instant_mode_scenarios (all platforms, newly wired into sync-tests.yml): full-pipeline pause/resume excision on both tracks and stall-burst A/V alignment, ending in validate_instant_recording uploadability. Four rotted tests repaired (segments only cut at keyframes; the harness now marks I-frames at the segment cadence and compares assembled media durations, not manifest estimates).
  • Hardware harnesses (real screen/mic on developer Macs): hardware_instant_recording gated to macOS (it broke every non-macOS test build via ungated imports) and extended with a real pause/resume cycle; new hardware_studio_recording exercises the exact field-failure path (non-fragmented AVFoundation display writer) with pause/resume and per-segment validation. sync-tests.yml now compile-checks all test targets so harnesses can't rot again.

CI repairs (sync-tests was failing on every branch, including main's nightlies)

  • cap-rendering notch golden tests: the windows-2022 runner image update broke WARP compositing with no repo change (passing Aug 4, failing since Aug 5, blocking every PR). Shape assertions stay at full strength on any adapter that renders (hardware everywhere, Ubuntu's lavapipe); a software adapter that fails render sanity skips loudly.
  • Sync matrix: above real-device delivery rates, bounded encoder-overload drops are tolerated but every muxed pts is verified against the nearest sent timestamp (timestamp bugs still fail; runner throughput doesn't). Heavy over-delivery cases get a 0.25s relative tolerance on top of the drift tracker's designed 0.1s wall-clock re-pinning. A per-frame emission-lateness guard skips cases the runner stalled through (>0.1s), the invisible-to-end-lag shape that failed the plain 30fps case with a 0.307s error.

Verification — full board green

Final run: 27 checks, 0 failed — A/V Sync Tests green on macOS, Windows, and Ubuntu (the first fully green sync-tests run on any branch since Aug 4), Clippy -D warnings green on macOS and Windows, Format/Typecheck/Biome/CodeQL green. macOS ran cap-enc-avfoundation 44/44 against the real AVAssetWriter, instant_mode_scenarios 61/61, lib 248/248; the matrix passed with one case skipped by its own pre-existing lag guard. Locally: 41-case matrix green with zero skips, cap-enc-ffmpeg 50/50, cap-recording --lib 237/237, instant_mode_scenarios 61/61. Property check across 200 randomized stall/burst scenarios × 20k frames: pre-fix 243,989 duplicate writer-PTS pairs, post-fix 0.

Follow-up candidates (out of scope)

  • Durable salvage of a non-fragmented AVAssetWriter after a non-timestamp writer failure (movieFragmentInterval or segment rotation).
  • Video manifest total_duration under-reports a tail ending between keyframes (bookkeeping estimate only; media is complete).
  • Dead code removal: win_segmented{,_camera}.rs, WindowsOOPFragmentedM4SMuxer.

Post-review hardening (second adversarial pass, 2026-08-07)

A two-reviewer adversarial pass over this branch confirmed the core -16364 fix airtight (independent traces of the monotonicity invariant, bump-chain bounds, backwards-blip and finish-while-paused edges all came back clean) and found four things worth fixing before merge, all now in:

Fixes

  • Camera writer finalization (the one real blocker). AVFoundationCameraMuxer::finish skipped encoder.finish() whenever the encoder thread returned an error — so the new disk guard (and the pre-existing WriterFailed exits) produced exactly the moov-less unplayable camera.mp4 the guard's comment promised to prevent. Finalization now runs whenever the thread has exited (mutex free; a panicked thread leaves it poisoned, which the lock arm already handles) and is skipped only on a finish-wait timeout, matching the screen writer's best-effort semantics.
  • Disk guards unified on the platform standard. The AVFoundation muxer threads hard-stopped at 200MB — 4x above the shared 50MB stop used by every fragmented muxer — with no health event, so a studio recording on a low-f_bavail disk (APFS purgeable space is excluded) died at ~10s behind a generic encode error. Both threads now poll the shared DiskSpaceMonitor (warn 200MB → DiskSpaceLow, stop 50MB → DiskSpaceExhausted, both surfaced to the user and telemetry via newly wired SharedHealthSenders), and the 500MB start preflight covers studio and camera-only recordings instead of instant only.
  • Telemetry truncation panic. truncate_reason used byte-indexed String::truncate(240); the switch to {err:?} NSError formatting made 300-600-byte reasons with multi-byte localized text routine, and a mid-codepoint cut panics the desktop process mid-recording on non-English macOS. Now char-boundary safe.
  • Windows stuck-clock visibility. The new MF monotonic guard silently rewrites every pts if a source's sample time stops advancing (timeline compresses one tick per frame). Consecutive bumps now warn at 30.

Coverage the review found missing

  • The deferred-offset gap shift (commit 8ae79598a) had zero test coverage — both new pause tests take the None branch. A new regression drives a tie-bumped held frame through a second pause; mutation-verified (fix disabled → timestamp_offset 2.07s vs expected 3.03s, container stretches ~1s).
  • A default-config segment-cadence test pins segment cuts to the encoder's own keyframe interval, since the scenario helpers' forced I-frames can't catch a GOP-option regression.

Sync-matrix honesty (the skips can't hide bugs now)

  • Mid-emission lateness is measured only after the pipeline consumer exists, so build-window backpressure can't masquerade as a runner stall; genuine skips are budgeted (>50% of the matrix skipping fails the run) and labeled SKIP in the CI summary.
  • The overload drop-tolerance branch now also enforces span and gap preservation (gap collapse was the 0.5.4 desync class) and budgets nearest-match reuse at 10% so burst clustering can't score as zero error.
  • Count mismatches name the missing sent indices ("missing sent indices: 31-39"), which turned a three-run flake hunt into a one-look diagnosis: the loss sits immediately after the first 2s segment boundary, a one-time cold-system stall (VideoToolbox bring-up + first DASH segment write) that never reproduces warm. The warm-up now runs past the first segment cut, and a failed video case retries once with a loud "passed on retry after cold-start failure" label — real regressions reproduce and still fail twice.

Verification on top of the existing board

  • Full local suite on Apple Silicon: cap-timestamp, cap-enc-ffmpeg 51/51, cap-recording lib 237/237, instant scenarios 61/61, cap-enc-avfoundation 45/45 against the real AVAssetWriter, cap-rendering, cargo check --tests, clippy -D warnings on all touched crates, desktop cargo check.
  • Sync matrix: 75/75 with 40 random cases (two labeled environment skips on 711/866fps synthetic over-delivery), plus fixed-set reruns.
  • The two hardware harnesses could not run on the verification machine: ScreenCaptureKit returns 0 displays to that terminal's process tree (lapsed monthly screen-recording re-approval on macOS 26; CGPreflightScreenCaptureAccess still reads true) — environment, not code. They pass on the authoring machine per the original description and should be re-run on any dev Mac with a fresh grant: cargo test -p cap-recording --test hardware_instant_recording / --test hardware_studio_recording.

Third review pass (two fresh adversarial reviewers, 2026-08-07 PM)

A second independent two-reviewer pass over the full branch re-confirmed the production fix from scratch: one reviewer reconstructed the pre-fix bug from main and proved the new invariant by induction (strictly increasing whole-microsecond PTS, provably disjoint extents); the other proved timestamp_offset can never leave zero on an unpaused recording, so healthy recordings emit bit-identical PTS to pre-PR. The MediaFoundation guard was proven inert for healthy Windows recordings by direct time-base analysis (a tie needs frames ≤16.6µs apart against a 33,333µs cadence, and the pre-change behavior was a dropped frame). No production defect found. Resulting hardening, all test/docs-only:

  • Sync-matrix skip loopholes closed. Emission lateness now anchors on when the channel was last free, so a consumer-side stall regression fails its case instead of skipping it; the cold retry is scoped to its two cold-start signatures (count mismatch, stop timeout) rather than any error; retried passes now count toward the ≤50% degradation budget; the tight-pair comment states the real bound (post-dedup sent timelines are provably never tight, so the operative bound is the 5% absolute slack).
  • AVFoundation test outputs namespaced per process — concurrent runs of the test binary clobbered shared fixed temp paths, failing AVAssetWriter init with "Cannot Save" mid-suite (both reviewers hit this independently).
  • Comments corrected: the camera finalize comment no longer claims finish_writing can salvage a writer already in Failed (finalization preserves the moov for disk-guard stops and mutex poison, which is the case the fix targets); the pause() comment documents the deliberate trade that a video-final pre-pause frame keeps a 1µs extent after the resume tie-bump (timeline length preserved, pinned by the container-duration assertions).
  • CI visibility: the cap-rendering step runs --nocapture so a WARP notch skip prints instead of looking identical to a pass.

Behavior changes the reviewers flagged for explicit sign-off (all deliberate, restated here so they are chosen rather than incidental):

  • The 500MB start preflight is now live for studio and camera-only recordings. On main it was dead code — the only AVFoundationMp4Muxer construction site passes instant_mode: false, so no AVFoundation path had any live disk guard at all. A Mac under 500MB reported-free (APFS f_bavail, which excludes purgeable space) now refuses to start instead of dying mid-recording near disk exhaustion.
  • The camera muxer's frame-drop tracker now has a live health sender, so sustained >5% camera frame drops surface the existing Degraded banner (previously silent).

Re-verified after the hardening on a quiet machine: sync matrix 41/41 isolated, cap-enc-avfoundation 45/45 against the real AVAssetWriter, cap-enc-ffmpeg 51/51, cap-recording lib + instant_mode_scenarios 61/61, cargo check --tests + cargo fmt --check on touched crates.

Additional follow-up candidates from this pass (out of scope):

  • Mirror the camera muxer's TimedOut gate in AVFoundationMp4Muxer::finish: wait_for_worker collapses Failed/TimedOut, so the screen path locks the encoder even after a timeout, and video_thread_timed_out is also set for plain failures. Reviewers disagreed on severity (the thread exits when the channel closes; only an OS-level wedge inside an append leaves the mutex held) — pre-existing either way.
  • Structured NSError fields (code, underlying code) in telemetry instead of relying on the 240-byte truncated Debug string, whose userInfo ordering makes the -16364 survival nondeterministic.
  • MF h264.rs: clamp packet duration on a tie-bump so stuck-clock output stays self-consistent; frame_count is never incremented so finish() always logs "wrote 0 frames".
Open in Web Open in Cursor 

Greptile Summary

The PR hardens macOS and Windows recording timestamp handling, preserves pending AVFoundation frames across pauses, expands low-disk protection, improves native writer diagnostics, and adds extensive regression and hardware-oriented coverage.

  • Quantizes AVFoundation video timestamps before monotonic correction and coordinates deferred offsets across pause gaps.
  • Enforces writer-visible monotonic timestamps in the Media Foundation muxer.
  • Extends AVFoundation disk-space checks to studio and camera recording paths.
  • Adds pause/resume, stall-recovery, container-validity, and CI compile coverage.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete changed-code defect identified.

The timestamp normalization, pause-offset coordination, low-disk shutdown paths, and expanded test coverage are consistent with their production callers and lifecycle behavior.

Important Files Changed

Filename Overview
crates/enc-avfoundation/src/mp4.rs Aligns monotonic correction with AVAssetWriter’s microsecond timescale and preserves pending-frame timing across pause/resume without an identified actionable regression.
crates/mediafoundation-ffmpeg/src/h264.rs Adds stream-tick PTS/DTS normalization for Media Foundation output; current no-B-frame configuration supports the shared timestamp assignment.
crates/recording/src/output_pipeline/macos.rs Expands periodic disk-space protection and improves NSError diagnostics across AVFoundation screen and camera writers.
crates/recording/src/output_pipeline/core.rs Adds focused tests for pause excision, repeated cycles, and anomalous resumed timestamps without changing production behavior.
crates/enc-ffmpeg/src/mux/segmented_stream.rs Adds end-to-end regression coverage for duplicate, backward, and same-microsecond recovery-burst timestamps.
crates/recording/tests/instant_mode_scenarios.rs Adds full-pipeline pause and stall scenarios while repairing segment tests to respect keyframe boundaries and actual assembled-media durations.
.github/workflows/sync-tests.yml Runs instant-mode scenarios and compile-checks all recording test harnesses across the CI matrix.

Reviews (1): Last reviewed commit: "test(recording): add real-hardware studi..." | Re-trigger Greptile

Context used:

… tie correction

AVAssetWriter receives PTS as whole microseconds (1MHz SampleTimingInfo),
but remapped capture timestamps carry nanosecond precision. During
stall-recovery bursts two frames can land inside the same microsecond:
they pass the nanosecond-space monotonicity guard yet collapse into
duplicate writer timestamps, which AVAssetWriter reports asynchronously a
few frames later as -11800/-16364 (InvalidTimestamp), aborting the whole
recording. Field logs from 0.5.8 (studio mode + camera, the non-fragmented
AVFoundation muxer path) show exactly this failure at 285s and 103s.

Truncate the PTS to whole microseconds before the monotonic tie
correction so the guard operates in the units the writer sees, and
surface the NSError code/domain/underlying error in WriterFailed
messages and append-site logs so future reports are diagnosable.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
cursoragent and others added 6 commits August 6, 2026 13:02
…mple extents disjoint

Flushing at pause wrote the pending frame with the full nominal duration,
so the first post-resume frame (tie-corrected +1us) landed inside that
sample's extent. Overlapping extents are the sporadic AVAssetWriter
failure shape reproduced by the overlapping-extents tests. Holding the
frame until resume writes it with the real clamped forward gap instead;
stop-while-paused still flushes it via finish_start.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
…t the segmented encoder

The 0.5.8 field-failure timeline (nanosecond-precision timestamps, a
multi-second stall, then backlogged frames landing hundreds of
nanoseconds apart, plus an exact duplicate and a backwards blip) must
encode with strictly monotonic PTS, no dropped frames, and survive the
production remux + decode probe.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
…pause/resume

The harness drives cidre/ShareableContent and the macOS builder
signature, so it never compiled on Linux and broke the whole
cap-recording test suite there. Gate it to macOS and extend the real
recording flow with a mid-recording pause/resume cycle, with duration
bounds tight enough to fail if the pause leaks into either timeline.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
…nt pipeline

SharedPauseState gets direct unit coverage (excision, no-frame pauses,
accumulated cycles, backwards resume timestamps), and the instant-mode
scenario harness gains two full-pipeline cases: a paused-and-resumed
recording whose output must excise the pause identically on both tracks
and stay uploadable, and a stall-recovery burst with same-microsecond
timestamps that must keep A/V aligned and uploadable.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
…ncoders

Four scenarios rotted because they never ran in CI: the DASH muxer only
cuts segments at keyframes and the encoder pins a 2s GOP (libx264
honors keyint_min strictly, hardware encoders emit extra IDRs), so
sub-GOP segment durations produced platform-dependent segment counts.
Mark source I-frames at the segment cadence so the counts are
deterministic everywhere, and compare assembled media durations instead
of manifest bookkeeping totals: a tail that ends between keyframes is
appended into the previous segment file, so the manifest's estimated
total under-reports while the assembled output carries the full
content.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
The scenario harness (assembly, validation, pause/resume excision,
stall-recovery bursts) never ran in CI, which is how four of its tests
rotted unnoticed.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
@cursor cursor Bot changed the title fix: prevent AVAssetWriter -16364 (InvalidTimestamp) killing recordings mid-stream fix: prevent AVAssetWriter -16364 (InvalidTimestamp) and harden instant/pause recording paths Aug 6, 2026
cursoragent and others added 4 commits August 6, 2026 14:13
…umed pause gaps

Holding the pending frame across pause created the first window where
timestamp_offset can change (pause-gap consumption) between a deferred
offset being snapshotted and applied: a tie-bumped frame held across a
pause would, on append after resume, overwrite the gap-adjusted offset
with its stale pre-pause snapshot and silently shift every later video
and audio timestamp forward by the gap. Shift the held snapshot when
either path consumes a gap so apply-on-append stays correct.

Also verify by container duration that a held-frame pause leaves the
muxed timeline untouched, retry writer-busy queues in the regression
tests (paravirtualized CI runners have no hardware VideoToolbox, so
single-shot queue calls flake), and wait for input readiness before the
finish-time flush in the stop-while-paused test.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
…-format writer failures

The critical disk-space check only ran for instant mode, so a studio or
camera recording filling the disk killed the AVAssetWriter on a failed
async write and lost the moov (unrecoverable file). Check in every mode
and stop while the writer is alive so finish() preserves the output.

The four fatal-message sites destructured the NSError and
Display-formatted it, which hides the code and NSUnderlyingError that
identify failures like -11800/-16364; debug-format them so dialogs and
logs carry the full error.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
…eam ticks

MediaFoundation stamps samples in 100ns ticks but the stream time base
is ~333x coarser (1/(fps*1000)): two strictly increasing sample times
can quantize onto the same output tick and the mov muxer rejects the
duplicate — the same unit-mismatch class as the AVFoundation -16364
failures, currently surfacing as dropped packets in the hardware
encoder path. Bump ties one tick in the writer-visible unit, like
normalize_input_pts in cap-enc-ffmpeg.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
…mpile all harnesses in CI

The non-fragmented AVFoundation display writer (the 0.5.8 field-failure
path) had no real-environment coverage: record the primary display
through the real studio actor with fragmented(false), pause and resume
mid-recording, and verify each segment's display.mp4 is a finalized,
decodable MP4 with the expected content duration. sync-tests now also
compile-checks every cap-recording test target so hardware harnesses
can't rot into non-compiling again.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
@richiemcilroy
richiemcilroy marked this pull request as ready for review August 6, 2026 15:24
@cursor

cursor Bot commented Aug 6, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

cursoragent and others added 2 commits August 6, 2026 15:39
…not composite

The windows-2022 runner image update broke WARP compositing under the
notch golden tests with no repo change (passing Aug 4, failing every run
since Aug 5), blocking every PR that triggers sync-tests. Keep the shape
assertions at full strength on any adapter that can actually render —
hardware everywhere, and software rasterizers like Ubuntu's lavapipe —
and skip loudly only when a software adapter fails the basic sanity of
clearing to white and drawing anything at all.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
… in the sync matrix

Shared runners cannot real-time-encode several hundred fps of synthetic
worst-case content, and the muxer's stall budget deliberately drops
frames rather than block capture, so exact frame-count equality above
real-device delivery rates asserts runner throughput, not timestamp
correctness — the shape behind every matrix failure on main's nightly
runs. Above 240fps delivered, allow bounded drops but verify every muxed
pts against the nearest sent timestamp so timestamp bugs still fail.

The heavy over-delivery cases also ride on the drift tracker's designed
wall-clock re-pinning (0.1s cap), leaving a 0.15s relative tolerance
only 50ms of scheduler headroom at 1000 timed emissions per second —
the macos-latest runner failed the curated 1000fps case at exactly
0.150s. Widen the relative tolerance to 0.25s for such cases only; the
bug class this matrix guards produces errors of a second or more.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
…ission

The end-of-emission lag guard misses a stall that later catches up, but
the contamination is the same: frames stamped with scheduled capture
times arrive late and the pipeline's designed wall-clock re-pinning
moves muxed pts by roughly the stall size — macos-latest failed the
plain 30fps steady case with a 0.307s error from exactly this. Measure
per-frame emission lateness directly and skip loudly past 0.1s; real
timestamp bugs reproduce on healthy runners, a stalled runner proves
nothing either way.

Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
Co-authored-by: Richie McIlroy <richiemcilroy@users.noreply.github.com>
…or and preflight every mode

The muxer-thread guards hard-stopped at 200MB — 4x above the platform-wide
stop threshold — and emitted no health event, so a studio or camera
recording on a low-f_bavail disk (APFS purgeable space is excluded from
statvfs) died within 10s behind a generic encode-failure message while
every fragmented muxer warns at 200MB and stops at 50MB with
DiskSpaceLow/DiskSpaceExhausted surfaced to the user and telemetry.

Route both AVFoundation encoder threads through the shared
DiskSpaceMonitor with a SharedHealthSender wired into each muxer (the
camera muxer had no health sender at all), and extend the 500MB start
preflight from instant-only to studio and camera-only recordings so a
full disk is a clean refusal instead of a 10-second recording.
…rors

AVFoundationCameraMuxer::finish skipped encoder.finish() whenever the
encoder thread returned an error, leaving camera.mp4 without a moov —
exactly the loss the disk guard exists to prevent, and the same outcome
for the pre-existing WriterFailed and poisoned-mutex exits. A thread that
has exited holds no encoder mutex (a panicked one leaves it poisoned,
which the lock arm below already handles), so finalizing is always safe
there; only a finish-wait timeout — the thread may still be mid-append —
skips finalization now, matching the screen writer's best-effort
semantics.
Reasons now carry debug-formatted NSErrors whose localized descriptions
are multi-byte; the byte-indexed String::truncate at 240 panics whenever
the cut lands mid-codepoint, which on CJK-localized macOS is the common
case. First became reachable when WriterFailed messages switched from
Display to Debug formatting, and a telemetry panic mid-recording is the
exact failure this branch exists to prevent.
The monotonic guard silently rewrites every pts when a source's MF sample
time stops advancing, compressing the muxed timeline one stream tick per
frame. Count consecutive bumps and warn at 30 (and every 300th after) so
a stuck capture clock shows up in logs instead of only as a mysteriously
short recording.
… pause

Both existing pause tests take the deferred_offset = None branch, so the
gap-shift fix (a tie-bumped frame held across a second pause) had zero
coverage — a regression there would ship silently. The new regression
drives a resume-tie frame (deferred_offset = Some) through a second pause
cycle; with the shift disabled the stale snapshot re-inserts the pause
into the mapping (timestamp_offset 2.07s instead of 3.03s, verified by
mutation) and every later timestamp jumps forward ~1s. Also note the
held-frame pixel-buffer retention in the pause() comment.
…frame interval

Production always runs segment_duration equal to the encoder GOP, so
segment cutting depends on the encoder emitting keyframes at its
configured cadence with no caller forcing I-frames. The scenario helpers
force I-frames at sub-GOP cadences for cross-encoder determinism, which
also makes them blind to a GOP-option regression (g/keyint_min or the
default interval) — this test encodes 6.6s at the untouched default
config and requires ~3 segments and the full assembled duration.
…label CI skips

DeviceType::Cpu also matched Ubuntu's lavapipe, so a regression that
draws nothing could skip on two of three CI legs at once. Scope the
escape to the WARP family by name: lavapipe (which renders correctly)
keeps full-strength shape assertions, so a real do-nothing regression
fails on at least two legs. The sync-tests job summary now labels
environment skips SKIP instead of PASS so they are auditable at a
glance.
The environment escapes added for runner tolerance could also hide real
bugs; close the gaps and make every escape auditable:

- Measure mid-emission lateness only once the pipeline consumer exists.
  The emitter starts before build so the builder can own the channel
  receiver; frames scheduled during the build window back up in the
  bounded channel and their lateness is structural, not a runner stall.
- Keep span and gap-preservation checks in the overload drop-tolerance
  branch (gap collapse is the 0.5.4 desync class; drops only widen gaps),
  and budget nearest-match reuse at 10% so a burst clustering many muxed
  frames onto one instant cannot score as zero error.
- Name the missing sent indices on a frame-count mismatch. This turned a
  three-run flake hunt into a one-look diagnosis: the local 15fps loss
  sits immediately after the first 2s segment cut.
- Cap environment skips at half the matrix and express the over-delivery
  tolerance as REL_TOLERANCE_SECS + DRIFT_REPIN_CAP_SECS.
- Run the warm-up past the first segment cut and retry a failed video
  case once with a loud 'passed on retry after cold-start failure' label:
  cold-system costs (VideoToolbox bring-up, first DASH segment write) hit
  inside the pipeline where no emitter guard can see them and never
  repeat warm, while a real regression reproduces and still fails twice.
- Optional RUST_LOG subscriber so pipeline drop warnings are visible when
  diagnosing a failing case.
…g nearest-match reuse

Ubuntu CI falsified the 10% reuse budget immediately: sorting a ±40%
jittered 835fps sent timeline produces legitimate sub-300us pairs, the
muxed timeline mirrors them, and ~20% of muxed frames nearest-map onto a
shared sent timestamp on both attempts of an honest case. Any fixed
budget loses to some random seed.

Burst collapse creates tight pairs the SENT timeline never had, so make
the claim relative: the muxed tight-pair rate (<0.25 delivered periods)
must not materially exceed the sent timeline's own rate (1.5x + 5
points). A real collapse pushes the muxed rate toward 100% against a
sent-mirrored baseline and still fails; jitter clusters appear in both
and pass. Verified against the exact failing Ubuntu seed
(1786101729252809648): the case passes, 41/41 green.
… retries

Lateness now anchors on when the channel was last free, so consumer-side
backpressure fails a case instead of converting it into a runner-stall
skip; the cold retry is scoped to its two cold-start signatures instead
of any error; retried passes count toward the degradation budget; the
tight-pair comment states the real bound (sent_tight is provably zero
post-dedup).
…the pause-hold extent trade

Concurrent runs of the test binary clobbered shared fixed temp paths,
failing AVAssetWriter init with "Cannot Save" mid-suite. The pause()
comment now states the deliberate trade: a video-final pre-pause frame
keeps a 1us extent after the resume tie-bump.
…ips in CI

finish_writing cannot salvage a writer already in Failed; finalization
preserves the moov only when the thread stopped with the writer alive
(disk guard, poison). The cap-rendering CI step runs --nocapture so a
WARP notch skip is visible instead of identical to a pass.
…esentative rates

The windows-2022 runner cannot consume >240fps synthetic over-delivery;
counting send blocking as pipeline fault there turned its saturation
into a hard fail on wall-clock re-pin drift (random/4 delivered864).
Overload cases go back to skipping loudly when the consumer saturates;
normal-rate cases keep the strict rule so a real consumer regression
cannot self-mask.
@richiemcilroy
richiemcilroy merged commit aba3875 into main Aug 7, 2026
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants