Repository navigation
FUG-112: supplemental captures + live registered 3D overlay - #83
Open
issuefleet[bot] wants to merge 5 commits into
Open
issuefleet[bot] wants to merge 5 commits into
issuefleet[bot] wants to merge 5 commits into
Conversation
Contributor
|
The keystone for registered overlays and the pose-reacquisition signal: recover the live camera pose from decoded-LED <-> centroid correspondences (gravity-up multi-start + damped Gauss-Newton on Huber reprojection, warm- started from the previous frame's lock). Unit-tested for from-scratch and warm-start recovery, noise tolerance, outlier rejection, and degeneracy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- projectionOverlay.ts: the registered 3D layer for the camera viewport — per-frame PnP (refine-only warm path, throttled from-scratch search) drives both the drawn solved-LED dots (confidence-shaded, low-confidence rings) and the pose-reacquisition chip (classifyRegistration). - fuseMaps.ts: fuse a supplemental scan into an existing map by registering the new solve into the prior frame via a robust similarity fit on common LEDs (one-pass outlier refit), then merging per-LED — new LEDs extend coverage, re-seen LEDs corroborate (noisy-or confidence) or defer on conflict. Refuses when too few LEDs overlap or the fixture moved. - pnp.ts: add refineOnly fast path for per-frame tracking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
capture.ts: - persist raw observations with every captured map (mapStore), so it can be supplemented later - registered 3D projection layer drawn each frame via ProjectionOverlay, plus Tracks/Solved layer toggles that isolate each viewport layer and a pose- reacquisition chip driven by the per-frame PnP lock - supplement mode (?supplement=<id>): seed the overlay from the prior map, re-solve the new session, fuse via fuseMaps, and present an in-screen review with Accept & merge / Reject (and a clear 'couldn't register' path) mapStore.ts: persist/observe CaptureObservations; setMap can replace them; getObservations; supplementable summary flag. mapDetail.ts: a Supplement action tile for maps that kept their observations. app.css: styles for the projection layer, toggles, reacquisition chip, review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pose chip now distinguishes a supplemental scan (a persistent no-lock means the fixture's gravity-relative transform likely changed) from a first-time capture (still converging -> soft 'Aligning', not an alarming warning). A failed supplemental solve routes back to the source map rather than the browser. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fughilli
force-pushed
the
agent/fug-112-feature-supplemental-captures
branch
from
September 28, 2026 19:13
7f0f91c to
edee9ac
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FUG-112: Supplemental captures + a live, registered 3D overlay
Lets the user re-scan an existing map to fill gaps / raise confidence, and adds a live 3D layer that projects the solved LEDs (with confidence) back into the camera viewport during a scan — with independent toggles for the two viewport layers and a clear signal for whether the phone has reacquired absolute pose.
What's here
Client-side PnP (
web/src/geom/pnp.ts) — the keystone. The WebXR-free capture path carries no per-frame camera pose, so we recover one each frame from the correspondences we already have (each decoded track → a solved 3D LED position paired with its current 2D centroid). Gravity-up multi-start + damped Gauss-Newton on Huber-robust reprojection, warm-started from the previous lock (a refine-only fast path keeps steady-state cheap). This is exactly the "client-side PnP against the solved map … the phase-4.5 follow-up that brings registered overlays back" the existinglabels.ts/markers.tscomments pointed to.Registered 3D overlay (
web/src/ui/projectionOverlay.ts) — a second canvas over the video (sibling to the 2D-trackLabelOverlay). Runs PnP per frame, projects all solved LEDs into the viewport (confidence-shaded dots, hollow rings on low-confidence LEDs so gaps pop,id·scorelabels), and derives the pose-reacquisition chip from the same solve.Layer toggles + reacquisition chip — the scan HUD gets
TracksandSolvedtoggles that isolate each viewport layer independently, and a chip that reads Registered / Aligning / Not registered. In supplement mode a persistent no-lock is called out as "fixture may have moved" (its gravity-relative transform changed since the map was built); in a first-time capture the same state is a soft "Aligning…" rather than an alarm.Supplemental fusion (
web/src/geom/fuseMaps.ts) — a captured map now persists its raw observations (mapStore), so a Supplement tile on Map Detail opens#/capture?supplement=<id>. The new session solves as its own reconstruction, then registers into the prior map's frame via a robust similarity fit over the LEDs both scans resolved (one-pass outlier refit), and merges per-LED: new LEDs extend coverage, re-seen LEDs corroborate (noisy-or confidence) or defer to the stronger observation on conflict. It refuses (and says why) when too few LEDs overlap or the fixture moved.Accept / Reject review — after the fused solve, the result is shown in the live inset with Accept & merge / Reject (a non-registerable scan only offers Discard). Accept replaces the library map in place and extends its observations so it can be supplemented again; Reject leaves the prior map untouched.
Why register-then-merge instead of re-solving the union
The VIO segment selector keeps only the longest continuous camera arc, so concatenating two disjoint sessions and re-solving would silently drop one of them — they can't be jointly bundle-adjusted. Registering the new solve into the prior frame is both the correct fusion and the honest "reacquired absolute pose" signal: if the new arc can't be registered, the residual stays high and we surface it. No solver/firmware changes.
Tests
New pure-logic suites —
pnp(6),fuseMaps(5),projectionOverlay(5): from-scratch & warm-start pose recovery, noise tolerance, outlier rejection, degeneracy; registration + per-LED merge, corroboration, conflict handling, too-few-overlap and fixture-moved refusals; correspondence assembly and chip classification. Full//web:unit_tests(63) passes, the app type-checks, and//web:distbuilds.Not verified on hardware: this environment can't drive a phone camera, so there's no end-to-end capture run — verification is unit tests + typecheck + bundle build.
🤖 Generated with Claude Code
Closes-Linear: FUG-112 (https://linear.app/fughilli/issue/FUG-112/feature-supplemental-captures)