Skip to content

Add fullscreen mouse navigation and smoother rendering - #3

Open
mrzv wants to merge 6 commits into
jrf:mainfrom
mrzv:mouse-navigation
Open

mrzv wants to merge 6 commits into
jrf:mainfrom
mrzv:mouse-navigation

Conversation

@mrzv

@mrzv mrzv commented Oct 5, 2026 •

Copy link
Copy Markdown

Dependency and review scope

Depends on #2. The branch stacks the four mouse/navigation commits after the two SVG commits.

Because svg-support exists in the fork, not as a base branch in this upstream repository, this PR remains based on main and includes the SVG prerequisites until they are integrated. Review the mouse-only delta here: mrzv/rview@svg-support...mouse-navigation

Mouse/navigation commits

  1. [ai] Add fullscreen mouse pan and zoom
  2. [ai] Smooth fullscreen zoom and mouse panning
  3. [ai] Keep navigation visible and reduce SVG pan latency
  4. [ai] Prioritize newest mouse position during navigation

Changes

  • Enable wheel zoom over the fullscreen image viewport and left-button drag panning, with edge clamping, dialog isolation and mouse-capture cleanup.
  • Move zoom rendering off the input thread and coalesce desired navigation state. Skip obsolete pan results instead of displaying intermediate positions in order; reject obsolete image, scale and viewport results.
  • Re-crop a completed SVG canvas at the newest desired pan when scale, fractional phase and dimensions match. Otherwise retain the displayed image and render only the newest view.
  • Keep the current fullscreen frame visible during rendering and transfer. Upload into an inactive image slot, place the completed replacement, then retire the previous image; bulk transfers occur outside synchronized updates.
  • Reuse a budgeted, phase-sensitive, straight-alpha SVG canvas for whole-pixel panning, avoiding repeated rasterization and alpha conversion. Preserve sharp vector detail, transparency and off-viewport filter inputs; retain viewport-only fallback for oversized unfiltered SVGs.
  • Start work immediately, poll pending results promptly and bound input draining. Publish ready frames without a fixed 60 Hz delay and without retransmitting unchanged pixels.
  • Stream base64 through a fixed-size buffer and use fast lossless PNG for fullscreen transfers. Gallery/video retain their existing local-raw versus SSH-PNG selection.
  • Preserve fit images when obsolete zoom-source errors arrive after reset, keep active errors visible, and prevent late completions from painting over overlays or navigation transitions.
  • Update fullscreen hints, help and README controls/rendering documentation.

Verification

  • Latest commit: cargo test 43 passed; cargo test --no-default-features 42 passed; formatting and release build passed.
  • Latest commit: controlling-PTY smoke verified continuous drag and direction reversals, intermediate progress, exact final frames against an independent direct-drag reference, sharp 1/16-unit SVG detail, fractional transparency, off-viewport filter inputs, PNG behavior, resize/reset, pending navigation and overlays, continuous visibility and clean exits.
  • Earlier continuity commit: cargo test --features video 44 passed using compatible FFmpeg 8 libraries in an isolated temporary environment; video release build and actual decoded frame advancement, pause/resume, image/video/gallery transitions and cleanup passed.
  • Earlier continuity commit: a deliberately throttled 1,077,208-byte / 351-chunk fullscreen upload took 14.8 seconds. The previous placement remained visible throughout, the new placement preceded old-image retirement, and both reserved image slots were cleaned on exit.

Latest responsiveness measurements

Same dense filtered SVG fixture and fixed 240 Hz drag stream, release binaries before/after the latest commit:

Measurement Before After
Catch-up after mouse movement stops 26.3 ms 9.6 ms
Estimated median pointer-to-image gap 49.6 px 33.3 px

The gap estimate has approximately 7.7 px pixel-quantization uncertainty. Both runs had zero blank upload states; the final decoded frame matched an independent fresh viewer at the same final displacement, and reset restored the original pixels.

Earlier continuity/performance measurements

Same fixtures and input streams, before/after the preceding continuity commit:

Fixture Median pan before Median pan after
Dense filtered SVG 50.7 ms 24.1 ms
Non-square filtered SVG 50.0 ms 22.3 ms
High-entropy raster PNG 50.6 ms 32.5 ms

All three after-runs observed zero blank upload states. Measurements are fixture-specific controlling-PTY/protocol observations, not a live Kitty GUI benchmark; pixel fidelity was checked from decoded graphics payloads.

mrzv added 4 commits October 5, 2026 14:29
Recognize SVG files in directory scans and rasterize them through the shared image decoder with resvg. Preserve transparency by converting premultiplied pixels to straight RGBA, resolve local image references relative to the SVG, and share a lazily loaded system font database. Fit renders use bounded dimensions without upscaling; zoom retains the existing intrinsic-raster path.

Bypass the persistent SVG thumbnail cache because its source-only key cannot track referenced images or system fonts. Apply the standard image decoder allocation budget before creating SVG raster buffers so excessive intrinsic dimensions return an error instead of aborting allocation, while bounded thumbnails and fit renders remain possible.

Add rendering, resource, cache freshness, allocation-limit and file-discovery coverage, update dependencies, and document SVG usage and raster zoom behavior.

Model: openai-codex/gpt-6.1-sol:high
Keep parsed SVG trees as fullscreen zoom sources instead of magnifying
intrinsic-resolution bitmaps. Re-render each SVG view at its output pixel
resolution with continuous crop geometry, retaining fractional detail and
matching mouse pan displacement. Leave raster crop/resize behavior intact.

Render filtered SVGs on a full zoomed canvas before cropping so effects
retain inputs outside the viewport. Preserve fractional alignment, budget
canvas and crop allocations together, and report rendering failures in the
viewer with recovery when the zoom or viewport changes. Unfiltered SVGs
continue to allocate only the visible viewport.

Add regressions for fine vector detail, fractional pan and transparency,
offscreen filter inputs, allocation limits and recovery. Update SVG viewing
documentation to describe vector zoom and filtered rendering limits.

Model: openai-codex/gpt-6.1-sol:high
Enable terminal mouse capture for the viewer and disable it on restoration
and terminal initialization failures. Route wheel events over the fullscreen
image viewport through the existing zoom controls, and track left-button
drags using cell-to-pixel displacements and the shared source-crop geometry.

Clamp drag panning at image edges and cancel gestures on release, keyboard
input, resize, focus loss and modal transitions. Keep mouse navigation
inactive in gallery, picker, video and dialog modes. Add pixel-displacement
and fit-scale regression coverage, and document the controls in README,
fullscreen hints and compact help entries.

Model: openai-codex/gpt-6.1-sol:high
Move zoom rendering off the terminal input thread using one background job
at a time and shared immutable image sources. Coalesce requests to the latest
desired geometry, publish ordered pan progress during continuous motion,
and reject completions for obsolete images, zoom scales or viewports.

Retain a budgeted premultiplied SVG canvas for whole-pixel pan crops. Re-render
when scale or subpixel phase changes, preserve filter inputs, and keep the
viewport-only fallback for unfiltered canvases exceeding the buffer limit.
Preserve fit images when obsolete zoom-source failures arrive after reset.

Bound input draining, poll pending work promptly, and pace fullscreen frame
publication without retransmitting unchanged images. Buffer Kitty output,
keep pending work from painting over overlays, and restore explicitly cleared
frames even when navigation stays at the same image.

Add state-transition and SVG pixel regressions and document the background
rendering and canvas reuse behavior.

Model: openai-codex/gpt-6.1-sol:high
@mrzv
mrzv force-pushed the mouse-navigation branch from 5790673 to a38d249 Compare October 5, 2026 23:56
@mrzv mrzv changed the title Add fullscreen mouse pan and zoom Add fullscreen mouse navigation and smoother rendering Oct 5, 2026
mrzv added 2 commits October 5, 2026 17:20
Upload fullscreen frames into an inactive Kitty image slot while the prior placement remains visible. Flush the bulk transfer before a short synchronized place-before-retire swap, alternate two reserved IDs, and clean both stored slots on real mode exits and terminal restoration. Preserve re-placement without retransmission and re-upload after external storage clears.

Use fast lossless PNG for fullscreen transfers and stream base64 through a fixed chunk buffer instead of allocating a whole-image encoded string and chunk list. Keep the existing gallery and video transport selection unchanged.

Retain cached SVG canvases in straight RGBA so whole-pixel pans copy the visible crop without repeating alpha conversion. Compute pixel phase in f64 before narrowing render transforms, avoiding accidental cache misses for whole-pixel movement while retaining genuine subpixel rendering, filter inputs and shared allocation limits. Start rendering independently of publication pacing and poll pending workers at 4ms to reduce input-to-frame delay.

Add publication ordering, storage invalidation, cleanup and transport round-trip regressions, and document continuous fullscreen swaps and cached SVG navigation.

Model: openai-codex/gpt-6.1-sol:high
Stop publishing obsolete pan results merely to preserve an ordered stream of intermediate frames. Keep the displayed image until a current result is ready, or recrop a completed SVG canvas at the newest desired pan when its source, scale, fractional phase and dimensions still match. Drop obsolete output before allocating the replacement crop and retain stale-source, viewport and scale rejection.

Share SVG viewport geometry between rendering and cache recropping so exact vector detail, fractional alpha, filter inputs and allocation limits remain consistent. Remove the fixed fullscreen publication cadence so ready images can follow input immediately without an artificial frame deadline.

Update regression coverage for stale raster results, latest SVG recropping and fractional-phase fallback, and document the responsiveness-first navigation policy.

Model: openai-codex/gpt-6.1-sol:high
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant