Skip to content

feat(chat): preview agent media on web and mobile - #5047

Closed
gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2
Closed

gabrielelpidio wants to merge 225 commits into
t3code/codex-turn-mappingfrom
t3code/media-previews-v2

Conversation

@gabrielelpidio

@gabrielelpidio gabrielelpidio commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Agent-produced images and saved browser evidence are currently shown as paths instead of useful previews. The previous implementation in #4872 depended on the V1 orchestration layer removed by #2829.

What changed

  • add a durable orchestration V2 image_view turn item and project Codex imageView / saved imageGeneration events into it
  • render V2 image outputs and markdown image/video paths in web and mobile chat
  • resolve workspace media, browser artifacts, and V2 thread images through short-lived signed asset URLs
  • persist preview_snapshot evidence either to a unique server artifact (save: true) or a validated workspace-relative PNG (savePath)
  • harden external-file serving and screenshot writes against traversal and symlink swaps

Codex has a native image event and now emits the first-class V2 item. Other providers still get provider-independent markdown media rendering until their adapters expose an equivalent native event.

Important

This PR is intentionally stacked on #2829. TODO: retarget/rebase it to main after #2829 merges.

Verification

  • 8 focused test files, 111 tests passed
  • contracts, shared, web, and mobile typechecks passed
  • real V2 Codex turn persisted a completed image_view item and rendered the same projection in web and iOS
  • web and iOS both rendered the first-class V2 image output and a relative markdown workspace image; image expansion was also exercised

The server typecheck currently reaches the existing #2829 error in untouched apps/server/src/mcp/toolkits/worktree/registration.test.ts (ServerConfig | WorkspacePaths missing from the expected test context). #2829's own Check and Test jobs are already failing at this base head.

Screenshots

Web — inline previews

web-media-previews-v2

Web — expanded preview

web-media-preview-expanded-v2

Mobile — iOS

mobile-media-previews-v2

Reimplements the media-preview work from #4872 against orchestration V2.

Model: GPT-5.6-Sol
Harness: T3 Code (Codex)


Note

High Risk
Touches signed asset serving, symlink-safe file I/O, and new asset claim types—security-sensitive paths that must stay fail-closed; broad surface across server, contracts, web, and mobile.

Overview
Adds inline image and video previews in web and mobile chat instead of raw paths, reimplemented on orchestration V2 after the V1 layer removal.

Orchestration & feed: Introduces a durable V2 image_view turn item (Codex imageView / saved imageGeneration). Completed items surface as dedicated image-output timeline/feed rows; failed ones stay in the work log.

Markdown media: New MarkdownMedia (web + mobile) resolves markdown img/video via resolveMarkdownMediaSource—direct URLs, thread workspace files, browser-artifact, or thread-image—and loads them through signed asset URLs with loading/error UI and image expand.

Assets & server: Extends AssetResource with browser-artifact and thread-image; video workspace previews use exact-file claims. resolveAsset can return streamed open-file responses. noFollowFile hardens reads/writes and screenshot persistence against traversal/symlinks.

Preview MCP: preview_snapshot can save: true (server artifact + savedScreenshotPath) or savePath (validated workspace PNG); tool hints updated accordingly.

Mobile markdown: renderImage / context lets native selectable markdown delegate image nodes to the same media pipeline.

Reviewed by Cursor Bugbot for commit 0387ef2. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add inline media preview for agent images and videos in web and mobile chat

  • Adds image_view as a new turn item type in the orchestration V2 contract, projected from Codex image/imageGeneration items and emitted as image-output timeline/feed entries in both web and mobile.
  • Web chat renders image-output rows as clickable thumbnail previews via ImageOutputTimelineRow, and markdown img/video tags are now rendered through a new MarkdownMedia component that resolves thread-scoped asset URLs.
  • Mobile chat adds MarkdownMedia and ThreadImageOutput components with loading, unavailable, and tap-to-expand states; custom image rendering is injected into markdown via a new renderImage prop and React context.
  • The preview_snapshot MCP tool now accepts optional save (boolean) or savePath parameters to persist screenshots either to the server's browser-artifacts store or a workspace-relative path, with symlink traversal protection.
  • New AssetResource variants (browser-artifact, thread-image) are added to contracts, with corresponding resolveAsset and issueAssetUrl logic backed by no-follow file opening on Linux, macOS, and other platforms.
  • Risk: resolveAsset now returns an open-file variant with a ReadStream in addition to the existing file variant; callers that exhaustively switch on ResolvedAsset must handle the new case.

Macroscope summarized 0387ef2.

juliusmarminge and others added 30 commits April 17, 2026 17:29
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state
- Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope`
- Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing
- Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state
- Ensure runtime scopes close on stop and startup failure
- Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down
- Make session runtime close idempotent with an atomic closed flag
- Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills
- Remove legacy CLI/config discovery paths and related helpers
- Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts

Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry.

Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests

Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 4 times, most recently from 22bd872 to a27c1cc Compare August 10, 2026 17:05
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 6 times, most recently from 519c42a to 4c55679 Compare August 17, 2026 10:28
@andybergon

Copy link
Copy Markdown

Fresh reproduction after #6433 merged:

  • Server: T3 Code Nightly 0.0.34-nightly.20260824.1172; Android client version not captured.
  • A Codex turn returned three valid PNGs through the image-view tool.
  • Desktop rendered all three in the work stream.
  • Mobile did not expose them as tappable or downloadable images, and the completed turn’s final message could not preserve them as normal attachments.
  • Current source routes normal attachments and workspace Markdown images to the full-screen viewer, but image-view results remain trapped in tool output.

This confirms that the first-class agent image-output path covered by this PR is still needed after #6433.

@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 13 times, most recently from ceea97b to d2f1f51 Compare September 2, 2026 18:07
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch 4 times, most recently from b82facd to 2ac9bfe Compare September 5, 2026 04:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants