feat(chat): preview agent media on web and mobile - #5047
Closed
gabrielelpidio wants to merge 225 commits into
Closed
gabrielelpidio wants to merge 225 commits into
gabrielelpidio wants to merge 225 commits into
Conversation
Co-authored-by: codex <codex@users.noreply.github.com>
- Initialize provider as unchecked in a pending state - Update initial probe message to reflect session-local status
- Type the runtime effect with `Scope` - Build the ACP session runtime without wrapping it in `Effect.scoped`
- Use strict TurnId and ProviderItemId parsing in Codex session routing - Decode in-memory stdio chunks in streaming mode to avoid split UTF-8 corruption
- Transfer session-owned scopes into adapter state - Ensure runtime scopes close on stop and startup failure - Add regression coverage for scoped lifecycle cleanup
- Close the managed native event logger when the adapter layer tears down - Make session runtime close idempotent with an atomic closed flag - Add coverage for flushing thread native logs on shutdown
- Use codex app-server snapshots for auth, models, and skills - Remove legacy CLI/config discovery paths and related helpers - Update tests for the new provider status flow
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model - Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types - Add replay runtime, fixtures, and integration coverage - Update shared contracts and probe transcripts Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring - Introduce in-memory orchestration projections and provider registry - Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections - Support cheap fork creation and Codex native fork rollback - Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work - Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions - Add shell snapshot projection support plus coverage tests - Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server` - Add Claude Agent SDK replay fixtures and test harness - Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module - Share the SDK query runner between live and replay paths - Add replay driver error wrapping for unexpected failures
Port orchestration V2 provider adapter wiring to the provider-instance driver registry. Co-authored-by: codex <codex@users.noreply.github.com>
- persist the selected model on run records - surface run model selection in the debug UI - update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes - Add approval and tool-call replay coverage for new orchestration fixtures - Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy - Add read-only replay fixture and policy mapping tests - Reuse shared approval-policy fixtures across orchestrator tests Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures - update Claude adapter/orchestrator turn handling for steering - refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex - log Claude Agent SDK protocol frames to native event traces - project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles - Update MCP capability, tool, and testing guidance for SDK-based injection
juliusmarminge
force-pushed
the
t3code/codex-turn-mapping
branch
4 times, most recently
from
August 10, 2026 17:05
22bd872 to
a27c1cc
Compare
juliusmarminge
force-pushed
the
t3code/codex-turn-mapping
branch
6 times, most recently
from
August 17, 2026 10:28
519c42a to
4c55679
Compare
|
Fresh reproduction after #6433 merged:
This confirms that the first-class agent image-output path covered by this PR is still needed after #6433. |
juliusmarminge
force-pushed
the
t3code/codex-turn-mapping
branch
13 times, most recently
from
September 2, 2026 18:07
ceea97b to
d2f1f51
Compare
juliusmarminge
force-pushed
the
t3code/codex-turn-mapping
branch
4 times, most recently
from
September 5, 2026 04:56
b82facd to
2ac9bfe
Compare
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Agent-produced images and saved browser evidence are currently shown as paths instead of useful previews. The previous implementation in #4872 depended on the V1 orchestration layer removed by #2829.
What changed
image_viewturn item and project CodeximageView/ savedimageGenerationevents into itpreview_snapshotevidence either to a unique server artifact (save: true) or a validated workspace-relative PNG (savePath)Codex has a native image event and now emits the first-class V2 item. Other providers still get provider-independent markdown media rendering until their adapters expose an equivalent native event.
Important
This PR is intentionally stacked on #2829. TODO: retarget/rebase it to
mainafter #2829 merges.Verification
image_viewitem and rendered the same projection in web and iOSThe server typecheck currently reaches the existing #2829 error in untouched
apps/server/src/mcp/toolkits/worktree/registration.test.ts(ServerConfig | WorkspacePathsmissing from the expected test context). #2829's ownCheckandTestjobs are already failing at this base head.Screenshots
Web — inline previews
Web — expanded preview
Mobile — iOS
Reimplements the media-preview work from #4872 against orchestration V2.
Model: GPT-5.6-Sol
Harness: T3 Code (Codex)
Note
High Risk
Touches signed asset serving, symlink-safe file I/O, and new asset claim types—security-sensitive paths that must stay fail-closed; broad surface across server, contracts, web, and mobile.
Overview
Adds inline image and video previews in web and mobile chat instead of raw paths, reimplemented on orchestration V2 after the V1 layer removal.
Orchestration & feed: Introduces a durable V2
image_viewturn item (CodeximageView/ savedimageGeneration). Completed items surface as dedicatedimage-outputtimeline/feed rows; failed ones stay in the work log.Markdown media: New
MarkdownMedia(web + mobile) resolves markdownimg/videoviaresolveMarkdownMediaSource—direct URLs, thread workspace files,browser-artifact, orthread-image—and loads them through signed asset URLs with loading/error UI and image expand.Assets & server: Extends
AssetResourcewithbrowser-artifactandthread-image; video workspace previews use exact-file claims.resolveAssetcan return streamedopen-fileresponses.noFollowFilehardens reads/writes and screenshot persistence against traversal/symlinks.Preview MCP:
preview_snapshotcansave: true(server artifact +savedScreenshotPath) orsavePath(validated workspace PNG); tool hints updated accordingly.Mobile markdown:
renderImage/ context lets native selectable markdown delegate image nodes to the same media pipeline.Reviewed by Cursor Bugbot for commit 0387ef2. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add inline media preview for agent images and videos in web and mobile chat
image_viewas a new turn item type in the orchestration V2 contract, projected from Codex image/imageGeneration items and emitted asimage-outputtimeline/feed entries in both web and mobile.image-outputrows as clickable thumbnail previews viaImageOutputTimelineRow, and markdownimg/videotags are now rendered through a newMarkdownMediacomponent that resolves thread-scoped asset URLs.MarkdownMediaandThreadImageOutputcomponents with loading, unavailable, and tap-to-expand states; custom image rendering is injected into markdown via a newrenderImageprop and React context.preview_snapshotMCP tool now accepts optionalsave(boolean) orsavePathparameters to persist screenshots either to the server'sbrowser-artifactsstore or a workspace-relative path, with symlink traversal protection.AssetResourcevariants (browser-artifact,thread-image) are added to contracts, with correspondingresolveAssetandissueAssetUrllogic backed by no-follow file opening on Linux, macOS, and other platforms.resolveAssetnow returns anopen-filevariant with aReadStreamin addition to the existingfilevariant; callers that exhaustively switch onResolvedAssetmust handle the new case.Macroscope summarized 0387ef2.