Feature: Generative UI with interactive HTML, CSS, and JavaScript #479

Problem and goal

Introduce Generative UI: CraftBot creates an interactive experience as part of its answer, using HTML, CSS, and JavaScript. Users can change inputs, explore data, follow steps, and interact with a design immediately.

The generated experience is frontend only: no generated backend, database, dependency installation, or separate server process.

This issue captures the feature direction and the research needed before implementation. The architecture below is a proposal to validate.

Initial use cases

Controls must produce meaningful changes. A weather refresh fetches real data; a serving selector recalculates ingredient quantities; a design control changes the preview. Ordinary interactions should run locally without another model call.

Proposed approach

• Add an explicit agent action, such as render_ui, producing a structured artifact with an ID, title, HTML document, revision, and declared capabilities.
• Start with a self-contained HTML document with embedded CSS and JavaScript.
• Render it in a sandboxed iframe with scripts enabled, isolated from CraftBot's DOM, cookies, credentials, and privileged APIs.
• Enforce a Content Security Policy and network policy. Generated code must not inherit unrestricted network access or broad browser permissions.
• Provide a minimal, validated host bridge for resizing, state persistence, runtime error reporting, and explicit follow-up actions.
• Support conversational revisions while retaining the previous working version. Execute complete revisions initially rather than partially generated JavaScript.
• Provide an inline preview with an expanded view, reset, and retry.

API access

Public APIs can be called directly when they support browser access through CORS and the sandbox network policy permits the request.

APIs requiring secret keys must use an approved capability in CraftBot's existing trusted host/service. Never embed credentials in generated JavaScript. Prefer specific approved providers/operations over an unrestricted URL proxy.

Open-Meteo is a candidate for the weather prototype: it supports CORS and keyless access for non-commercial use. Confirm commercial terms before selecting the production provider.

State and lifecycle

Keep user state separate from generated code and scope it to the artifact. Navigation, chat virtualization, and reopening a preview should restore selections and cooking progress. Persist timer deadlines so remounting does not restart them; background notification guarantees are outside the initial scope.

Generative UI provides interactive answers within conversations. Existing Agent Apps remain the path for ongoing applications with their own backend and data model.

CraftBot integration points

• app/ui_layer/browser/frontend/src/pages/Chat/ChatMessage.tsx and components/ui/MarkdownContent.tsx: introduce an explicit artifact representation instead of executing ordinary Markdown/code blocks.
• app/ui_layer/browser/frontend/src/types/index.ts: extend the chat model with artifact references.
• app/ui_layer/browser/frontend/src/pages/AgentApp/iframePool.ts: reuse relevant lifecycle patterns, but create a restricted runtime rather than copying the current Agent App browser capability delegation.

Research and design decisions

• Compare generated HTML/CSS/JS with prebuilt components and declarative approaches such as A2UI/OpenUI. Evaluate flexibility and reliability using the three initial use cases.
• Define the artifact/action schema, revision model, host bridge, placement, save/reopen behavior, and state migration rules.
• Validate sandbox isolation, CSP enforcement, message source/payload validation, API access from the sandbox, and handling of expensive or unresponsive generated code.
• Define generation instructions and verification for responsive layout, keyboard operation, labels, meaningful interactions, and runtime failures.

Acceptance criteria for the first version

• CraftBot generates and displays interactive HTML/CSS/JavaScript without creating a separate application or backend.
• Cooking demo supports serving adjustments, ingredient checklists, step navigation, and timers.
• Weather demo uses real API data and supports city search, unit switching, refresh, loading/error states, and a visible data timestamp.
• Design demo supports working style controls, responsive previews, and interactive component states.
• Ordinary interactions do not trigger a model call.
• Artifact state survives preview reopening and chat navigation, including virtualized message remounts.
• Users can request revisions, expand the preview, reset state, and retry failed generation.
• Runtime errors are surfaced with a recovery path.
• Generated code cannot access CraftBot's parent DOM, credentials, arbitrary host operations, or undeclared network destinations.
• Demos are verified in the actual preview, including keyboard interaction, narrow layouts, API failures, and state restoration.

References

• Vercel AI SDK: tool results rendered as UI components
• A2UI: declarative agent-driven interfaces
• OpenUI: controlled component libraries and streaming rendering
• Claude custom visuals: interactive HTML within conversations
• Anthropic Playground: self-contained interactive HTML experiences
• MDN: iframe srcdoc security considerations
• MDN: cross-window messaging and validation
• Open-Meteo: weather API and usage terms