Let a Bot visualize with Open Generative UI, not describe markup

.env.example

@@ -19,6 +19,23 @@ KEY_ENCRYPTION_KEY=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
 PORT=3001
 SERVER_PORT=3001
 TENANT_PACKAGE_DIR=../examples/fintech
+# Let a Bot answer with an interface it wrote itself: markup, styles and a script it generates for
+# that one answer, streamed into the transcript and rendered in a sandboxed iframe with no
+# same-origin access to this app. Off unless you set this to `true` or `1`.
+#
+# Asked for rather than inherited, unlike most of the switches in this file. It decides whether a
+# model may put code it wrote on somebody's screen, so a deployment should choose it rather than
+# acquire it by upgrading — including a deployment that builds its default branch automatically.
+#
+# This is not the component catalogue. A component is something this deployment holds and an
+# administrator grants per Bot; this has nothing to grant, because the Bot writes it on the spot and
+# it is gone when the conversation moves on. It is a deployment switch rather than a per-Bot grant
+# because the SDK offers no seam for one — see DeploymentConfig.generativeUi in server/src/config.ts.
+#
+# What the interface can reach is what the sandbox hands it, and this deployment hands it nothing: no
+# session, no same-origin access, and no route into your data. It can load libraries from a CDN, so a
+# deployment that must not reach the public internet from a browser tab should leave this off.
+# OPENBOT_GENERATIVE_UI=true
 # What this deployment calls itself, when more than one shares an Intelligence project. A copy of a
 # deployment made for development uses the same project key, and threads are listed per Bot with
 # nothing to say which deployment a conversation came from. The name goes into every thread id this

app/src/components/channels/chat-messages.ts

@@ -1,4 +1,4 @@
-import type { Message, ToolCall } from "@ag-ui/core";
+import type { ActivityMessage, Message, ToolCall } from "@ag-ui/core";
 
 /**
  * Transcript projection that pairs assistant tool calls with later tool-result messages.
@@ -12,7 +12,25 @@ export type VisibleChatItem =
       toolCall: ToolCall;
       /** The result, once there is one. Absent means the call is still in flight. */
       result?: string;
-    };
+    }
+  /**
+   * Something a Bot is drawing rather than saying.
+   *
+   * Carried whole rather than projected into fields of our own, because what is inside an activity
+   * belongs to whoever renders it: an interface a Bot generated arrives here as partial HTML that
+   * grows on every chunk, and the renderer that paints it is the one that knows what a half-finished
+   * one looks like. Reshaping it on the way past would mean this file had to understand every
+   * activity type anybody registers.
+   */
+  | { kind: "activity"; id: string; message: ActivityMessage };
+
+/**
+ * The SDK's own tool for drawing an interface, whose output is an activity rather than a result.
+ *
+ * Named here rather than imported because the SDK exports the renderer and the argument schema but
+ * not the tool name; it is the string the runtime middleware matches on to emit the activity.
+ */
+const GENERATE_SANDBOXED_UI = "generateSandboxedUi";
 
 /** A tool result, as it arrives, its own message, pointing back at the call it answers. */
 type ToolResultMessage = { role: "tool"; toolCallId: string; content?: string };
@@ -44,6 +62,15 @@ export function toVisibleChatItems(
         });
       }
       for (const toolCall of message.toolCalls ?? []) {
+        /*
+         * The call that draws an interface is not a row of its own; the interface is.
+         *
+         * Its renderer shows the waiting message and then returns nothing, so once the interface has
+         * arrived this leaves an empty item behind — invisible in itself, but still a child of a
+         * `gap-6` column, so every generated interface gained a stray gap under it. The activity
+         * beside it already shows its own progress while it is being written.
+         */
+        if (toolCall.function.name === GENERATE_SANDBOXED_UI) continue;
         items.push({
           kind: "tool",
           // One assistant message can carry multiple tool calls.
@@ -57,6 +84,18 @@ export function toVisibleChatItems(
       return items;
     }
 
+    /*
+     * Activities are their own messages, in order, beside the prose.
+     *
+     * Kept rather than dropped, which is what this projection used to do with every role it did not
+     * name. A Bot that draws its own interface says nothing in `content` and calls no tool the
+     * transcript can pair a result with — the whole answer is the activity. Falling through to the
+     * bail below meant the turn rendered as silence.
+     */
+    if (message.role === "activity") {
+      return [{ kind: "activity", id: message.id, message }];
+    }
+
     if (message.role !== "user") return [];
 
     const text =

app/src/components/channels/chat-transcript.tsx

@@ -1,5 +1,8 @@
-import type { Message } from "@ag-ui/core";
-import { useRenderToolCall } from "@copilotkit/react-core/v2";
+import type { ActivityMessage, Message } from "@ag-ui/core";
+import {
+  useRenderActivityMessage,
+  useRenderToolCall,
+} from "@copilotkit/react-core/v2";
 import { IconBox } from "@tabler/icons-react";
 import { motion, useReducedMotion } from "motion/react";
 import { memo, useEffect, useMemo, useRef } from "react";
@@ -464,6 +467,52 @@ const TranscriptMessage = memo(function TranscriptMessage({
   );
 });
 
+/**
+ * What a failed activity is called on screen.
+ *
+ * An `activityType` is a protocol name and reads like one, so the boundary's sentence gets a phrase
+ * a person can read instead. An unknown type falls back to its own name rather than to something
+ * vague: whoever registered it will recognise it, and nobody else can act on either wording.
+ */
+function activityName(activityType: string): string {
+  return activityType === "open-generative-ui"
+    ? "This Bot's generated interface"
+    : activityType;
+}
+
+/**
+ * One activity, drawn by whichever renderer claims its type.
+ *
+ * The renderer comes from the SDK's registry, so an activity nobody registered draws nothing and
+ * this returns null rather than an empty row — unlike a tool call, which always has a line to fall
+ * back to because a call that happened is worth reporting even undrawn. An activity is the drawing;
+ * with no renderer there is nothing to say about it.
+ *
+ * The memo boundary earns its place differently here than on a tool call. It cannot spare this
+ * component its own churn — a generated interface streams, and `content` is a new object on every
+ * chunk, which is exactly when it must re-render — but it does stop the sentence being typed after it
+ * from re-rendering a mounted iframe.
+ */
+const TranscriptActivity = memo(function TranscriptActivity({
+  delay,
+  message,
+}: {
+  delay: number;
+  message: ActivityMessage;
+}) {
+  const { renderActivityMessage } = useRenderActivityMessage();
+  const drawn = renderActivityMessage(message);
+  if (!drawn) return null;
+
+  return (
+    <Arriving delay={delay}>
+      <ToolRenderBoundary name={activityName(message.activityType)}>
+        {drawn}
+      </ToolRenderBoundary>
+    </Arriving>
+  );
+});
+
 /**
  * One drawn tool call, memoised on the same terms.
  *
@@ -653,6 +702,13 @@ export function ChatTranscript({
                     toolCallId={item.toolCall.id}
                   />
                 </MessageScrollerItem>
+              ) : item.kind === "activity" ? (
+                <MessageScrollerItem key={item.id} messageId={item.id}>
+                  <TranscriptActivity
+                    delay={delays.delayFor(item.id, index, items.length)}
+                    message={item.message}
+                  />
+                </MessageScrollerItem>
               ) : (
                 <MessageScrollerItem
                   key={item.id}

app/src/lib/copilot/generative-ui.ts

@@ -0,0 +1,50 @@
+/**
+ * What a Bot is told about drawing an interface it wrote itself.
+ *
+ * The SDK ships a default set of guidelines, and they describe shadcn/ui: rounded cards, a violet
+ * accent, its own spacing scale. OpenBot does not look like that. The app's palette is deliberately
+ * without colour — every token in styles.css sits at chroma zero except the destructive red and the
+ * success teal — so the default guidance produces something that reads as a foreign widget dropped
+ * into the transcript rather than part of it.
+ *
+ * This is a prompt, so it is written for a model rather than for a person: concrete values it can
+ * copy, and the few rules that are actually load-bearing.
+ *
+ * WHY THE COLOURS ARE LITERAL. The generated interface renders inside a sandboxed iframe with no
+ * same-origin access to this app. It cannot reach the stylesheet, the theme class, or the CSS custom
+ * properties the rest of the UI is built from, so anything it should match has to be written out
+ * here in full. Referring it to `--muted-foreground` would produce an unstyled document.
+ *
+ * WHY prefers-color-scheme AND NOT THIS APP'S THEME. For the same reason: the iframe is a separate
+ * document and cannot see which theme the person picked. `prefers-color-scheme` is the only signal
+ * that crosses, so a generated interface follows the browser rather than the app's own switch. A
+ * person who has overridden their OS theme in OpenBot will see a generated interface that disagrees
+ * with the surface around it. That is a known limitation of the sandbox rather than something this
+ * text can fix.
+ */
+export const GENERATIVE_UI_DESIGN_SKILL = `You are generating a self-contained interface that renders inside a sandboxed iframe in OpenBot's chat transcript. It must look like it belongs to OpenBot, not like a widget from somewhere else.
+
+PALETTE. OpenBot is neutral by design. Use greys for structure and reserve colour for meaning.
+- Light: background #fafafa, surface #ffffff, text #0a0a0a, muted text #636363, border #e5e5e5.
+- Dark: background #0a0a0a, surface #171717, text #fafafa, muted text #a1a1a1, border rgba(255,255,255,0.10).
+- Only two accents, and only when they carry meaning: #e7000b destructive and #009689 success in light, #ff6467 and #00bba7 in dark.
+- Never introduce a brand hue, gradient, or coloured header. A purple or blue accent is wrong here.
+
+CHARTS. Series colours are steps of grey, not a rainbow: #d4d4d4, #737373, #525252, #404040, #262626. Distinguish series by ordering, direct labels, and shape rather than by hue. If a series means "bad", the destructive red is allowed for that one series.
+
+TYPE. font-family: Inter, ui-sans-serif, system-ui, sans-serif. Body 14px/1.5. Headings 15-16px, weight 600, no letter-spacing tricks. Numerals in tables and metrics: font-variant-numeric: tabular-nums.
+
+SHAPE AND SPACING. border-radius: 0.55rem on cards and controls, 0.375rem on small chips. 1px solid borders, never a drop shadow for elevation. Pad containers 12-16px. Space stacked blocks 8-12px.
+
+DARK MODE IS REQUIRED. Define the light palette first, then override inside @media (prefers-color-scheme: dark). Set an explicit background and colour on body — the iframe paints on nothing, so a transparent body shows through wrongly.
+
+LAYOUT. Assume a narrow column: roughly 320-680px wide, inside a chat message. Design for the narrow case first and let it grow. Never set a fixed pixel width on the outermost element; use max-width: 100%, flexbox or grid, and box-sizing: border-box everywhere. Anything wide — a table, a chart, a code block — scrolls inside its own container with overflow-x: auto. The page itself must never scroll sideways.
+
+HONESTY ABOUT DATA. You have no access to this deployment's data. Every number you render is one you were given or one you made up, so never present an invented figure as a reading from OpenBot. If you are illustrating rather than reporting, label it as an example on the interface itself.
+
+MECHANICS.
+- Keep it self-contained: inline the CSS and the JS. CDN <script> and <link> tags do load, so Chart.js, D3 and similar are available when a chart genuinely needs them; prefer plain SVG or CSS for anything simple.
+- No network calls to this deployment. The iframe has no session and no same-origin access; a fetch to /api will fail.
+- Guard every read of browser storage in try/catch. It throws outright in some contexts.
+- Interactive controls need a visible focus ring, real button elements, and hit targets of at least 32px.
+- Prefer one clear interface over a dashboard of panels. A single legible chart beats four cramped ones.`;

app/src/lib/copilot/provider.tsx

@@ -1,9 +1,12 @@
 import { CopilotKitProvider } from "@copilotkit/react-core/v2";
+import { useQuery } from "@tanstack/react-query";
 import type { ReactNode } from "react";
+import { deploymentCapabilitiesQueryOptions } from "@/lib/deployment/queries";
 import { ActiveBotProvider } from "./active-bot";
 import { ComputerTools } from "./computer-tools";
 import { EscalationTool } from "./escalation-tool";
 import { GalleryTools } from "./gallery-tools";
+import { GENERATIVE_UI_DESIGN_SKILL } from "./generative-ui";
 import { HandoffTool } from "./handoff-tool";
 import { SandboxedTools } from "./sandboxed-tools";
 
@@ -22,8 +25,30 @@ import { SandboxedTools } from "./sandboxed-tools";
  * runtime rather than returning it).
  */
 export function CopilotProvider({ children }: { children: ReactNode }) {
+  const { data: capabilities } = useQuery(deploymentCapabilitiesQueryOptions());
+
   return (
-    <CopilotKitProvider runtimeUrl="/api/copilotkit" credentials="include">
+    <CopilotKitProvider
+      runtimeUrl="/api/copilotkit"
+      credentials="include"
+      /*
+       * Passed only when this deployment actually has the capability, and this is the load-bearing
+       * part rather than a tidiness. The SDK reads generative UI as on when EITHER the runtime says
+       * so OR this prop is present at all, so passing it unconditionally would switch the browser
+       * half on in a deployment that had switched the server half off. The Bot would then be offered
+       * the tool, generate a whole interface, and nothing would draw it, because the events that
+       * paint one come from the runtime middleware this deployment declined to run.
+       *
+       * Absent, the SDK asks the runtime and believes the answer, which is the behaviour we want
+       * while this query is still in flight.
+       *
+       * The object carries guidance only. It does not turn anything on that the server has not
+       * already turned on; it replaces the SDK's shadcn-flavoured house style with OpenBot's.
+       */
+      {...(capabilities?.generativeUi
+        ? { openGenerativeUI: { designSkill: GENERATIVE_UI_DESIGN_SKILL } }
+        : {})}
+    >
       {/* Computer tools target the Bot declared by the mounted surface. */}
       <ActiveBotProvider>
         <ComputerTools />

app/src/lib/deployment/queries.ts

@@ -0,0 +1,59 @@
+import { queryOptions } from "@tanstack/react-query";
+import { client } from "@/lib/client";
+
+/**
+ * What this deployment can do, as the server is willing to say it.
+ *
+ * A projection, not the runtime. `/api/capabilities` is reachable before anybody has signed in, so
+ * only fields somebody may know unauthenticated appear on it; the Intelligence contract and every
+ * deployment secret stay on the server. See server/src/app.ts.
+ */
+export type DeploymentCapabilities = {
+  /**
+   * Whether a Bot may answer with an interface it wrote itself.
+   *
+   * Read by the browser because the browser owns half of this capability: the SDK's provider is what
+   * offers the model the tool that generates one. A deployment that turned the server half off while
+   * the browser kept offering the tool would have Bots writing whole interfaces that nothing draws,
+   * so both halves read this one answer.
+   */
+  generativeUi: boolean;
+};
+
+export const deploymentKeys = {
+  all: ["deployment"] as const,
+  capabilities: () => ["deployment", "capabilities"] as const,
+};
+
+/**
+ * What this deployment can do.
+ *
+ * From the server rather than from the build, like the sign-in options beside it: the container image
+ * is built once and knows nothing about the deployment that will run it, so a capability compiled
+ * into the bundle can only ever describe the build machine.
+ *
+ * Absent fields read as off. A server too old to answer, or one that failed to, is a server this app
+ * should not assume a capability of — and for generated interfaces the fail-closed direction is the
+ * safe one, because claiming it wrongly is what makes a Bot generate something nothing renders.
+ */
+export function deploymentCapabilitiesQueryOptions() {
+  return queryOptions({
+    queryKey: deploymentKeys.capabilities(),
+    // Configuration, not data. It cannot change without the process restarting.
+    staleTime: Number.POSITIVE_INFINITY,
+    queryFn: async (): Promise<DeploymentCapabilities> => {
+      /*
+       * The whole body, then one field off it. `/api/capabilities` answers with a bare object rather
+       * than the envelope most endpoints use, which is why this reads the Response itself instead of
+       * naming a key — the same shape the sign-in query uses.
+       */
+      const body = (await (
+        await client("/api/capabilities", {
+          fallback: "This deployment's capabilities could not be loaded.",
+        })
+      ).json()) as { generativeUi?: boolean };
+
+      return { generativeUi: body.generativeUi === true };
+    },
+  });
+}

app/src/styles.css

@@ -255,3 +255,37 @@ body {
 [data-slot="item-description"] {
   text-wrap: pretty;
 }
+
+/*
+ * A generated interface is never allowed to occupy nothing.
+ *
+ * The SDK sizes the frame it draws into from a single measurement taken inside the sandbox, and it
+ * writes the result as an inline height on the frame's own container. Two races in that code make
+ * the number wrong, and both were reproduced against @copilotkit/react-core 1.69.0:
+ *
+ *   - The measurement is queued alongside the expressions that build the interface. When it runs
+ *     first the body it measures is still empty, the container is set to `height: 0px`, and because
+ *     the container also clips, the interface is invisible. Nothing retries: the measurement is
+ *     one-shot and its listener is removed after the first reply.
+ *   - The effect that measures bails out when the sandbox has not finished loading, and its only
+ *     dependency is "generation finished". An interface restored from thread history is already
+ *     finished on its first render, while the sandbox is still being imported, so the effect bails
+ *     and never runs again.
+ *
+ * A floor is the smallest thing that removes the failure that matters. `min-height` clamps above an
+ * inline `height`, so a collapsed measurement can no longer render nothing, and a measurement that
+ * came back correct and larger is left alone. The reader gets an interface that is present and may
+ * be cut off, instead of a turn that looks like the Bot said nothing at all.
+ *
+ * Selected through the frame rather than through a class of ours because both chat surfaces render
+ * this container and only one of them is ours: the packaged chat builds it too, and a rule that
+ * needed our wrapper would fix the transcript and leave the other surface broken. `websandbox__frame`
+ * is the class the sandbox library puts on the frame it creates.
+ *
+ * DELETE THIS WHEN THE SDK IS FIXED. It compensates for someone else's arithmetic, so it should not
+ * outlive the bug: once the measurement re-runs on sandbox-ready and after the queue drains, this
+ * rule can only make small interfaces taller than they asked to be.
+ */
+:where(div:has(> iframe.websandbox__frame)) {
+  min-height: 200px;
+}