Automated system prompt #670

Activity

1. ngoiyaeric commented on Jun 24, 2026

   

   ngoiyaeric

   on Jun 24, 2026

   CollaboratorAuthor

   More actions

   @coderabbitai plan
2. coderabbitai commented on Jun 24, 2026

   

   coderabbitaiBot

   on Jun 24, 2026 – with coderabbitai

   Last edited by coderabbitai

   Contributor

   More actions

   Coding Plan

   Summary

   • Model the long-running scrape+synthesis as a true asynchronous job fully decoupled from the request lifecycle: a durable Postgres-backed prompt_generation_jobs row is the single source of truth for an explicit progress lifecycle (pending → processing → complete/error), fitting the existing Drizzle/standalone-Node stack without new queue infrastructure.
   • Split the backend into a fast non-blocking kickoff action (returns a jobId immediately, holding no connection), a detached background worker that runs independently of any HTTP request and writes only to the job row, and a lightweight status action that exposes progress purely from persisted state.
   • Treat retryability and resilience as first-class concerns: explicit terminal error states for Firecrawl/empty-content/AI failures, a worker-level timeout, staleness detection via an updatedAt heartbeat to recover jobs lost to process restarts, and re-kickoff to retry — never leaving a job stuck or the UI hanging.
   • Extend the Settings System Prompt tab with a domain input, kickoff trigger, and a polling loop whose UI is driven directly by the job's progress states (loading/complete/error/retry), populating the existing textarea on completion and reusing the current saveSystemPrompt persistence path.

   Design Choices

   Design Choice 1: What async architecture should drive the long-running scrape + AI synthesis?

   Options Considered:

   1. Synchronous server action — block the request until scrape + generation finish. Simplest, but couples the long operation to the request lifecycle, risks exceeding the page maxDuration (60s) / proxy timeouts, and gives no progress feedback.
   2. Durable DB jobs table + fire-and-forget execution + frontend polling — a kickoff action inserts a pending job and returns immediately; the scrape/generation runs independently of the HTTP request, transitioning the durable job row through processing and finally complete/error; the frontend polls a lightweight status action and reads progress purely from the persisted job state.
   3. SSE/streaming endpoint — stream progress over a ReadableStream. This keeps a connection open for the operation's full duration (re-coupling work to the request) and adds connection-lifetime concerns; no SSE infrastructure exists today.
   4. External queue (Inngest / BullMQ / pg-boss) — the most durable form of decoupling, but introduces a new dependency, worker process, and infra setup not present in the project.

   Chosen Option: 2

   Rationale: Option 2 — A durable Postgres-backed jobs table with frontend polling gives a genuinely decoupled async model (the request returns instantly; all progress/result state lives in the job row, not in the request) while fitting the existing Drizzle/Postgres + server-action stack and standalone-Node deployment, avoiding new infra/dependencies. Because all state is persisted, the model also supports stuck-job detection and retry without holding any connection open.

   Design Choice 2: After a job completes, should the generated prompt auto-save or populate the editor?

   Options Considered:

   1. Populate the systemPrompt textarea so the user can review/edit, then save via the existing Save button.
   2. Scrape and immediately persist the generated prompt to the DB.

   Chosen Option: 1

   Rationale: Populating the field lets the user review the AI-generated prompt before committing, matching the existing review-then-save UX and reusing the current save path.

   💡 User Tips

   Regenerate the plan with different choices with @coderabbitai <feedback>.

   Implementation Steps

   Phase 1: Job State Model & Firecrawl Foundation

   Establish the durable, request-independent state model that tracks an async scrape-and-generate job through its full progress lifecycle, plus the Firecrawl dependency/configuration. This persisted state is what makes the work decoupled from any request: all progress and results are read from the database, never from in-request memory. This phase is the foundation that both the background worker and the frontend polling rely on.

   Task 1: Define the prompt-generation jobs table and migration

   Create the durable record that represents a single domain-scrape + prompt-synthesis operation and is the single source of truth for its progress state.

   • Add a new table (e.g., prompt_generation_jobs) to lib/db/schema.ts following existing Drizzle conventions, with columns: id (uuid pk), userId (uuid), domain (text), status (text discriminator representing the explicit progress lifecycle — pending | processing | complete | error, defaulting to pending, mirroring the text-as-enum pattern used by chats.visibility/users.role), resultPrompt (text, nullable), errorMessage (text, nullable), createdAt, updatedAt.
   • Treat updatedAt as a heartbeat/progress timestamp: it advances on every status transition so stuck/abandoned jobs (e.g., after a process restart) can be detected later (Task 2.4).
   • Generate a corresponding Drizzle migration under drizzle/migrations (consistent with the existing 0000_*/0001_* files).
   • Add minimal DB helpers (create job, update status/result, fetch job by id+userId) — either in lib/actions/chat-db.ts (raw Drizzle ops) or a small new module, matching the existing data-access split.

   Task 2: Add the Firecrawl dependency and configuration

   Introduce Firecrawl following established secret-management conventions.

   • Add the firecrawl package to package.json.
   • Introduce a FIRECRAWL_API_KEY environment variable, documenting it in .env.local.example alongside the other *_API_KEY entries (Tavily, Serper, Composio).
   • Initialize the Firecrawl client server-side only, reading the key from process.env — mirroring how Tavily/Serper clients are constructed in lib/agents/tools/.

   🤖 Prompt for AI agents

   Establish the durable job state model and Firecrawl integration that the rest of
   the feature depends on. All progress and results will be read from the database,
   never from in-request memory.
   
   **Jobs Table & Migration** in `lib/db/schema.ts`:
   - Add a new `prompt_generation_jobs` table following existing Drizzle
   conventions with columns: `id` (uuid pk), `userId` (uuid), `domain` (text),
   `status` (text discriminator defaulting to `pending`, with values `pending` |
   `processing` | `complete` | `error`, mirroring the `text`-as-enum pattern used
   by `chats.visibility`/`users.role`), `resultPrompt` (text, nullable),
   `errorMessage` (text, nullable), `createdAt`, `updatedAt`.
   - Treat `updatedAt` as a heartbeat/progress timestamp — it must advance on every
   status transition so stuck/abandoned jobs (e.g., after a process restart) can be
   detected later.
   - Generate a corresponding Drizzle migration under `drizzle/migrations`,
   consistent with the existing `0000_*`/`0001_*` migration files.
   - Add minimal DB helper functions (create job, update status/result, fetch job
   by id+userId) either in `lib/actions/chat-db.ts` or a small new module, matching
   the existing data-access split.
   
   **Firecrawl Setup:**
   - Add the `firecrawl` npm package to `package.json`.
   - Add `FIRECRAWL_API_KEY` to `.env.local.example` alongside the other
   `*_API_KEY` entries (Tavily, Serper, Composio).
   - Initialize the Firecrawl client server-side only, reading from
   `process.env.FIRECRAWL_API_KEY`, following the same pattern as Tavily/Serper
   client construction in `lib/agents/tools/`.
   

   Phase 2: Backend — Decoupled Job Kickoff & Background Scrape/Generation

   Split the backend so that no part of the long-running work is tied to the originating request. A fast kickoff path creates the durable job and returns immediately; a background execution path — with no connection to the original HTTP request — drives the job through its progress states (processing → complete/error) by writing only to the persisted job row; and a lightweight status action lets the frontend observe that progress.

   Task 1: Create the job kickoff server action

   Provide a fast, non-blocking entry point that registers the durable job and hands off execution without waiting for it.

   • Add a server action (e.g., startSystemPromptGeneration(domain)) in lib/actions/ (alongside getSystemPrompt/saveSystemPrompt in lib/actions/chat.ts, or a new lib/actions/system-prompt.ts module if separation is preferred).
   • Resolve the authenticated user (getCurrentUserIdOnServer) and return an { error } result if unauthenticated, consistent with other actions/routes.
   • Normalize/validate the incoming domain (accept bare domains or full URLs) using a small Zod check, consistent with the URL validation in lib/schema/retrieve.tsx.
   • Insert a new job row with status pending (Task 1.1 helper), then trigger the background execution (Task 2.2) as a detached, fire-and-forget call — explicitly not awaited so the action returns before any scrape/AI work begins. The kickoff action's response is decoupled from the work's completion entirely.
   • Return { jobId } immediately so the frontend can begin polling; the request that started the job carries no result and holds no connection for its duration.

   Task 2: Implement the background scrape + prompt synthesis worker

   Run the heavy work independently of the request and persist every progress transition to the durable job row.

   • Implement an internal async function that runs with no reference to the original request/response (no streaming back to the caller, no request-scoped context): given a jobId and domain, it owns the job's progress states and writes them to the DB.
   • First transition the job from pending to processing (advancing updatedAt) so the frontend immediately observes forward progress.
   • Use the Firecrawl client to scrape the domain and extract its main textual content (business description, products/services, tone).
   • Feed the scraped content into the AI model (using the provider helpers in lib/utils/index.ts, as agents in lib/agents/ do) with a meta-instruction asking the model to produce a concise business copilot system prompt describing the company, its domain, and how the assistant should behave.
   • Constrain the generated output to the existing system-prompt length limits (10–2000 chars, matching settingsFormSchema).
   • On success, transition the job to complete with resultPrompt; on any failure, transition it to error with a user-readable errorMessage. All state lives in the job row, so progress is observable purely via polling regardless of the kickoff request's status.

   Task 3: Add a job status-check action for polling

   Expose the durable progress state so the frontend can observe the lifecycle and retrieve results, independent of the kickoff request.

   • Add a server action (e.g., getSystemPromptGenerationJob(jobId)) in the same module as Task 2.1 that fetches the job by id, scoped to the authenticated user, and returns the current progress state { status, resultPrompt, errorMessage } (or { error }).
   • This action reads only the persisted job row — it has no link to the background worker or the original kickoff request — making progress observable across page reloads or even a different session/tab.
   • Keep it lightweight (single DB read) so it is safe to call repeatedly on an interval.
   • (Server action preferred over a new app/api/ route to stay consistent with how Settings already loads/saves via actions; an app/api/system-prompt-jobs/[id]/route.ts GET endpoint is an acceptable alternative if a fetch-based poll is preferred.)

   Task 4: Define failure, timeout, and process-restart resilience

   Make failure modes explicit and ensure the job model is resilient to the worker dying or the process restarting mid-flight — never leaving a job stuck in a non-terminal state.

   • In the worker (Task 2.2), explicitly handle and map to a terminal error state with a distinct, user-readable errorMessage: Firecrawl timeout/unreachable, empty/insufficient scraped content, and AI generation failure — rather than leaving the job in processing.
   • Wrap the worker in try/finally so the row is always finalized to a terminal state (complete or error) within the same process run.
   • Add a bounded execution timeout inside the worker (e.g., abort scrape/generation past a max duration) that transitions the job to error, so a hung external call cannot pin a job in processing indefinitely.
   • Account for process restarts: because a detached worker is lost if the standalone Node container restarts, treat jobs left in pending/processing beyond a staleness threshold (detected via the updatedAt heartbeat from Task 1.1) as failed — surfaced to the polling client as a recoverable error/timeout rather than an infinite spinner. Implement this as a lazy check at status-read time (Task 2.3) and/or a guard in the kickoff path, avoiding any new cron/queue infrastructure.
   • Make retry a first-class path: allow the kickoff action (Task 2.1) to be re-invoked for the same domain, creating a fresh pending job; the frontend simply starts a new poll against the new jobId. No partial state from a failed/stale job blocks a new attempt.

   🤖 Prompt for AI agents

   Implement the backend split into a fast kickoff action, a detached background
   worker, a polling status action, and explicit failure/timeout/process-restart
   resilience. No part of the long-running work should be tied to the originating
   request.
   
   **Kickoff Action (`startSystemPromptGeneration(domain)`)** in
   `lib/actions/chat.ts` or a new `lib/actions/system-prompt.ts`:
   - Resolve the authenticated user via `getCurrentUserIdOnServer`; return `{ error
   }` if unauthenticated.
   - Validate and normalize the `domain` input (bare domains or full URLs) using a
   Zod check consistent with `lib/schema/retrieve.tsx`.
   - Insert a new job row with status `pending` using the Phase 1 DB helpers.
   - Fire the background worker as a detached, fire-and-forget call — explicitly
   **not awaited** — so the action returns before any scrape/AI work begins. The
   kickoff action's response is entirely decoupled from the work's completion.
   - Return `{ jobId }` immediately; the request carries no result and holds no
   connection for the work's duration.
   
   **Background Worker** (internal async function, same module, no reference to the
   original request/response):
   - Accept `jobId` and `domain`; first transition the job from `pending` to
   `processing` (advancing `updatedAt`) so the frontend immediately observes
   forward progress.
   - Use the Firecrawl client to `scrape` the domain and extract main textual
   content (business description, products/services, tone).
   - Feed scraped content into the AI model via provider helpers in
   `lib/utils/index.ts` (following patterns in `lib/agents/`), with a
   meta-instruction to produce a concise business copilot system prompt.
   - Constrain generated output to 10–2000 characters (matching
   `settingsFormSchema` limits).
   - On success, transition the job to `complete` with `resultPrompt`.
   - Wrap all work in try/finally to guarantee the job row is always finalized to a
   terminal state; handle Firecrawl timeout/unreachable, empty/insufficient scraped
   content, and AI generation failure — each sets status `error` with a distinct,
   user-readable `errorMessage`.
   - Add a bounded execution timeout inside the worker that transitions the job to
   `error` if scrape/generation exceeds a max duration, preventing hung external
   calls from pinning a job in `processing` indefinitely.
   
   **Status Action (`getSystemPromptGenerationJob(jobId)`)** in the same module:
   - Fetch the job by id scoped to the authenticated user.
   - Before returning, apply a staleness check: if the job is still in
   `pending`/`processing` but `updatedAt` exceeds a staleness threshold (detecting
   process-restart job loss), treat it as a recoverable `error`/timeout rather than
   an infinite spinner.
   - Return `{ status, resultPrompt, errorMessage }` or `{ error }`; single DB
   read, safe to call on a polling interval.
   - This action has no link to the background worker or the original kickoff
   request — progress is observable across page reloads or different sessions.
   
   **Retry:** The kickoff action always creates a fresh job row, so retry is
   handled by re-invoking the kickoff and polling the new `jobId`. No partial state
   from a failed/stale job blocks a new attempt.
   

   Phase 3: Settings UI — Domain Entry, Progress Polling & Result Handling

   Surface the capability in the existing Settings System Prompt tab: a domain input that kicks off the decoupled job, a UI that reflects the job's explicit progress states (pending/processing loading, complete, error) purely from polling, population of the textarea on success, and clear error/retry handling that does not depend on keeping the original kickoff request alive.

   Task 1: Extend the settings form schema and defaults

   Add the domain field to the form model.

   • Update settingsFormSchema in components/settings/components/settings.tsx to include an optional domain field (URL/domain string, optional or empty literal, mirroring the existing optional newUserEmail pattern).
   • Add a corresponding default value for domain in defaultValues.

   Task 2: Add the domain input and "Generate from domain" trigger

   Provide the UI entry point that kicks off the decoupled async job and immediately reflects its initial progress state.

   • Update components/settings/components/system-prompt-form.tsx to render a domain <Input> (using the same FormField pattern as the textarea) above the existing system-prompt textarea.
   • Add a "Generate from domain" button that, on click, calls the kickoff action (Task 2.1) with the entered domain and stores the returned jobId in local component state. Because kickoff returns immediately, the UI transitions to a pending progress indicator instantly without waiting for the scrape/generation.
   • Disable the button and show a loading/pending indicator once a job is in flight (driven by the job's progress state, not by an open request).

   Task 3: Implement progress polling, completion population, and error/retry UI

   Track the in-flight job by observing its persisted progress states and resolve it into the editor or an error state.

   • Add a polling loop (local useState + useEffect with setInterval, consistent with the codebase's local-state approach — no Zustand needed) that calls the status action (Task 2.3) on an interval while a jobId is active, reading the job's current progress state.
   • Drive the UI directly from the lifecycle states: render pending/processing as loading, complete as success, and error as failure; clear the interval on terminal states and enforce a max-poll timeout (surfaced as an error state) so the UI never hangs even if the background worker was lost to a process restart (Task 2.4).
   • On complete, set the systemPrompt form value via form.setValue so the generated prompt populates the textarea and respects the live character counter; show a success toast.
   • On error (or timeout/stale), show a destructive toast with the errorMessage and re-enable the "Generate from domain" button so the user can retry, which re-kicks off a fresh job per Task 2.4.
   • Keep the existing Save flow unchanged — once populated, the user saves through the current onSubmit/saveSystemPrompt path.

   🤖 Prompt for AI agents

   Wire up the domain entry UI, async job trigger, progress polling loop, and
   result/error handling inside the existing Settings System Prompt tab. The UI is
   driven entirely by polling the persisted job state — not by any open request.
   
   **Schema & Defaults** in `components/settings/components/settings.tsx`:
   - Add an optional `domain` field to `settingsFormSchema` (URL/domain string,
   optional or empty literal), mirroring the existing optional `newUserEmail`
   pattern.
   - Add a corresponding default value for `domain` in `defaultValues`.
   
   **Domain Input & Trigger** in
   `components/settings/components/system-prompt-form.tsx`:
   - Render a domain `<Input>` using the same `FormField` pattern as the existing
   textarea, placed above the system-prompt textarea.
   - Add a "Generate from domain" button that on click calls
   `startSystemPromptGeneration(domain)` (Phase 2 kickoff action) with the entered
   domain value and stores the returned `jobId` in local component state. Because
   kickoff returns immediately, the UI transitions to a `pending` progress
   indicator instantly without waiting for scrape/generation.
   - Disable the button and show a loading/pending indicator while a job is in
   flight (driven by the job's progress state, not by an open request).
   
   **Polling Loop & Result Handling** (in the same component):
   - Use local `useState` + `useEffect` with `setInterval` (no Zustand) to poll
   `getSystemPromptGenerationJob(jobId)` on an interval while a `jobId` is active.
   - Drive the UI directly from the lifecycle states: render `pending`/`processing`
   as loading, `complete` as success, and `error` as failure.
   - Clear the interval on terminal states (`complete` or `error`) and enforce a
   max-poll timeout, surfaced as an `error` state, so the UI never hangs even if
   the background worker was lost to a process restart.
   - On `complete`, call `form.setValue` to set the `systemPrompt` field with the
   returned `resultPrompt`, updating the textarea and live character counter; show
   a success `toast`.
   - On `error` or timeout/stale, show a destructive `toast` with the
   `errorMessage` and re-enable the "Generate from domain" button for retry (which
   re-kicks off a fresh job).
   - Do not modify the existing Save flow — the user continues to save via
   `onSubmit`/`saveSystemPrompt` after reviewing the populated prompt.
   

   Research

   This is a Next.js (App Router) application using the Vercel AI SDK for an AI "copilot." User-configurable behavior is driven by a per-user system prompt stored in the users.systemPrompt column (Drizzle/Postgres), read/written via getSystemPrompt/saveSystemPrompt in lib/actions/chat.ts, and consumed by agents in lib/agents/. Users edit this prompt through a tabbed Settings panel (components/settings/) built with react-hook-form + Zod + shadcn. External content fetching already exists as AI tools (Jina, Tavily, Serper) under lib/agents/tools/, with API keys following the *_API_KEY convention and AI model providers aggregated in lib/utils/index.ts.

   Critically, the codebase has no background-job infrastructure: no queue/cron/task-runner packages, no jobs tables, and no status/state columns (only text discriminators like chats.visibility and users.role). API routes (app/api/*/route.ts) all respond synchronously with NextResponse.json and declare no runtime/maxDuration. The app deploys as output: 'standalone' (long-lived Node container, not Vercel serverless), and lib/db/schema.ts (Drizzle) is the single source of truth with migrations under drizzle/migrations. There is one Zustand store (lib/store/settings.ts, map provider only) and no existing polling hooks. Firecrawl is not present anywhere.

   ---

   🚀 Next Steps

   🤖 All AI agent prompts combined

   Task: 1
   
   Establish the durable job state model and Firecrawl integration that the rest of
   the feature depends on. All progress and results will be read from the database,
   never from in-request memory.
   
   **Jobs Table & Migration** in `lib/db/schema.ts`:
   - Add a new `prompt_generation_jobs` table following existing Drizzle
   conventions with columns: `id` (uuid pk), `userId` (uuid), `domain` (text),
   `status` (text discriminator defaulting to `pending`, with values `pending` |
   `processing` | `complete` | `error`, mirroring the `text`-as-enum pattern used
   by `chats.visibility`/`users.role`), `resultPrompt` (text, nullable),
   `errorMessage` (text, nullable), `createdAt`, `updatedAt`.
   - Treat `updatedAt` as a heartbeat/progress timestamp — it must advance on every
   status transition so stuck/abandoned jobs (e.g., after a process restart) can be
   detected later.
   - Generate a corresponding Drizzle migration under `drizzle/migrations`,
   consistent with the existing `0000_*`/`0001_*` migration files.
   - Add minimal DB helper functions (create job, update status/result, fetch job
   by id+userId) either in `lib/actions/chat-db.ts` or a small new module, matching
   the existing data-access split.
   
   **Firecrawl Setup:**
   - Add the `firecrawl` npm package to `package.json`.
   - Add `FIRECRAWL_API_KEY` to `.env.local.example` alongside the other
   `*_API_KEY` entries (Tavily, Serper, Composio).
   - Initialize the Firecrawl client server-side only, reading from
   `process.env.FIRECRAWL_API_KEY`, following the same pattern as Tavily/Serper
   client construction in `lib/agents/tools/`.
   ===============================================================================
   
   Task: 2
   
   Implement the backend split into a fast kickoff action, a detached background
   worker, a polling status action, and explicit failure/timeout/process-restart
   resilience. No part of the long-running work should be tied to the originating
   request.
   
   **Kickoff Action (`startSystemPromptGeneration(domain)`)** in
   `lib/actions/chat.ts` or a new `lib/actions/system-prompt.ts`:
   - Resolve the authenticated user via `getCurrentUserIdOnServer`; return `{ error
   }` if unauthenticated.
   - Validate and normalize the `domain` input (bare domains or full URLs) using a
   Zod check consistent with `lib/schema/retrieve.tsx`.
   - Insert a new job row with status `pending` using the Phase 1 DB helpers.
   - Fire the background worker as a detached, fire-and-forget call — explicitly
   **not awaited** — so the action returns before any scrape/AI work begins. The
   kickoff action's response is entirely decoupled from the work's completion.
   - Return `{ jobId }` immediately; the request carries no result and holds no
   connection for the work's duration.
   
   **Background Worker** (internal async function, same module, no reference to the
   original request/response):
   - Accept `jobId` and `domain`; first transition the job from `pending` to
   `processing` (advancing `updatedAt`) so the frontend immediately observes
   forward progress.
   - Use the Firecrawl client to `scrape` the domain and extract main textual
   content (business description, products/services, tone).
   - Feed scraped content into the AI model via provider helpers in
   `lib/utils/index.ts` (following patterns in `lib/agents/`), with a
   meta-instruction to produce a concise business copilot system prompt.
   - Constrain generated output to 10–2000 characters (matching
   `settingsFormSchema` limits).
   - On success, transition the job to `complete` with `resultPrompt`.
   - Wrap all work in try/finally to guarantee the job row is always finalized to a
   terminal state; handle Firecrawl timeout/unreachable, empty/insufficient scraped
   content, and AI generation failure — each sets status `error` with a distinct,
   user-readable `errorMessage`.
   - Add a bounded execution timeout inside the worker that transitions the job to
   `error` if scrape/generation exceeds a max duration, preventing hung external
   calls from pinning a job in `processing` indefinitely.
   
   **Status Action (`getSystemPromptGenerationJob(jobId)`)** in the same module:
   - Fetch the job by id scoped to the authenticated user.
   - Before returning, apply a staleness check: if the job is still in
   `pending`/`processing` but `updatedAt` exceeds a staleness threshold (detecting
   process-restart job loss), treat it as a recoverable `error`/timeout rather than
   an infinite spinner.
   - Return `{ status, resultPrompt, errorMessage }` or `{ error }`; single DB
   read, safe to call on a polling interval.
   - This action has no link to the background worker or the original kickoff
   request — progress is observable across page reloads or different sessions.
   
   **Retry:** The kickoff action always creates a fresh job row, so retry is
   handled by re-invoking the kickoff and polling the new `jobId`. No partial state
   from a failed/stale job blocks a new attempt.
   ===============================================================================
   
   Task: 3
   
   Wire up the domain entry UI, async job trigger, progress polling loop, and
   result/error handling inside the existing Settings System Prompt tab. The UI is
   driven entirely by polling the persisted job state — not by any open request.
   
   **Schema & Defaults** in `components/settings/components/settings.tsx`:
   - Add an optional `domain` field to `settingsFormSchema` (URL/domain string,
   optional or empty literal), mirroring the existing optional `newUserEmail`
   pattern.
   - Add a corresponding default value for `domain` in `defaultValues`.
   
   **Domain Input & Trigger** in
   `components/settings/components/system-prompt-form.tsx`:
   - Render a domain `<Input>` using the same `FormField` pattern as the existing
   textarea, placed above the system-prompt textarea.
   - Add a "Generate from domain" button that on click calls
   `startSystemPromptGeneration(domain)` (Phase 2 kickoff action) with the entered
   domain value and stores the returned `jobId` in local component state. Because
   kickoff returns immediately, the UI transitions to a `pending` progress
   indicator instantly without waiting for scrape/generation.
   - Disable the button and show a loading/pending indicator while a job is in
   flight (driven by the job's progress state, not by an open request).
   
   **Polling Loop & Result Handling** (in the same component):
   - Use local `useState` + `useEffect` with `setInterval` (no Zustand) to poll
   `getSystemPromptGenerationJob(jobId)` on an interval while a `jobId` is active.
   - Drive the UI directly from the lifecycle states: render `pending`/`processing`
   as loading, `complete` as success, and `error` as failure.
   - Clear the interval on terminal states (`complete` or `error`) and enforce a
   max-poll timeout, surfaced as an `error` state, so the UI never hangs even if
   the background worker was lost to a process restart.
   - On `complete`, call `form.setValue` to set the `systemPrompt` field with the
   returned `resultPrompt`, updating the textarea and live character counter; show
   a success `toast`.
   - On `error` or timeout/stale, show a destructive `toast` with the
   `errorMessage` and re-enable the "Generate from domain" button for retry (which
   re-kicks off a fresh job).
   - Do not modify the existing Save flow — the user continues to save via
   `onSubmit`/`saveSystemPrompt` after reviewing the populated prompt.
   

   💡 Iterate on the plan with: @coderabbitai <feedback>

   Example Feedback
   - @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
   - @coderabbitai For design choice 1 go ahead with option 3 and replan.
   

   ---

   💬 Have feedback or questions? Drop into our discord!
3. ngoiyaeric commented on Jun 25, 2026

   

   ngoiyaeric

   on Jun 25, 2026

   Last edited by ngoiyaeric

   CollaboratorAuthor

   More actions

   @coderabbitai plan this should be highly asynchronous
4. coderabbitai commented on Jun 25, 2026

   

   coderabbitaiBot

   on Jun 25, 2026 – with coderabbitai

   Contributor

   More actions

   I've updated the plan above based on your feedback. Ready to proceed when you are!
5. coderabbitai commented on Jun 25, 2026

   

   coderabbitaiBot

   on Jun 25, 2026 – with coderabbitai

   Contributor

   More actions

   I've updated the plan above based on your feedback. Ready to proceed when you are!