feat: Mini Browser — a live Chromium that agents drive and users can take over

[korivi-CraftOS]

Adds the Mini Browser: a real Chromium that CraftBot's agents drive while the user watches live in a new sidebar page, and can take over at any moment. It replaces the earlier "Web Agent" prototype, rebuilt to fit current main (sessions, i18n, provisioning, WebSocket lanes).

Commits: the first imports the old prototype onto main, the next two replace it, the fourth and the last two address the review comments, and the fifth and sixth fix a retype loop on Threads and let pages play sound. Review the branch as a whole, not commit by commit.

What users get

• A Mini Browser item in the sidebar. The page has a live view, tabs, a URL bar, saved passwords, and a chat with its own session where the browser tools are preloaded.
• The agent moves the mouse and types the way a person does, so the user can follow along. Clicking into the page takes control, which pauses the agent on that tab immediately, even mid-typing. Hand back resumes it.
• Saved logins: the agent signs in through an encrypted password vault and never sees the password.
• Pages can play sound (videos, calls, notifications) through the computer's speakers. Turn it off with mini_browser.sound: false.
• Post boxes (Threads, X and similar): addresses are entered whole, the box is read back after typing, the page view shows the whole post (never half a link), and the agent is stopped from deleting and retyping the same post in a loop. Checked against the real Lexical editor with its auto-link plugin.
• The page is fully translated (all 7 locales), and the live view accepts IME input (Japanese etc.), paste and shortcuts.

How it works

• app/mini_browser/ is a new package that is import-light: it never imports Playwright at import time.
  • Threading: a dedicated mini-browser thread with its own event loop owns Playwright. Every caller marshals in via host.call(), so it works from the main loop and from sub-agents, which run on their own loops.
  • Tabs: each agent session gets its own tab in its own Chromium window, keyed on _session_id. Without separate windows, background tabs are throttled to about 1 s per input event.
  • Live view: CDP screencast of the viewed tab, streamed only while a UI tab is subscribed. Frames are latest-wins and never queued.
  • Ad blocking: CDP Network.setBlockedURLs. context.route is not used because it disables the HTTP cache.
  • Blocking CraftBot's own UI: the UI origins are blocked browser-wide, however the address is spelled.
  • Resilience: crash and driver-death recovery, a timeout on every Playwright call, idle shutdown, a cross-process profile lock, and a clean close during launch.
• Agent actions: app/data/action/mini_browser_actions.py holds 13 thin mini_browser_* stubs: navigate, read, click, type, press_key, select_option, hover, scroll, wait, upload_file, login, screenshot and tabs. Every result carries a fresh, compact page observation, so agents need fewer LLM round-trips.
• WebSocket protocol: mini_browser_* messages are handled in app/mini_browser/ws.py. Navigation, input, control and the vault each get their own lane, so a frozen page can't block "Take control".

Every kind of agent can use it

• Main and chat sessions, the dedicated Mini Browser chat (new session type mini_browser), scheduled runs, Agent App trigger runs, and a new browser_agent sub-agent type.
• Agents discover it through:
  • the catalog description;
  • a when-to-use rule in the action prompt, shown only when the browser can actually run;
  • an enabled mini-browser skill;
  • AGENT.md.
• Stop cancels in-flight browser work for the session and its sub-agents.
• Tab privacy: agents see only their own tabs' URLs. They can't take over a tab the user opened unless it is blank, or the user is viewing it and the agent switches to it explicitly.

Password vault

• Fernet-encrypted, with the key wrapped by Windows DPAPI, stored in .credentials/ (0700 dir, 0600 files).
• Writes are atomic. An unreadable vault is never overwritten: it can only be reset explicitly, which keeps a backup.
• Autofill happens only on the exact site the login was saved for (with or without www.), over https or on localhost, and only for the current page's origin.
• Passwords never reach the model, the logs or the UI, including their URL-, HTML- and JSON-encoded forms.

Changes outside app/mini_browser/ (please review closely)

File	Change
app/agent_base.py	Lifecycle hooks: run state, Stop, chat deletion, sub-agent removal, shutdown, settings hot-reload. A full /reset now recreates the streams of surviving sessions. The Mini Browser chat can be cleared but not deleted.
run.py, app/state_sync.py	Managed installs now receive newly shipped action files and skills on upgrade, using a manifest. User edits are never overwritten and deleted files never come back. This also fixes the gap for any future new action.
agent_core/core/impl/action/router.py	Optional prompt_filter hook, used to hide the Mini Browser rule when the browser is unavailable.
agent_core/core/session/session.py, impl/session/manager.py	New session type mini_browser.
agent_core/core/prompts/action.py, context.py	When-to-use rule for websites that must be used, not just read.
app/internal_action_interface.py	add_action_sets matches names loosely (-/_, case) and reports unknown sets with hints instead of silently succeeding.
app/ui_layer/adapters/browser_adapter.py, ws_lanes.py	One delegation branch, a strict send-to-requester helper, start/stop hooks, new lanes.
app/config.py, app/errors/codebook.py	mini_browser settings section and the Mini Browser error codes.
skills/playwright-mcp/SKILL.md	Its bare browser_* tool names collided with the new actions, so it now uses the real mcp_playwright-mcp_* names and points user-facing browsing to the Mini Browser.
.gitignore, .dockerignore	Profile and lock-file patterns, so secrets and cookies never get committed or baked into images.
Frontend outside the page	Route and sidebar item; miniBrowser slice; resource; UI state; ToastContext gains optional action buttons; ActivityBlocks previews for mini_browser_* actions; a small onboarding fix so long error text wraps.

Testing

• Full Python suite: 2452 passed, 0 failed. Excluded: the 3 test files that already fail to collect on main (tests/llm/golden, test_installer_stream_routing, test_prompt_profile) and the stale macOS shortcut test in test_craftbot_service.
• 16 new or extended test files, many driving real headless Chromium against local test servers. They cover:
  • vault matching against lookalike domains, and password masking;
  • stale element ids;
  • concurrent owners from several loops and threads;
  • close during launch, crash recovery, and take-control mid-typing;
  • downloads, popups and URL policy;
  • the WebSocket protocol and lanes;
  • the upgrade sync;
  • every action stub executed exactly as the action executor runs it;
  • the review fixes: no agent input after take-control, tab limits under concurrent opens, screenshot masking, download names, copy/cut, and both settings readers agreeing.
• Browser tests skip only when Playwright or Chromium is not installed; any other startup failure fails them. CRAFTBOT_REQUIRE_BROWSER_TESTS=1 makes a missing browser a failure too.
• ruff format --check . and ruff check . are clean (ruff 0.16.1, same as CI), and compileall passes.
• Frontend: tsc --noEmit reports 0 errors, the production vite build succeeds, and the i18n keys match across all 7 locales.
• An internal multi-agent review (concurrency, security, agent effectiveness, platform integration, frontend, end-to-end) confirmed 53 issues. All are fixed with regression tests; the residual edge cases are listed below.
• No CI workflow runs on PRs to main for these paths, so all of the above was run locally on Windows (Python 3.10, Playwright 1.58 / Chromium 145).

How to try it

1. Check out the branch and start CraftBot as usual (run.py rebuilds the frontend).
2. Click Mini Browser in the sidebar and enter a URL, or ask its chat for a web task, e.g. "find X on Y and add it to the cart".
3. Click into the page to take control, then press Hand back.
4. Save a login under Passwords and ask the agent to sign in.

If Chromium isn't installed (it is an optional provisioning stage), the page offers an Install browser button.

Known limits / follow-ups

• Existing managed installs receive new built-in files, but not updates to built-in files they already have (e.g. web_fetch.py). This predates this PR.
• scripts/package_source.py leaves out the AGENT.md template. This also predates this PR.
• The Docker image ships without Chromium, so the prompt rule is hidden there. Enabling it would add about 500 MB.
• Downloads get the "from the internet" marker (Mark-of-the-Web) on Windows only.
• A login saved for example.com no longer fills on other subdomains. A per-login "also use on subdomains" option would need a UI decision.
• One rare residual: quitting the app during a launch that hangs for more than 90 s leaves the Playwright driver until the process exits.
• Only Windows was exercised end to end. Headed mode and the macOS/Linux profile lock are implemented but untested.
• Follow-ups from the review:
  • give the WebSocket layer a small typed adapter interface, injected, instead of reaching into the adapter's private members;
  • split BrowserCore's browser lifecycle (launch, recovery, teardown) from tab registry and ownership;
  • a CI job that installs Chromium and runs the browser tests with CRAFTBOT_REQUIRE_BROWSER_TESTS=1.

[ahmad-ajmal]

I reproduced the six issues below with real browser tests and isolated storage. End-to-end checks used the production Mini Browser frontend and backend with a stub agent controller. Browser testing used Playwright 1.58 and installed Edge because the downloaded Chromium builds could not launch on this machine.

[ahmad-ajmal]

Four additional maintainability comments. The adapter contract and BrowserCore decomposition comments are follow-up suggestions.

[ahmad-ajmal]

Re-review of ccf5237: the original reproductions now pass, including the slow Ctrl+A/Ctrl+X production UI scenario. These three additional findings require specific timing conditions. I deliberately created the race windows to confirm the behavior; this does not establish how frequently users would encounter it. The dropdown issue is medium severity.

[ahmad-ajmal]

Consolidated review of d9800ff: four requests to address before merge. Each inline comment includes the condition and reproduction. Controlled timing tests establish possible behavior, not production frequency. The earlier reported scenarios now pass.