Extend testkit with Playwright UI and API test scaffolding (sub-issue of plugins-skills#13) #7

Summary

The testkit today provides Python/pytest assertions against A0's extension points (assert_extension_at_surface, assert_js_at_hook, assert_no_dead_plugin_hooks, etc.). These catch structural drift at PR time.

What's missing is the runtime test surface — actually exercising the plugin's UI and API against a live A0 instance. This sub-issue tracks adding that layer to the testkit so every plugin can declare its UI/API tests using shared scaffolding.

Parent umbrella: agent-zero-plugins/agent-zero-plugins-skills#13 — Track 2 ("Standard test surface").

Motivation

• The structural assertions confirm the surface exists but not the surface behaves. Install/uninstall flows pass structurally but break visibly in the UI all the time.
• Plugin authors today reinvent UI/API test wiring for every plugin → uneven coverage, repeated boilerplate, fragile selectors.
• Autonomous coding agents need a single pattern they can follow without inventing test infrastructure each time.

Maps to PRINCIPLES.md §10 (testing pyramid, offline-first) — adds the runtime layers above the existing structural ones.

Proposal

Add three new modules to the testkit:

1. a0_plugin_testkit.playwright

A thin wrapper around Playwright that knows about A0's WebUI conventions:

• Standard fixtures for: launching the A0 container, logging in, opening the plugin's settings panel
• Helpers for common flows: install_plugin, enable_plugin, disable_plugin, uninstall_plugin, open_settings, save_settings
• Selector helpers tied to A0's stable DOM landmarks (not brittle CSS-in-flux)
• Screenshot capture on failure (already a pattern in some plugin tests; standardize)

from a0_plugin_testkit.playwright import A0Page

def test_mermaid_install_uninstall(a0_page: A0Page):
    a0_page.install_plugin("mermaid_diagrams")
    assert a0_page.plugin_is_enabled("mermaid_diagrams")
    a0_page.uninstall_plugin("mermaid_diagrams")
    assert not a0_page.plugin_is_installed("mermaid_diagrams")

2. a0_plugin_testkit.api

A curl/httpx wrapper that handles A0 API conventions:

• Token acquisition (login → bearer extraction)
• Common endpoint helpers: get_settings(plugin), update_settings(plugin, payload), health(), plugin_status(name)
• Response assertion helpers (status code, schema, error shape)
• Reusable across all plugins regardless of their custom endpoints

3. a0_plugin_testkit.taxonomy

A decorator-based way to annotate test class for the CI:

@testkit.smoke
def test_install(): ...

@testkit.functional
def test_configure_flow(): ...

@testkit.privileged(scopes=["read:secrets"])
def test_with_user_token(): ...

@testkit.regression(linked_issue=42)
def test_specific_bug(): ...

The CI workflow (umbrella Track 3) reads these annotations to decide what runs where (e.g. skip privileged on fork PRs).

Open questions

• Container launch fixture — does the testkit own the container lifecycle, or does the CI workflow provide a running instance via env var? (Probably both modes — local dev vs CI.)
• Headless vs headed Playwright — defaults? Override via env?
• Selector strategy: data-test-id attributes in A0 webui (cleanest), or fall back to ARIA roles? Coordinate with nuevanext/agent-zero since selectors are stable contracts.
• Token storage in tests — env var? pytest.ini? Fixture-managed?
• Should the API helpers cover MCP tool invocation too, or strictly REST?

Acceptance criteria

1. a0_plugin_testkit.playwright, a0_plugin_testkit.api, a0_plugin_testkit.taxonomy modules shipped with documented public APIs
2. At least one existing plugin retrofitted to use these modules end-to-end (Mermaid or Share-Chat as a small-surface pilot)
3. The reusable CI workflow (umbrella Track 3) consumes the taxonomy annotations
4. Selector strategy decided + documented (likely a SPEC.md section)
5. README of the testkit updated with the new public surface

Related

• Parent umbrella: agent-zero-plugins/agent-zero-plugins-skills#13
• agent-zero-plugins/agent-zero-plugins-skills — where the CI workflow that consumes this will live
• nuevanext/agent-zero — selector contracts need coordination
• PRINCIPLES.md §10 (testing pyramid), §7 (submodule-based distribution since this is consumed via the skills submodule)