From a0f18d5f4a6ece2ea364fac8279388abe6cd4206 Mon Sep 17 00:00:00 2001 From: Danyang Chen Date: Wed, 21 Oct 2026 18:15:00 +0800 Subject: [PATCH 1/2] Map user journeys to today's coverage (#358) --- docs/subsystems/live-e2e.md | 4 +++ docs/testing/user-scenarios.md | 51 ++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) create mode 100644 docs/testing/user-scenarios.md diff --git a/docs/subsystems/live-e2e.md b/docs/subsystems/live-e2e.md index 3ba23171f..79b16ddea 100644 --- a/docs/subsystems/live-e2e.md +++ b/docs/subsystems/live-e2e.md @@ -13,6 +13,10 @@ DeepSeek endpoint. Commands, budgets and the evidence rules are in [the suite README](../../dev/live-e2e/README.md); the adapter contract is in [docs/interop/live-e2e-adapter.md](../interop/live-e2e-adapter.md). +Which user journeys this suite is meant to cover, and what covers each one today, is the +[user-scenario matrix](../testing/user-scenarios.md) (#358). The matrix is the map; this +suite is one of its columns, not the map's owner. + ## Owns - `dev/live-e2e/`: the fixture and scenario/assertion modules, the FlyBot web adapter, diff --git a/docs/testing/user-scenarios.md b/docs/testing/user-scenarios.md new file mode 100644 index 000000000..68f7fc990 --- /dev/null +++ b/docs/testing/user-scenarios.md @@ -0,0 +1,51 @@ +# User-scenario matrix + +One table that maps the real user journeys to what covers them **today** and what the +Nightly rewrite should cover next (#358). It is the map; the tests are elsewhere. The +journeys below are the draft from the issue, kept as the proposal; corrections belong in +review, not in a quiet edit. + +Two things this page deliberately does **not** decide: + +- **`Nightly target` is the owner's call.** Every cell reads `待用户定` with the proposal + from [#358](https://github.com/ThinkFlowLab/FlyBot/issues/358) in parentheses, so the + column can be signed off in one pass instead of reading a decision this PR invented. +- **Which journeys are in scope at all** is still the proposal in the issue body. A row is + added or dropped by that discussion, not here. + +Rules for the `Today` column: it must point at something concrete — a test file, +`.github/workflows/nightly.yml`, `.github/workflows/live-e2e.yml`, or +`core/dev/eval/specs.mjs` — or say `none`. A row may not claim coverage without a pointer. + +| ID | Scenario | Persona | Surfaces | Plugins | Today | Nightly target | Blocked by | +| --- | --- | --- | --- | --- | --- | --- | --- | +| P1 | Cold start → first message sendable | personal solo | local Desktop, local web | core | none | 待用户定（提议：perf, #357） | #356, #357 | +| P2 | Single Bot chat with one tool call | personal solo | local web | core | `.github/workflows/live-e2e.yml`, `core/test/rounds.test.js` | 待用户定（提议：live E2E + E2E mock） | — | +| P3 | Tree run: Chief → children → one combined answer | personal solo | local web | core, tree | `core/dev/eval/specs.mjs` (`P3-tree-split`), `core/test/bot-tree.test.js` | 待用户定（提议：live E2E） | — | +| P4 | Invite a friend, message their Bot, approve the held reply | personal + friend | local web | social | `core/dev/eval/specs.mjs` (`P4-invite-approve`), `core/test/held-reply.test.js` | 待用户定（提议：live E2E） | — | +| P5 | Face-to-face room: join, @Bot, leave | personal + friend | local web | social (face rooms) | `core/test/face-people.test.js` | 待用户定 | — | +| P6 | Timeline: read swimlanes after a multi-Bot run | personal solo | local web | timeline | `core/dev/eval/specs.mjs` (`P6-timeline`) | 待用户定（提议：live E2E + perf） | — | +| P7 | Long session: 500+ messages, scroll and reopen | personal solo | local web | core | none | 待用户定（提议：perf, #356/#357） | #356, #357 | +| P8 | Background tab hidden 5 min, then return | personal solo | local web | core | none | 待用户定 | #259 | +| X1 | Slow or failing model API: waiting, timeout and retry are clear | any | local web | core | none | 待用户定（提议：E2E mock） | — | +| X2 | Permission denied / no grant | team member | deployed instance (#352) | social, grants | `core/test/authz.test.js` | 待用户定 | #51, #352 | +| T1 | Assign a task, the owner sees Needs you, approves | team | deployed instance (#352) | inbox, social | none | 待用户定 | #349, #149, #352 | +| T2 | Approval latency across two instances | team | deployed instance (#352) | social | none | 待用户定 | #352 | +| T3 | Reminder: an approval waits too long | personal, team | local web | reminders (#353) | none | 待用户定 | #353 | +| T4 | Scheduled morning summary arrives | personal | local web | reminders (#354) | none | 待用户定 | #354 | +| T5 | Reminder while the desktop app is closed (24/7) | personal | external channel (#355) | reminders, #352 | none | 待用户定 | #355, #352 | + +## How to use it + +1. **A journey changes → the row changes the same day.** The ids are stable: `P*` are + personal-mode journeys, `X*` cross-cutting failures, `T*` team-mode journeys. +2. **`core/dev/eval/specs.mjs` follows these ids.** Its `journey` field is the P-id of the + row it scores (`P3`, `P4`, `P6` today), so a new P-row that gets a canonical rubric + needs a spec with the same id — and the id in this table is what `run.mjs` reports. +3. **The Nightly rewrite reads the `Nightly target` column** once the owner signs it + (#357 wires the metrics into it). Until then, `nightly.yml` stays as it is. + +## Out of scope here + +Rewriting `.github/workflows/nightly.yml` and implementing any missing test. This page is +signed first; the suites follow. From 127998631a3e2f98ba2abbc405cc6e4935d52d7e Mon Sep 17 00:00:00 2001 From: Danyang Chen Date: Wed, 21 Oct 2026 18:15:37 +0800 Subject: [PATCH 2/2] Keep the matrix in English; Nightly target stays TBD --- docs/testing/user-scenarios.md | 37 +++++++++++++++++----------------- 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/docs/testing/user-scenarios.md b/docs/testing/user-scenarios.md index 68f7fc990..2721bc4ee 100644 --- a/docs/testing/user-scenarios.md +++ b/docs/testing/user-scenarios.md @@ -7,9 +7,10 @@ review, not in a quiet edit. Two things this page deliberately does **not** decide: -- **`Nightly target` is the owner's call.** Every cell reads `待用户定` with the proposal - from [#358](https://github.com/ThinkFlowLab/FlyBot/issues/358) in parentheses, so the - column can be signed off in one pass instead of reading a decision this PR invented. +- **`Nightly target` is the owner's call.** Every cell reads `TBD (owner decides)`, with + the proposal from [#358](https://github.com/ThinkFlowLab/FlyBot/issues/358) in + parentheses where the issue made one, so the column can be signed off in one pass + instead of reading a decision this PR invented. - **Which journeys are in scope at all** is still the proposal in the issue body. A row is added or dropped by that discussion, not here. @@ -19,21 +20,21 @@ Rules for the `Today` column: it must point at something concrete — a test fil | ID | Scenario | Persona | Surfaces | Plugins | Today | Nightly target | Blocked by | | --- | --- | --- | --- | --- | --- | --- | --- | -| P1 | Cold start → first message sendable | personal solo | local Desktop, local web | core | none | 待用户定（提议：perf, #357） | #356, #357 | -| P2 | Single Bot chat with one tool call | personal solo | local web | core | `.github/workflows/live-e2e.yml`, `core/test/rounds.test.js` | 待用户定（提议：live E2E + E2E mock） | — | -| P3 | Tree run: Chief → children → one combined answer | personal solo | local web | core, tree | `core/dev/eval/specs.mjs` (`P3-tree-split`), `core/test/bot-tree.test.js` | 待用户定（提议：live E2E） | — | -| P4 | Invite a friend, message their Bot, approve the held reply | personal + friend | local web | social | `core/dev/eval/specs.mjs` (`P4-invite-approve`), `core/test/held-reply.test.js` | 待用户定（提议：live E2E） | — | -| P5 | Face-to-face room: join, @Bot, leave | personal + friend | local web | social (face rooms) | `core/test/face-people.test.js` | 待用户定 | — | -| P6 | Timeline: read swimlanes after a multi-Bot run | personal solo | local web | timeline | `core/dev/eval/specs.mjs` (`P6-timeline`) | 待用户定（提议：live E2E + perf） | — | -| P7 | Long session: 500+ messages, scroll and reopen | personal solo | local web | core | none | 待用户定（提议：perf, #356/#357） | #356, #357 | -| P8 | Background tab hidden 5 min, then return | personal solo | local web | core | none | 待用户定 | #259 | -| X1 | Slow or failing model API: waiting, timeout and retry are clear | any | local web | core | none | 待用户定（提议：E2E mock） | — | -| X2 | Permission denied / no grant | team member | deployed instance (#352) | social, grants | `core/test/authz.test.js` | 待用户定 | #51, #352 | -| T1 | Assign a task, the owner sees Needs you, approves | team | deployed instance (#352) | inbox, social | none | 待用户定 | #349, #149, #352 | -| T2 | Approval latency across two instances | team | deployed instance (#352) | social | none | 待用户定 | #352 | -| T3 | Reminder: an approval waits too long | personal, team | local web | reminders (#353) | none | 待用户定 | #353 | -| T4 | Scheduled morning summary arrives | personal | local web | reminders (#354) | none | 待用户定 | #354 | -| T5 | Reminder while the desktop app is closed (24/7) | personal | external channel (#355) | reminders, #352 | none | 待用户定 | #355, #352 | +| P1 | Cold start → first message sendable | personal solo | local Desktop, local web | core | none | TBD (owner decides); proposal: perf, #357 | #356, #357 | +| P2 | Single Bot chat with one tool call | personal solo | local web | core | `.github/workflows/live-e2e.yml`, `core/test/rounds.test.js` | TBD (owner decides); proposal: live E2E + E2E mock | — | +| P3 | Tree run: Chief → children → one combined answer | personal solo | local web | core, tree | `core/dev/eval/specs.mjs` (`P3-tree-split`), `core/test/bot-tree.test.js` | TBD (owner decides); proposal: live E2E | — | +| P4 | Invite a friend, message their Bot, approve the held reply | personal + friend | local web | social | `core/dev/eval/specs.mjs` (`P4-invite-approve`), `core/test/held-reply.test.js` | TBD (owner decides); proposal: live E2E | — | +| P5 | Face-to-face room: join, @Bot, leave | personal + friend | local web | social (face rooms) | `core/test/face-people.test.js` | TBD (owner decides) | — | +| P6 | Timeline: read swimlanes after a multi-Bot run | personal solo | local web | timeline | `core/dev/eval/specs.mjs` (`P6-timeline`) | TBD (owner decides); proposal: live E2E + perf | — | +| P7 | Long session: 500+ messages, scroll and reopen | personal solo | local web | core | none | TBD (owner decides); proposal: perf, #356/#357 | #356, #357 | +| P8 | Background tab hidden 5 min, then return | personal solo | local web | core | none | TBD (owner decides) | #259 | +| X1 | Slow or failing model API: waiting, timeout and retry are clear | any | local web | core | none | TBD (owner decides); proposal: E2E mock | — | +| X2 | Permission denied / no grant | team member | deployed instance (#352) | social, grants | `core/test/authz.test.js` | TBD (owner decides) | #51, #352 | +| T1 | Assign a task, the owner sees Needs you, approves | team | deployed instance (#352) | inbox, social | none | TBD (owner decides) | #349, #149, #352 | +| T2 | Approval latency across two instances | team | deployed instance (#352) | social | none | TBD (owner decides) | #352 | +| T3 | Reminder: an approval waits too long | personal, team | local web | reminders (#353) | none | TBD (owner decides) | #353 | +| T4 | Scheduled morning summary arrives | personal | local web | reminders (#354) | none | TBD (owner decides) | #354 | +| T5 | Reminder while the desktop app is closed (24/7) | personal | external channel (#355) | reminders, #352 | none | TBD (owner decides) | #355, #352 | ## How to use it