Skip to content

TEST-46 docs: align guides with voice testing best practices - #1206

Merged
scott-lowe-vapi merged 1 commit into
mainfrom
codex/align-testing-guidance
Sep 10, 2026
Merged

TEST-46 docs: align guides with voice testing best practices#1206
scott-lowe-vapi merged 1 commit into
mainfrom
codex/align-testing-guidance

Conversation

@scott-lowe-vapi

@scott-lowe-vapi scott-lowe-vapi commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Description

  • Align Evals and Simulations guides with the testing best practices introduced in Rewrite voice agent testing best practices #1193.
  • Clarify tool, audio, and external-state boundaries; replace brittle examples with focused decision checks.
  • Replace incomplete CI snippets with a workflow that waits for completion and blocks on critical failures.
  • Add risk-based coverage guidance to tutorials, monitoring, and scorecards; correct stale Eval-suite API descriptions through Fern overrides.
  • Follow-up for TEST-46.

Testing Steps

  • Run the app locally using fern docs dev or navigate to preview deployment. Local preview stalled during startup; visual review remains pending.
  • Ensure that the changed pages and code snippets work. Static checks passed; no live calls or Eval runs were made.
  • fern check: 0 errors, 12 existing API-schema warnings.
  • MDX/YAML syntax checks for all 13 changed files, changed cURL JSON payloads, new internal links, and git diff --check.
  • CI example validated with 12 mocked cases covering passing results, failures, cancellation, missing/empty results, timeout, HTTP/JSON errors, and missing configuration.

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@scott-lowe-vapi
scott-lowe-vapi marked this pull request as ready for review September 9, 2026 22:33
@lightsage-app

lightsage-app Bot commented Sep 9, 2026

Copy link
Copy Markdown

Lightsage docs evals

Result: passed
Staging docs: https://vapi-preview-01a08de0-3571-707a-b978-7459302f1527.docs.buildwithfern.com
Commit: 6c5325e

Average score: 100/100
Passed: 3/3

Eval ID Status Score Model Tools Docs 404
08e24c18-82a0-45de-abdc-d237bd12bc0f Pass 100 claude-code/global.anthropic.claude... 26 0
08e24c18-82a0-45de-abdc-d237bd12bc0f Pass 100 cursor/auto 17 1
08e24c18-82a0-45de-abdc-d237bd12bc0f Pass 100 codex/gpt-5.4 39 0

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@stephenvapiai stephenvapiai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only made one change and that was splitting a large paragraph into two.

Comment thread fern/assistants/structured-outputs-examples.mdx Outdated
@github-actions

Copy link
Copy Markdown
Contributor

A record of an eval execution, including its target, status, results,
costs, completion details, and lifecycle timestamps.
properties:
results:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting, I don't know if these overrides can/should get backpropagated into comments in the code, haven't looked into how our autogeneration works

## Description

- Align Evals and Simulations guides with the testing best practices introduced in #1193.
- Clarify tool, audio, and external-state boundaries; replace brittle examples with focused decision checks.
- Replace incomplete CI snippets with a workflow that waits for completion and blocks on critical failures.
- Add risk-based coverage guidance to tutorials, monitoring, and scorecards; correct stale Eval-suite API descriptions through Fern overrides.
- Follow-up for [TEST-46](https://linear.app/vapi/issue/TEST-46/update-docs-for-testing-best-practices).

## Testing Steps

- [ ] Run the app locally using `fern docs dev` or navigate to preview deployment. Local preview stalled during startup; visual review remains pending.
- [ ] Ensure that the changed pages and code snippets work. Static checks passed; no live calls or Eval runs were made.
- [x] `fern check`: 0 errors, 12 existing API-schema warnings.
- [x] MDX/YAML syntax checks for all 13 changed files, changed cURL JSON payloads, new internal links, and `git diff --check`.
- [x] CI example validated with 12 mocked cases covering passing results, failures, cancellation, missing/empty results, timeout, HTTP/JSON errors, and missing configuration.
@scott-lowe-vapi
scott-lowe-vapi force-pushed the codex/align-testing-guidance branch from 0591673 to 6c5325e Compare September 10, 2026 23:40
@scott-lowe-vapi
scott-lowe-vapi merged commit c562c5d into main Sep 10, 2026
7 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants