Repository navigation
TEST-201 docs: bring Simulations pages up to date with Composer test generation and the run API - #1342
Merged
scott-lowe-vapi merged 2 commits intoOct 27, 2026
Conversation
This was referenced Oct 19, 2026
Lightsage docs evalsLightsage could not queue docs evals for this PR. Docs URL: https://vapi-preview-01a1527c-c7e7-742f-bb13-ed300ead57a9.docs.buildwithfern.com |
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a1526d-3cb5-753e-be8d-81136127992c.docs.buildwithfern.com |
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a1527c-c7e7-742f-bb13-ed300ead57a9.docs.buildwithfern.com |
stephenvapiai
left a comment
stephenvapiai
left a comment
Contributor
There was a problem hiding this comment.
Thanks for pulling all of this together. A few edit-pass suggestions inline, mostly wording, plus a couple of questions. Most are suggestion blocks you can apply directly.
One item couldn't go inline because the line isn't part of the diff:
fern/observability/simulations-advanced.mdx:4 (frontmatter description): this PR adds the "Handoffs and transfers in simulations" and "Hooks and call endings" sections, but the description (which feeds llms.txt) doesn't mention them. Suggested:
description: "Configure simulation variables, tool mocks, lifecycle webhooks, latency limits, and structured outputs, and see how handoffs and hooks behave in simulations."
…generation and the run API Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scott-lowe-vapi
force-pushed
the
scottlowe/test-201-simulations-docs-catch-up
branch
from
October 22, 2026 06:31
d16141f to
15fb590
Compare
Contributor
|
🌿 Preview your docs: https://vapi-preview-01a16267-0e8d-717d-ab19-935893ef0b4f.docs.buildwithfern.com |
stephenvapiai
approved these changes
Oct 22, 2026
scott-lowe-vapi
added a commit
that referenced
this pull request
Oct 30, 2026
## Description **Draft: merge when VapiAI/vapi#23422 (TEST-200) deploys.** That PR publishes the onboarding endpoints and lists this guide as a merge gate. Linear: [TEST-203](https://linear.app/vapi/issue/TEST-203/document-the-public-simulations-onboarding-api-and-its-on-call-runbook). Stacked on #1342. - New page **Generate a simulation suite with the API** (`/observability/simulations-generate-api`), added to the nav after Advanced. It covers: - beta key access, and the 404 for orgs without it - the five-step flow with cURL: draft, revise, generate with `idempotencyKey`, edit with `patchKey`, run with `Idempotency-Key` - request and response field tables - coverage and tool gap reasons - idempotency and retries: the in-progress and key-reused 409s - daily limits: 60 drafts, 20 generations and 60 edits per UTC day, at most 2 generating at once, with what doesn't count - the error table - **Advanced:** troubleshooting rows for the TEST-192 run-create 400 (VapiAI/vapi#23407) and the `Failed to create target call` failure reason (VapiAI/vapi#23400). - **Composer:** a Limitations bullet on the shared daily allowance. Sources are the open stack's diffs (#23400, #23407, #23418–#23422). The 524 in the retry list comes from the endpoint's own operation description. **Before merge:** - Re-check every value against the merged code. - Limits were verified against the code: they apply to all organizations, including Enterprise; no request field raises them. - Confirm "contact support" is the right path for beta key access. - Link the API reference pages once `fern api update` pulls the routes in. **What's New blurb:** > **Generate a simulation suite with the API (beta).** Three new endpoints under `/eval/simulation/onboarding` run the same test-generation flow as Composer: draft numbered requirements from an assistant's prompt, tools, and handoffs, revise them with your feedback, then generate a saved suite whose simulations mock every tool that would reach an external system. The response reports which requirements are covered, which aren't and why, and any handoffs or transfers the simulations don't exercise. Edit the suite in place, then run it with `POST /eval/simulation/run`; an `idempotencyKey` makes every retry safe. Private API keys work for organizations enabled during the beta. Contact support to ask about beta access; and generation shares a free daily allowance per organization with Composer that resets at 00:00 UTC. Read [Generate a test suite with the API](/observability/simulations-generate-api). ## Testing Steps - [x] `fern check` (5.151.1) passes: 0 errors. The 15 warnings match the untouched main baseline. - [ ] Check the preview deployment once the endpoints are live on staging
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Linear: TEST-201. Companion in-repo docs: VapiAI/vapi#23428.
/composer#generate-tests-for-an-assistant): requirements draft → Generate coverage → suite card with mocked tools and not-tested gaps → Run suite, plus edits and a tip to run five or more iterations.discard-draft, for your own unpublished draft.customer.speech.timeoutand the tester hang-up ending ascustomer-ended-call.Idempotency-Keyheader onPOST /eval/simulation/run.suiteId,targetType,targetId,status,simulationId,assistantId,squadId,name).{ metadata, results }.Evidence
Every behavior was checked against VapiAI/vapi
main:features/simulationsOnboarding/constants.ts.simulationRunIdempotencyKeyParse.tsandsimulationRunIdempotentReplayGet.ts.simulationRun.types.tsandGetSimulationSuitesDTO.ts.handoffSimulationRefusalGet.tsandsimulationTargetReengagementHooks/.Testing Steps
fern check(5.151.1) passes: 0 errors. The 15 warnings match the untouched main baseline.