Skip to content

feat(evaluations)!: add test proving harness edge case matches spec - #200

Merged
aknight-ld merged 1 commit into
mainfrom
AIC-3642-match-spec-harness
Oct 22, 2026
Merged

feat(evaluations)!: add test proving harness edge case matches spec#200
aknight-ld merged 1 commit into
mainfrom
AIC-3642-match-spec-harness

Conversation

@aknight-ld

@aknight-ld aknight-ld commented Oct 22, 2026 •
edited by cursor Bot
Loading

Copy link
Copy Markdown
Contributor

Specifications for evaluations from code require that a judge that no handler covers fails the run before any record is created. Behavior is already consistent, but adding in a test to prove it and catch regressions.

Note

Overview
Adds regression tests for pre-run validation when judge or generation prompts don’t match handler mode (agent vs messages).

A new case covers an agent-mode judge whose resolved config has messages instead of instructions: evals.run must raise EvaluationsError, perform no HTTP, and not invoke generation or judge handlers. The existing agent-generation mismatch test is parametrized so both directions are checked (agent handler + messages, messages handler + instructions), with exact error text that includes the handler’s provides_for tuple.

Reviewed by Cursor Bugbot for commit 1c74580. Bugbot is set up for automated code reviews on this repo. Configure here.

@aknight-ld
aknight-ld merged commit fa7162e into main Oct 22, 2026
8 checks passed
@aknight-ld
aknight-ld deleted the AIC-3642-match-spec-harness branch October 22, 2026 19:27
@github-actions github-actions Bot mentioned this pull request Oct 22, 2026
Merged
jeffdupont pushed a commit that referenced this pull request Oct 23, 2026
@github-actions
chore: release main (#166)
662d3f2 
🤖 I have created a release *beep* *boop*
---


<details><summary>launchdarkly-ai-server: 0.4.0</summary>

##
[0.4.0](launchdarkly-ai-server-0.3.0...launchdarkly-ai-server-0.4.0)
(2026-10-23)


### ⚠ BREAKING CHANGES

* **evaluations:** keep the management API client internal
([#202](#202))
* **evaluations:** rename base uri to match spec
([#199](#199))
* **evaluations:** add test proving harness edge case matches spec
([#200](#200))
* **evaluations:** route handlers by provider and mode
* **evaluations:** mark evaluations from code as experimental
([#157](#157))

### Experimental

* **evaluations:** add test proving harness edge case matches spec
([#200](#200))
([fa7162e](fa7162e))
* **evaluations:** keep the management API client internal
([#202](#202))
([17270ed](17270ed))
* **evaluations:** mark evaluations from code as experimental
([#157](#157))
([1389765](1389765))
* **evaluations:** rename base uri to match spec
([#199](#199))
([7d93f9c](7d93f9c))
* **evaluations:** route handlers by provider and mode
([f62591f](f62591f))
* **evaluations:** Set JSON as standard for output to pass to judge
([#203](#203))
([ffc728b](ffc728b))
* **skills:** Correct the HTTP 403 advice now that server-side FDv2 is
available to every account
([#178](#178))
([b17e322](b17e322))


### Features

* **agui:** add a shared reader for handler streams
([#164](#164))
([fc08dd5](fc08dd5))
* **AIC-3534:** add a TypeSafe Jev judge handler
([#149](#149))
([1b41ea4](1b41ea4))
* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))
* **langchain:** emit AG-UI text and tool events
([#163](#163))
([ab5e9e1](ab5e9e1))
* **openai:** emit AG-UI text and tool events
([#161](#161))
([2b1b2d4](2b1b2d4))


### Bug Fixes

* **AIC-3570:** re-resolve graph flags on every invoke and stream
([#187](#187))
([8fdc4d4](8fdc4d4))
* **AIC-3571:** visit every child before its parent in reverse_traverse
([#189](#189))
([87c73d9](87c73d9))
* **AIC-3572:** treat a variation without _ldMeta.enabled as disabled
([#184](#184))
([7a0d260](7a0d260))
* **AIC-3574:** build the LaunchDarkly client off the event loop
([#185](#185))
([bf03b6c](bf03b6c))
* **AIC-3576:** record deferred judge scores under the evaluated config
([#190](#190))
([354d3f1](354d3f1))
* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
* **AIC-3596:** name the [otel] extra in the missing-OTel warning
([#186](#186))
([90ce7bf](90ce7bf))
* **client:** match a handler's provides_for after normalizing its mode
([f62591f](f62591f))
* give native graph spans their LaunchDarkly identity
([#121](#121))
([e863b6d](e863b6d))
</details>

<details><summary>launchdarkly-ai-python: 0.3.0</summary>

##
[0.3.0](launchdarkly-ai-python-0.2.0...launchdarkly-ai-python-0.3.0)
(2026-10-23)


### ⚠ BREAKING CHANGES

* **evaluations:** mark evaluations from code as experimental
([#157](#157))

### Experimental

* **evaluations:** mark evaluations from code as experimental
([#157](#157))
([1389765](1389765))
</details>

<details><summary>launchdarkly-ai-claude-agents: 0.2.6</summary>

##
[0.2.6](launchdarkly-ai-claude-agents-0.2.5...launchdarkly-ai-claude-agents-0.2.6)
(2026-10-23)


### Features

* **claude:** emit AG-UI text and tool events
([#165](#165))
([3f3a0ce](3f3a0ce))
* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))


### Bug Fixes

* **AIC-3571:** visit every child before its parent in reverse_traverse
([#189](#189))
([87c73d9](87c73d9))
* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
* **claude-agents:** fail the run when the CLI reports an API error as
success
([#174](#174))
([3d1a768](3d1a768))
* give native graph spans their LaunchDarkly identity
([#121](#121))
([e863b6d](e863b6d))
</details>

<details><summary>launchdarkly-ai-claude-messages: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-claude-messages-0.2.4...launchdarkly-ai-claude-messages-0.2.5)
(2026-10-23)


### Features

* **claude:** emit AG-UI text and tool events
([#165](#165))
([3f3a0ce](3f3a0ce))
* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))


### Bug Fixes

* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
</details>

<details><summary>launchdarkly-ai-openai-agents: 0.2.6</summary>

##
[0.2.6](launchdarkly-ai-openai-agents-0.2.5...launchdarkly-ai-openai-agents-0.2.6)
(2026-10-23)


### Features

* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))
* **openai:** emit AG-UI text and tool events
([#161](#161))
([2b1b2d4](2b1b2d4))


### Bug Fixes

* **AIC-3571:** visit every child before its parent in reverse_traverse
([#189](#189))
([87c73d9](87c73d9))
* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
* give native graph spans their LaunchDarkly identity
([#121](#121))
([e863b6d](e863b6d))
</details>

<details><summary>launchdarkly-ai-vercel-agents: 0.1.1</summary>

##
[0.1.1](launchdarkly-ai-vercel-agents-0.1.0...launchdarkly-ai-vercel-agents-0.1.1)
(2026-10-23)


### Features

* **AIC-3408:** add Vercel AI SDK messages and agents adapters
([#103](#103))
([64d679e](64d679e))
</details>

<details><summary>launchdarkly-ai-litellm-agents: 0.1.1</summary>

##
[0.1.1](launchdarkly-ai-litellm-agents-0.1.0...launchdarkly-ai-litellm-agents-0.1.1)
(2026-10-23)


### Features

* add LiteLLM messages and agents integrations
([#102](#102))
([f2e9886](f2e9886))
</details>

<details><summary>launchdarkly-ai-openai-messages: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-openai-messages-0.2.4...launchdarkly-ai-openai-messages-0.2.5)
(2026-10-23)


### Features

* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))
* **openai:** emit AG-UI text and tool events
([#161](#161))
([2b1b2d4](2b1b2d4))


### Bug Fixes

* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
</details>

<details><summary>launchdarkly-ai-vercel-messages: 0.1.1</summary>

##
[0.1.1](launchdarkly-ai-vercel-messages-0.1.0...launchdarkly-ai-vercel-messages-0.1.1)
(2026-10-23)


### Features

* **AIC-3408:** add Vercel AI SDK messages and agents adapters
([#103](#103))
([64d679e](64d679e))
</details>

<details><summary>launchdarkly-ai-litellm-messages: 0.1.1</summary>

##
[0.1.1](launchdarkly-ai-litellm-messages-0.1.0...launchdarkly-ai-litellm-messages-0.1.1)
(2026-10-23)


### Features

* add LiteLLM messages and agents integrations
([#102](#102))
([f2e9886](f2e9886))
</details>

<details><summary>launchdarkly-ai-langchain-agents: 0.2.6</summary>

##
[0.2.6](launchdarkly-ai-langchain-agents-0.2.5...launchdarkly-ai-langchain-agents-0.2.6)
(2026-10-23)


### Features

* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))
* **langchain:** emit AG-UI text and tool events
([#163](#163))
([ab5e9e1](ab5e9e1))


### Bug Fixes

* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
* give native graph spans their LaunchDarkly identity
([#121](#121))
([e863b6d](e863b6d))
</details>

<details><summary>launchdarkly-ai-langchain-messages: 0.2.5</summary>

##
[0.2.5](launchdarkly-ai-langchain-messages-0.2.4...launchdarkly-ai-langchain-messages-0.2.5)
(2026-10-23)


### Features

* **client:** add configure for SDK defaults
([#158](#158))
([3736767](3736767))
* **langchain:** emit AG-UI text and tool events
([#163](#163))
([ab5e9e1](ab5e9e1))


### Bug Fixes

* **AIC-3577:** send object tool results to the model as JSON
([#188](#188))
([dead4bd](dead4bd))
</details>

<details><summary>launchdarkly-ai-opencode-agents: 0.2.0</summary>

##
[0.2.0](launchdarkly-ai-opencode-agents-0.1.0...launchdarkly-ai-opencode-agents-0.2.0)
(2026-10-23)


### ⚠ BREAKING CHANGES

* **opencode:** start and stop opencode serve in the handler
([#198](#198))

### Features

* **opencode-agents:** add an OpenCode handler
([#159](#159))
([fc08e8c](fc08e8c))
* **opencode-agents:** emit the AG-UI text and tool subset
([#162](#162))
([7677621](7677621))
* **opencode-agents:** open one OpenCode session per call
([#179](#179))
([b225113](b225113))
* **opencode-agents:** render agent instruction templates
([#182](#182))
([83175aa](83175aa))
* **opencode:** start and stop opencode serve in the handler
([#198](#198))
([0748dcf](0748dcf))


### Bug Fixes

* **opencode-agents:** own the HTTP client
([#194](#194))
([ed2d419](ed2d419))
* **opencode-agents:** skip judges and record structured history
([#177](#177))
([6fadf1d](6fadf1d))
</details>

<details><summary>launchdarkly-ai-typesafe: 0.1.1</summary>

##
[0.1.1](launchdarkly-ai-typesafe-0.1.0...launchdarkly-ai-typesafe-0.1.1)
(2026-10-23)


### Features

* **AIC-3534:** add a TypeSafe Jev judge handler
([#149](#149))
([1b41ea4](1b41ea4))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@donei003 donei003 donei003 approved these changes
@jeffdupont jeffdupont Awaiting requested review from jeffdupont

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@aknight-ld @donei003