Repository navigation
feat(evaluations)!: rename base uri to match spec - #199
Conversation
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 89a0063. Configure here.
jeffdupont
left a comment
There was a problem hiding this comment.
Thanks, the rename and _resolve_host look right. They match spec §8.2: a blank option now falls through to LD_API_BASE_URI, _env strips whitespace, both hosts drop a trailing /, and the name lines up with JS apiBaseUri. One thing blocks merge, and one follow-up is on the spec side.
1. Must fix: the new UI-host test fails before it reaches its assertions (Tests is red on 89a0063, the only failure of 2,863):
FAILED tests/test_evaluations_run.py::test_api_host_env_alone_leaves_result_url_on_default_ui_host
EvaluationsError: handler does not declare provides_for.
run() rejects the plain async def handler and the generation dict with no mode before any request goes out. So the spec rule this test exists for (§8.2: the UI host is not derived from LD_API_BASE_URI) is still untested. The fix is two lines, matching the other run() tests in this file: handler=tagged(handler) and "mode": "completion" in generation. Bugbot flagged the same thing.
2. Should fix (separate monorepo PR): remove the "Python pending" notes from the spec. TESTING.md on main still has three [host overrides: Python pending] tags (§8.2 bullets 1 and 4, and the cross-SDK parity paragraph). The "What Python still lacks" bullet describes exactly what this PR fixes. Once this merges, those can go, so the spec doesn't claim a gap that no longer exists.
🤖 I have created a release *beep* *boop* --- <details><summary>launchdarkly-ai-server: 0.4.0</summary> ## [0.4.0](launchdarkly-ai-server-0.3.0...launchdarkly-ai-server-0.4.0) (2026-10-23) ### ⚠ BREAKING CHANGES * **evaluations:** keep the management API client internal ([#202](#202)) * **evaluations:** rename base uri to match spec ([#199](#199)) * **evaluations:** add test proving harness edge case matches spec ([#200](#200)) * **evaluations:** route handlers by provider and mode * **evaluations:** mark evaluations from code as experimental ([#157](#157)) ### Experimental * **evaluations:** add test proving harness edge case matches spec ([#200](#200)) ([fa7162e](fa7162e)) * **evaluations:** keep the management API client internal ([#202](#202)) ([17270ed](17270ed)) * **evaluations:** mark evaluations from code as experimental ([#157](#157)) ([1389765](1389765)) * **evaluations:** rename base uri to match spec ([#199](#199)) ([7d93f9c](7d93f9c)) * **evaluations:** route handlers by provider and mode ([f62591f](f62591f)) * **evaluations:** Set JSON as standard for output to pass to judge ([#203](#203)) ([ffc728b](ffc728b)) * **skills:** Correct the HTTP 403 advice now that server-side FDv2 is available to every account ([#178](#178)) ([b17e322](b17e322)) ### Features * **agui:** add a shared reader for handler streams ([#164](#164)) ([fc08dd5](fc08dd5)) * **AIC-3534:** add a TypeSafe Jev judge handler ([#149](#149)) ([1b41ea4](1b41ea4)) * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) * **langchain:** emit AG-UI text and tool events ([#163](#163)) ([ab5e9e1](ab5e9e1)) * **openai:** emit AG-UI text and tool events ([#161](#161)) ([2b1b2d4](2b1b2d4)) ### Bug Fixes * **AIC-3570:** re-resolve graph flags on every invoke and stream ([#187](#187)) ([8fdc4d4](8fdc4d4)) * **AIC-3571:** visit every child before its parent in reverse_traverse ([#189](#189)) ([87c73d9](87c73d9)) * **AIC-3572:** treat a variation without _ldMeta.enabled as disabled ([#184](#184)) ([7a0d260](7a0d260)) * **AIC-3574:** build the LaunchDarkly client off the event loop ([#185](#185)) ([bf03b6c](bf03b6c)) * **AIC-3576:** record deferred judge scores under the evaluated config ([#190](#190)) ([354d3f1](354d3f1)) * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) * **AIC-3596:** name the [otel] extra in the missing-OTel warning ([#186](#186)) ([90ce7bf](90ce7bf)) * **client:** match a handler's provides_for after normalizing its mode ([f62591f](f62591f)) * give native graph spans their LaunchDarkly identity ([#121](#121)) ([e863b6d](e863b6d)) </details> <details><summary>launchdarkly-ai-python: 0.3.0</summary> ## [0.3.0](launchdarkly-ai-python-0.2.0...launchdarkly-ai-python-0.3.0) (2026-10-23) ### ⚠ BREAKING CHANGES * **evaluations:** mark evaluations from code as experimental ([#157](#157)) ### Experimental * **evaluations:** mark evaluations from code as experimental ([#157](#157)) ([1389765](1389765)) </details> <details><summary>launchdarkly-ai-claude-agents: 0.2.6</summary> ## [0.2.6](launchdarkly-ai-claude-agents-0.2.5...launchdarkly-ai-claude-agents-0.2.6) (2026-10-23) ### Features * **claude:** emit AG-UI text and tool events ([#165](#165)) ([3f3a0ce](3f3a0ce)) * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) ### Bug Fixes * **AIC-3571:** visit every child before its parent in reverse_traverse ([#189](#189)) ([87c73d9](87c73d9)) * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) * **claude-agents:** fail the run when the CLI reports an API error as success ([#174](#174)) ([3d1a768](3d1a768)) * give native graph spans their LaunchDarkly identity ([#121](#121)) ([e863b6d](e863b6d)) </details> <details><summary>launchdarkly-ai-claude-messages: 0.2.5</summary> ## [0.2.5](launchdarkly-ai-claude-messages-0.2.4...launchdarkly-ai-claude-messages-0.2.5) (2026-10-23) ### Features * **claude:** emit AG-UI text and tool events ([#165](#165)) ([3f3a0ce](3f3a0ce)) * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) ### Bug Fixes * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) </details> <details><summary>launchdarkly-ai-openai-agents: 0.2.6</summary> ## [0.2.6](launchdarkly-ai-openai-agents-0.2.5...launchdarkly-ai-openai-agents-0.2.6) (2026-10-23) ### Features * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) * **openai:** emit AG-UI text and tool events ([#161](#161)) ([2b1b2d4](2b1b2d4)) ### Bug Fixes * **AIC-3571:** visit every child before its parent in reverse_traverse ([#189](#189)) ([87c73d9](87c73d9)) * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) * give native graph spans their LaunchDarkly identity ([#121](#121)) ([e863b6d](e863b6d)) </details> <details><summary>launchdarkly-ai-vercel-agents: 0.1.1</summary> ## [0.1.1](launchdarkly-ai-vercel-agents-0.1.0...launchdarkly-ai-vercel-agents-0.1.1) (2026-10-23) ### Features * **AIC-3408:** add Vercel AI SDK messages and agents adapters ([#103](#103)) ([64d679e](64d679e)) </details> <details><summary>launchdarkly-ai-litellm-agents: 0.1.1</summary> ## [0.1.1](launchdarkly-ai-litellm-agents-0.1.0...launchdarkly-ai-litellm-agents-0.1.1) (2026-10-23) ### Features * add LiteLLM messages and agents integrations ([#102](#102)) ([f2e9886](f2e9886)) </details> <details><summary>launchdarkly-ai-openai-messages: 0.2.5</summary> ## [0.2.5](launchdarkly-ai-openai-messages-0.2.4...launchdarkly-ai-openai-messages-0.2.5) (2026-10-23) ### Features * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) * **openai:** emit AG-UI text and tool events ([#161](#161)) ([2b1b2d4](2b1b2d4)) ### Bug Fixes * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) </details> <details><summary>launchdarkly-ai-vercel-messages: 0.1.1</summary> ## [0.1.1](launchdarkly-ai-vercel-messages-0.1.0...launchdarkly-ai-vercel-messages-0.1.1) (2026-10-23) ### Features * **AIC-3408:** add Vercel AI SDK messages and agents adapters ([#103](#103)) ([64d679e](64d679e)) </details> <details><summary>launchdarkly-ai-litellm-messages: 0.1.1</summary> ## [0.1.1](launchdarkly-ai-litellm-messages-0.1.0...launchdarkly-ai-litellm-messages-0.1.1) (2026-10-23) ### Features * add LiteLLM messages and agents integrations ([#102](#102)) ([f2e9886](f2e9886)) </details> <details><summary>launchdarkly-ai-langchain-agents: 0.2.6</summary> ## [0.2.6](launchdarkly-ai-langchain-agents-0.2.5...launchdarkly-ai-langchain-agents-0.2.6) (2026-10-23) ### Features * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) * **langchain:** emit AG-UI text and tool events ([#163](#163)) ([ab5e9e1](ab5e9e1)) ### Bug Fixes * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) * give native graph spans their LaunchDarkly identity ([#121](#121)) ([e863b6d](e863b6d)) </details> <details><summary>launchdarkly-ai-langchain-messages: 0.2.5</summary> ## [0.2.5](launchdarkly-ai-langchain-messages-0.2.4...launchdarkly-ai-langchain-messages-0.2.5) (2026-10-23) ### Features * **client:** add configure for SDK defaults ([#158](#158)) ([3736767](3736767)) * **langchain:** emit AG-UI text and tool events ([#163](#163)) ([ab5e9e1](ab5e9e1)) ### Bug Fixes * **AIC-3577:** send object tool results to the model as JSON ([#188](#188)) ([dead4bd](dead4bd)) </details> <details><summary>launchdarkly-ai-opencode-agents: 0.2.0</summary> ## [0.2.0](launchdarkly-ai-opencode-agents-0.1.0...launchdarkly-ai-opencode-agents-0.2.0) (2026-10-23) ### ⚠ BREAKING CHANGES * **opencode:** start and stop opencode serve in the handler ([#198](#198)) ### Features * **opencode-agents:** add an OpenCode handler ([#159](#159)) ([fc08e8c](fc08e8c)) * **opencode-agents:** emit the AG-UI text and tool subset ([#162](#162)) ([7677621](7677621)) * **opencode-agents:** open one OpenCode session per call ([#179](#179)) ([b225113](b225113)) * **opencode-agents:** render agent instruction templates ([#182](#182)) ([83175aa](83175aa)) * **opencode:** start and stop opencode serve in the handler ([#198](#198)) ([0748dcf](0748dcf)) ### Bug Fixes * **opencode-agents:** own the HTTP client ([#194](#194)) ([ed2d419](ed2d419)) * **opencode-agents:** skip judges and record structured history ([#177](#177)) ([6fadf1d](6fadf1d)) </details> <details><summary>launchdarkly-ai-typesafe: 0.1.1</summary> ## [0.1.1](launchdarkly-ai-typesafe-0.1.0...launchdarkly-ai-typesafe-0.1.1) (2026-10-23) ### Features * **AIC-3534:** add a TypeSafe Jev judge handler ([#149](#149)) ([1b41ea4](1b41ea4)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

JS SDK build cannot use the same variable names as what we have in python, so we have updated our spec to indicate target variable names. Updates tests accordingly to prove consistency with current standard.
This will be a breaking change, which is an acceptable risk for experimental features.
Note
Overview
Breaking change for experimental evaluations:
init_evaluations()renamesbase_uritoapi_base_uriso Python matches the cross-SDK spec (callers passingbase_uriget aTypeError).Host resolution is centralized in
_resolve_host: option → env → default, with blank/whitespace treated as unset and trailing slashes stripped. Management API traffic usesapi_base_uri/LD_API_BASE_URI; run links still useui_base_uri/LD_UI_BASE_URIonly—LD_BASE_URIis never used for evals API calls (relay/SDK delivery). Docs in README andagents.mdspell out that split.Tests replace the old override checks with coverage of precedence, blanks, slash normalization, SDK-delivery isolation, and an end-to-end run asserting staging API URLs do not change the default UI result link.
Reviewed by Cursor Bugbot for commit 941c216. Bugbot is set up for automated code reviews on this repo. Configure here.