feat: add the experiment-orchestrator custom agent to aws-devops-agent setup by cchu-ld · Pull Request #972 · launchdarkly/ldcli

cchu-ld · 2026-10-26T22:25:00Z

Human

Updates ldcli to set up custom agent in AWS DevOps agent space from ai-tooling repo
Update SKILL.md to be imported from ai-tooling repo
Update allowlist of MCP tooling - flagged from changes to SKILL.md referencing them.

Summary

ldcli aws-devops-agent setup creates the experiment-orchestrator custom agent after the MCP server and the skill.
- tools: the allowlisted LaunchDarkly tools, as <MCP server name>_<tool name>, for example LaunchDarkly-OAuth_get-flag. Setup reads the server name from AWS, so a renamed server works.
- skills: the asset ID of the experiment-orchestration skill.
- Prompt: one paragraph that tells the agent to load the skill and follow it.
On an existing agent:
- No agent: setup creates it, also in a space that already has the skill.
- An agent with the latest prompt: setup adds the missing ldcli tools and the skill, and keeps what the customer added.
- A prompt that differs from the latest: setup skips the agent. The plan says to delete the agent in the AWS console and run setup again.
Setting up the custom agent is optional. It needs the MCP server and the skill. If either is missing or the step fails, setup warns and does the other steps. The step needs aidevops:ListAssets, aidevops:GetAssetFile, aidevops:CreateAsset, aidevops:UpdateAsset, and aidevops:ListAssociations.
The MCP allowlist covers the whole experiment flow, not only the tools that the skill names:
- Read-only: list-metric-events, get-flag-health, get-flag-unique-context-exposures, search-contexts.
- Mutative: update-off-variation, update-experiment, save-and-start-experiment-iteration, create-automated-rollout-config.
Setup adds the missing ldcli tools to the allowlist of an existing space, so spaces from older ldcli versions get the new tools. The rewrite gives every tool the ldcli classification. --add-mcp-tools still parses, has no effect, and is deprecated.
Setup downloads SKILL.md and the custom agent prompt from the main branch of launchdarkly/ai-tooling when it plans, and writes the texts that the plan compared. ldcli no longer embeds them, so a change in ai-tooling reaches customers without an ldcli release.
If the space already has the skill and it differs from the latest SKILL.md, the plan says so and setup skips it. --replace-skill updates it to the latest version.

Testing

go test ./... passes.
Tests use a fake source and make no network calls. A test of the GitHub client runs against a local server.
Both files download from raw.githubusercontent.com without authentication.
Setup has not been run against AWS with this branch. It is not yet tested whether tools also limits the code and deployment tools that the skill needs.

Requirements

I have added test coverage for new or changed functionality
I have followed the repository's pull request submission guidelines
I have validated my changes against all supported platform versions

Related issues

Follows #970, which added the experiment-orchestration skill. launchdarkly/ai-tooling#230 added the skill and the prompt to ai-tooling.

Describe alternatives you've considered

A --replace-agent flag, the same as --replace-skill. It is left out, because the prompt is short and the procedure is in the skill.
Embedding the files at a pinned ai-tooling commit, with a sync script and a CI drift check. That ties each release to a skill that matches its allowlist, but every skill change needs an ldcli release. Reading from main was chosen instead.

Additional context

Status does not show the skill or the agent.

🤖 Generated with Claude Code

Note

Overview
aws-devops-agent setup now provisions an experiment-orchestrator custom agent after the MCP server and skill. The plan and JSON result include a Custom agent step with AgentError warnings when the step fails without aborting setup.

Skill and agent content no longer ship inside ldcli. At plan time, setup downloads SKILL.md and the agent prompt from main on launchdarkly/ai-tooling via a new Sources client (GitHub raw URLs). The run writes the same bytes the plan compared. Download failure skips both optional steps with warnings. Existing skills/agents that differ from upstream are skipped unless --replace-skill; customized agent prompts require deleting the agent in the console and re-running setup.

MCP allowlists on reused servers automatically gain missing ldcli-classified tools (no --add-mcp-tools, now deprecated). The built-in tool lists grow (e.g. list-metric-events, update-experiment, guarded-rollout helpers) to match the updated experiment flow.

^{Reviewed by Cursor Bugbot for commit 03c96a9. Bugbot is set up for automated code reviews on this repo. Configure here.}

Setup now adds the experiment-orchestration skill to the agent space. The skill tells the agent how to move a goal through an experiment, a guarded rollout, and a report. The skill text comes from the setup guide. If the space has a skill with the same name and other instructions, setup does not change it. Pass --replace-skill to replace the instructions. The status and agent types of the skill stay the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Setup now does the skill step after the MCP server step. If the skill step fails, setup shows a warning and puts the error in skillError. The other steps still run. If setup cannot read the skills of the space, the plan skips the skill. Setup keeps the SHA-256 of each SKILL.md that it sent. If a space has one of these texts, setup updates it without --replace-skill. Other text is a customer edit. The plan keeps the version of the skill asset. If the version changes before the write, setup does not change the skill. The plan line and the help text say that every agent type in the space can load the skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The skill warning now keeps the full AWS error, so it names the denied action. If AWS denies access, the warning also names the permissions that the skill step needs. The warning shows one time, at the end of the run. An empty SKILL.md now gets an update without --replace-skill. A skill with no SKILL.md gets its own plan line. A SKILL.md that AWS sends as bytes is compared as text. The setup result now has skillAction, so a JSON run shows if setup kept, updated, or left the skill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ON runs If the skill changes between the plan and the write, skillAction is now skipped, because setup tried no write. The warning for a denied GetAssetFile now names the skill one time. The plan line for a skipped skill starts with "Skip skill". A JSON run now also writes the skill warning to stderr. If someone deletes the skill after the list, the plan now creates a new skill. Before, the plan said that the skill had no SKILL.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… a result A scheduled run of the agent has no memory of the run that started the experiment. The skill now keeps an experiment record, tags the flags that it owns, and resumes in-flight work before it starts a new goal. The experiment now runs at least 2 weeks before a decision, not 30 minutes. A result that is not a win stops the iteration with control as the winner and turns the flag off. Guarded rollout stages stay at or below 50%, as LaunchDarkly requires. If no coding agent is available, the skill waits for a person to confirm the deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

In AWS DevOps Agent, every LaunchDarkly write needs a person to approve it in chat, and no coding agent or CI/CD is connected. The skill now runs as two chats: one that starts an experiment and one that checks on experiments. A person implements and deploys the change, and the agent stops at each wait. - The flag serves false when it is on, so turning it on before the experiment starts does not give everyone the treatment. - The experiment uses 10% of traffic through 5% and 5% treatments and notInExperimentVariationId, with sequential testing. - The agent shows the draft experiment and waits for a confirmation before it starts it, as the create-experiment tool asks. - The record in the flag description holds the project and environment. - The release review, merge, and deploy steps are gone. The allowlist adds update-flag-settings, for the record, and create-metric. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…eep the win out of the 100% path Stopping an experiment iteration serves the chosen winner to everyone that the rule matches. The skill now stops a won experiment with control as the winner, records the win first, makes sure that the default rule serves control, and then ramps the treatment with the guarded rollout. A space that reuses an MCP server kept its old allowlist, so the tools that the skill now uses were missing. Setup now names the missing tools in the plan and warns. With --add-mcp-tools it adds them. It rewrites the allowlist only if every tool on it is an ldcli tool, because AWS replaces the whole configuration and does not send back each tool's classification, and an unclassified tool runs without approval. The run skips the rewrite if the allowlist changed after the plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- The report template is in a fenced markdown block, so its headings stay out of the skill's outline. - The report compares control with treatment, not before with after. - "Design waiting for confirmation" is a record state, so a later chat resumes the draft experiment instead of making another one. - The skill says why the 2-week minimum holds with sequential testing on: sequential testing lets a guardrail regression end the experiment early, and the minimum covers weekly cycles and novelty effects. - The report recommends a flag cleanup for a winner too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… in setup Setup now creates a custom agent that runs the experiment-orchestration skill with the LaunchDarkly MCP tools that the space allows. The tool IDs are "<server name>_<tool name>", the format that AWS uses for a custom agent made in the console. An update adds the ldcli tools and skill to the agent and keeps the ones that the customer added. If someone edited the prompt, setup leaves the agent unless --replace-agent is passed. The agent is optional: if its step fails, or the MCP server or the skill is missing, setup warns and does the other steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

status now reports the experiment-orchestration skill and the experiment-orchestrator custom agent of each agent space: whether each is the ldcli version, an older one that setup updates, edited, without its file, or missing. For the agent, it also gives the number of tools and whether the skill is attached. An agent with the ldcli prompt but without the skill or an allowed tool is outdated, because setup updates it. Like the operator app, an asset that status cannot read does not stop it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The skill now runs in chat, where a person approves each write. Outside chat, for example in a scheduled run, every write fails. In chat, the agent follows the skill. Outside chat, it only reads: it reports, for each experiment in flight, what is due and the message to send in chat, and it never writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Since #968, setup registers a server for each agent space, named LaunchDarkly-<agent space ID>, and a replace can rename a server. The agent's tool IDs used the fixed name LaunchDarkly, so they would not match. The run now reads the name of the registered server, and the plan shows the name that the server will have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… in setup Setup now creates a custom agent that runs the experiment-orchestration skill with the LaunchDarkly MCP tools that the space allows. The tool IDs are "<server name>_<tool name>", the format that AWS uses for a custom agent made in the console. An update adds the ldcli tools and skill to the agent and keeps the ones that the customer added. If someone edited the prompt, setup leaves the agent unless --replace-agent is passed. The agent is optional: if its step fails, or the MCP server or the skill is missing, setup warns and does the other steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

status now reports the experiment-orchestration skill and the experiment-orchestrator custom agent of each agent space: whether each is the ldcli version, an older one that setup updates, edited, without its file, or missing. For the agent, it also gives the number of tools and whether the skill is attached. An agent with the ldcli prompt but without the skill or an allowed tool is outdated, because setup updates it. Like the operator app, an asset that status cannot read does not stop it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The skill now runs in chat, where a person approves each write. Outside chat, for example in a scheduled run, every write fails. In chat, the agent follows the skill. Outside chat, it only reads: it reports, for each experiment in flight, what is due and the message to send in chat, and it never writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Since #968, setup registers a server for each agent space, named LaunchDarkly-<agent space ID>, and a replace can rename a server. The agent's tool IDs used the fixed name LaunchDarkly, so they would not match. The run now reads the name of the registered server, and the plan shows the name that the server will have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The prompt now loads the experiment-orchestration skill from #970 and follows it as is. It adds only the LaunchDarkly MCP tools and one rule: never start a goal that a person did not ask for. The old prompt never shipped, so its hash is replaced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The tree is the rebased branch. This merge keeps the old commits, so the push needs no force. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…m-agent

The prompt now has one paragraph that loads the experiment-orchestration skill and follows it. The help text for the custom agent is shorter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Setup never overwrites an edited prompt of the custom agent. The plan and the log say that setup leaves the agent, and that a person can delete the agent in the AWS console and run setup again to get the ldcli version. Status no longer names a flag for an edited agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The comments that stay give a reason that the code does not show. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Setup always connects the MCP server before the agent step, so the check for a missing server is gone. planAgent no longer takes the options, and the unused AllowedTools test helper is gone. The message for a missing permission now names aidevops:ListAssociations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The MCP step knows the name of the server, so it keeps the name in the result. The agent step no longer lists the servers again, and the OAuth poll test is the same as on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cursor

Stale Bugbot comment from a previous run.

The plan compared the agent with the allowlist before the MCP step. With --add-mcp-tools, the MCP step adds tools first, so the plan kept the agent without them. The plan now expects those tools, so setup updates the agent in the same run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ari-launchdarkly

If this is predicated on being used by an agent with the MCP, shouldn't this be in ai-tooling?

The experiment-orchestration skill and the custom agent prompt now come from launchdarkly/ai-tooling. scripts/sync-aws-devops-agent.sh downloads them at the commit in AI_TOOLING_COMMIT. go generate and make aws-devops-agent-sync run it. ldcli embeds the files, so they stay committed. CI fails if they differ from that commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The sync now pins 887d4cb, the merge commit of launchdarkly/ai-tooling#230. SKILL.md changed, so its hash is in shippedSkillHashes. The agent prompt did not change. The skill test now compares the description with the front matter of the embedded SKILL.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cchu-ld · 2026-10-29T17:15:02Z

If this is predicated on being used by an agent with the MCP, shouldn't this be in ai-tooling?

@ari-launchdarkly that's true - it's been added to ai-tooling now.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1c7eeb4. Configure here.}

"The agent is optional" can mean that a space does not need the agent. The help text now says that the setup step is optional. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…to existing spaces The allowlist now has the tools that the new SKILL.md names and that were missing: list-metric-events and update-off-variation. It also has tools for the rest of that work that the skill does not name: update-experiment, save-and-start-experiment-iteration, create-automated-rollout-config, get-flag-health, get-flag-unique-context-exposures, and search-contexts. Setup now adds the missing ldcli tools to the allowlist of an existing space without --add-mcp-tools, so spaces from older ldcli versions get the new tools. The flag still parses, has no effect, and is deprecated. A new test fails when SKILL.md names a tool that the allowlist does not have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…main Setup no longer embeds SKILL.md and the custom agent prompt. The plan downloads both from the main branch of launchdarkly/ai-tooling, and the run writes the texts that the plan compared. If the download fails, setup skips the skill and custom agent steps and warns. A space text counts as ours when it is empty, matches a frozen hash of a released ldcli, or equals an earlier version of the file on ai-tooling main. Setup reads that history only when nothing else matches, and skips the step if it cannot read it. The plan and the result warn when the skill names MCP tools that this ldcli version does not allow. The sync and check scripts, their Makefile targets, and the CI step are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A future GitHub Action will compare the skill on ai-tooling main with the allowlist and open a pull request when they differ, so setup no longer warns about tools that the skill names. Setup also no longer reads the history of the files on ai-tooling main. A space whose text is the current main text or a text that a released ldcli embedded gets the new text. Other text counts as an edit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Setup no longer keeps the hashes of the texts that older ldcli versions embedded. If the skill in the space is not the latest SKILL.md on ai-tooling main, the plan says so and setup skips it. --replace-skill updates it to the latest version. A custom agent whose prompt differs is skipped the same way, with the hint to delete it and run setup again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Sources.File drops the ref, which is always main, and GitHubSources drops the nil-client fallback. planSkill is one function again, and skillText is inlined. Comments that repeat the code, or that describe the embedded files and the hash lists, are gone. The allowlist comment no longer says that the skill keeps its record in the flag description. Tests named for edits are named for a different text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Comments whose code did not change get their wording from main back, and skillText and two test names return. Comments now change only where the behavior did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cchu-ld force-pushed the cchu/experiment-custom-agent branch 3 times, most recently from 48e4076 to 7fc4b98 Compare October 26, 2026 23:30

cchu-ld added this pull request to stack #973 October 26, 2026 23:32

cchu-ld force-pushed the cchu/experiment-custom-agent branch from 7fc4b98 to 2c06a4c Compare October 27, 2026 16:52

cchu-ld and others added 8 commits October 27, 2026 10:32

cchu-ld and others added 4 commits October 27, 2026 10:36

cchu-ld force-pushed the cchu/experiment-custom-agent branch from 2c06a4c to 642f7e3 Compare October 27, 2026 17:39

Base automatically changed from cchu/experiment-skill to main October 28, 2026 18:27

cchu-ld and others added 9 commits October 28, 2026 14:13

Merge the old branch history after the rebase onto main

ce5eaf8

The tree is the rebased branch. This merge keeps the old commits, so the push needs no force. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into cchu/experiment-custo…

aa3f5ea

…m-agent

fix: shorten the custom agent prompt to one paragraph

5fe9961

The prompt now has one paragraph that loads the experiment-orchestration skill and follows it. The help text for the custom agent is shorter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cchu-ld requested a review from a team as a code owner October 28, 2026 21:44

cchu-ld requested review from davidbrackbill and removed request for a team October 28, 2026 21:44

cchu-ld and others added 3 commits October 28, 2026 14:47

chore: remove comments that repeat the code

549cb81

The comments that stay give a reason that the code does not show. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cchu-ld requested a review from a team October 28, 2026 21:56

cursor Bot reviewed Oct 28, 2026

View reviewed changes

Comment thread internal/awsdevops/agent.go Resolved

ari-launchdarkly reviewed Oct 28, 2026

View reviewed changes

Comment thread cmd/aws_devops_agent/setup.go Resolved

Comment thread internal/awsdevops/agents/experiment-orchestrator.md Resolved

cchu-ld mentioned this pull request Oct 28, 2026

feat(aws-devops-agent): add the experiment-orchestration skill and agent prompt launchdarkly/ai-tooling#230

Merged

cchu-ld and others added 2 commits October 28, 2026 15:17

cursor Bot reviewed Oct 29, 2026

View reviewed changes

Comment thread internal/awsdevops/skills/experiment-orchestration/SKILL.md Resolved

cchu-ld and others added 2 commits October 29, 2026 10:16

docs: say that setting up the skill and the custom agent is optional

07ed010

"The agent is optional" can mean that a space does not need the agent. The help text now says that the setup step is optional. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

cchu-ld changed the title ~~feat: add the experiment-orchestrator custom agent to the agent space in setup~~ feat: sync experiment-orchestrator custom agent and SKILL.md from ai-tooling Oct 29, 2026

cchu-ld changed the title ~~feat: sync experiment-orchestrator custom agent and SKILL.md from ai-tooling~~ feat: add the experiment-orchestrator custom agent to aws-devops-agent setup Oct 29, 2026

cchu-ld requested a review from ari-launchdarkly October 29, 2026 17:42

cchu-ld and others added 6 commits October 29, 2026 11:18

docs: clarify why setup downloads the skill and prompt once

e8db6e7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chore: restore comments that the earlier cleanup only rephrased

03c96a9

Comments whose code did not change get their wording from main back, and skillText and two test names return. Comments now change only where the behavior did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ari-launchdarkly approved these changes Oct 29, 2026

View reviewed changes

cchu-ld merged commit 66a30c6 into main Oct 29, 2026
10 checks passed

cchu-ld deleted the cchu/experiment-custom-agent branch October 29, 2026 19:45

github-actions Bot mentioned this pull request Oct 29, 2026

chore(main): release 3.16.0 #984

Open

Navigation Menu

Repository navigation

feat: add the experiment-orchestrator custom agent to aws-devops-agent setup - #972

feat: add the experiment-orchestrator custom agent to aws-devops-agent setup#972
cchu-ld merged 36 commits into
mainfrom
cchu/experiment-custom-agent

cchu-ld commented Oct 26, 2026 •

edited by cursor Bot

Loading

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

ari-launchdarkly left a comment

Uh oh!

Uh oh!

Uh oh!

cchu-ld commented Oct 29, 2026 •

edited

Loading

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cchu-ld commented Oct 26, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Human

Summary

Testing

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ari-launchdarkly left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cchu-ld commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cchu-ld commented Oct 26, 2026 •

edited by cursor Bot

Loading

cursor Bot left a comment •

edited

Loading

cchu-ld commented Oct 29, 2026 •

edited

Loading