[cchu-ld]

Requirements

• I have added test coverage for new or changed functionality
• I have followed the repository's pull request submission guidelines
• I have validated my changes against all supported platform versions

Describe the solution you've provided

ldcli aws-devops-agent setup now adds the experiment-orchestration skill to the agent space. A skill is a set of instructions that the agent loads for a related task. This skill tells the agent how to move a goal through an experiment, a guarded rollout, and a report.

Details
The skill text is in internal/awsdevops/skills/experiment-orchestration/SKILL.md, and the binary embeds it. The text is the skill from the setup guide, with a small number of fixes. The skill runs in one long chat, as an orchestrator that moves a goal through Steps 1 to 8. These are the fixes:

• The flag serves false when it is on (defaults of 1 and 1). In the guide, Step 6 turns the flag on before the experiment starts, so every user got the treatment.
• The flag is available to every SDK (usingEnvironmentId and usingMobileKey). Without this, a browser or mobile app cannot read the flag, and the experiment collects no data.
• The flag key follows the naming conventions of the project.
• Each stop of an iteration uses control as the winner. Stopping serves the winner to everyone that the rule matches, so a treatment winner reached 100% of users and skipped the guarded rollout. A loss or an inconclusive result now stops the iteration too, so 5% of users do not keep the treatment.
• The experiment is inconclusive after 4 weeks with no significance. The guide used 30 minutes.
• A loss or a rollback turns the flag off. The skill does not archive a flag that the code still evaluates.
• The report template is in a code fence, so its headings do not mix with the headings of the skill.
• If the environment needs approval for a change, such as a flag toggle or an experiment start, the agent creates an approval request. After a reviewer approves it, the agent applies it with apply-approval-request.

The allowlist also adds update-flag-settings, create-metric, and apply-approval-request. With create-metric, the agent can add a missing metric on an event that the app already sends. With apply-approval-request, the agent can apply an approved request. The skill no longer uses update-flag-settings.

A space that reuses an MCP server keeps its allowlist, so it can miss tools that this version allows, such as these tools. Setup names the missing tools in the plan and warns. With the new --add-mcp-tools flag, it adds them with UpdateAssociation. Setup cannot tell an older ldcli allowlist from one that someone trimmed, so the update needs the flag. AWS replaces the whole configuration on an update and does not send back each tool's classification or the capabilities, so:

• Setup rewrites the allowlist only if every tool on it is an ldcli tool. Otherwise it warns, because an unclassified tool runs as read-only, without approval.
• The rewrite gives every tool the ldcli classification. A customer who made an ldcli read-only tool mutative loses that, and the flag help says so.
• The run skips the rewrite if the allowlist changed after the plan.

Setup reads the name and the description from the front matter of the file.

Setup adds the skill in a full setup and in a run that only connects the MCP server. The plan has a new Skill: line. Setup does one of these actions:

• If the space has no skill with the name, setup creates the skill with agent_types set to ["GENERIC"]. With this value, every agent type in the space can load the skill. The agent loads a skill when its description matches the task. The plan line and the help text say this.
• If the space has the skill with the same SKILL.md text, setup keeps it.
• If the text matches a version that ldcli sent before, or the file is empty, setup updates it. shippedSkillHashes holds the SHA-256 of each shipped SKILL.md. A test fails if the current text is not in the list.
• If the text matches no shipped version, someone edited it, and setup does not change it. A skill with no SKILL.md gets the same action with its own plan line. The guide tells users to edit the skill if they do not have Guarded Releases.
• If someone edited the text and you pass the new --replace-skill flag, setup overwrites the text and the description. AWS changes only the metadata keys in the request, so the status and the agent types stay the same.

The skill is optional. Setup does the skill step after the MCP server step. If the skill step fails, setup shows one warning at the end, puts the error in skillError, and ends with no error. The warning keeps the full AWS error, so it names the denied action. If AWS denies access, the warning also names the permissions that the step needs. If setup cannot read the skills of the space, the plan skips the skill.

The result has skillAction with one of these values: created, kept, updated, left, overwritten, skipped, or failed. A JSON run can use it to find a skill that is not the ldcli version.

The plan keeps the asset ID and the version of the skill. If the skill appears, goes away, or gets a new version before the write, setup does not change it and shows a warning.

The skill step needs these permissions: aidevops:ListAssets, aidevops:GetAssetFile, aidevops:CreateAsset, and aidevops:UpdateAsset. Without them, setup does the other steps and warns about the skill.

Describe alternatives you've considered

Setup can always replace the skill text. I did not do this, because a run of setup will then remove changes that the user made.

A skill failure can stop setup. I did not do this, because the MCP server is the main step and the skill is optional.

Additional context

The custom agent of Part 4 will come in a different pull request. The asset API can create it as a custom_agent asset that refers to this skill by asset ID. CreateTrigger can run it on a schedule. The setup result now includes skillAssetId for that step.

The status command does not show the skill yet.

The code reads the skill name from the JSON of the metadata document. This works the same for AWS responses and for the documents that the tests make with NewLazyDocument. UnmarshalSmithyDocument returns an error on a NewLazyDocument document.

🤖 Generated with Claude Code

---

Note

Overview
ldcli aws-devops-agent setup now provisions the embedded experiment-orchestration skill (create, keep, auto-update from shipped hashes, or leave user edits unless --replace-skill). The plan adds a Skill: line; JSON/plain output expose skillAssetId, skillAction, and skillError. Skill failures are optional—setup still finishes and prints a single warning (stderr in JSON mode).

Connect and full setup runs also sync LaunchDarkly MCP tool allowlists when reusing a server: new tools include update-flag-settings, create-metric, and apply-approval-request. Missing ldcli-only tools are called out in the plan; --add-mcp-tools rewrites the association with ldcli classifications (skipped if foreign tools or post-plan drift). mcpToolsAdded / mcpToolsError capture outcomes without failing setup.

Help text and tests cover plan/cancel behavior, permission denials, and allowlist edge cases.

^{Reviewed by Cursor Bugbot for commit f9893d2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cchu-ld]

Test run results - first time sets up the skill, second time is a no-op and re-uses the existing skill, which is expected.

First run:

Setup plan:
  AWS account:   506919356135
  Region:        us-east-1
  AWS profile:   launchdarkly-development_Administrator
  IAM role:      Leave role DevOpsAgentRole-AgentSpace unchanged because it has no ldcli:managed tag
  IAM role:      Leave role DevOpsAgentRole-WebappAdmin unchanged because it has no ldcli:managed tag
  Agent space:   Create agent space cchu-test-skill
  Association:   Associate AWS account 506919356135 with the agent space
  Operator app:  Enable the operator app with IAM sign-in
  MCP server:    Reuse MCP server LaunchDarkly (9f4fc339-80ea-42a9-8b0c-daa49b9b420a)
  Skill:         Create skill experiment-orchestration for every agent type in the space

Second:

Setup plan:
  AWS account:   506919356135
  Region:        us-east-1
  AWS profile:   launchdarkly-development_Administrator
  IAM role:      Leave role DevOpsAgentRole-AgentSpace unchanged because it has no ldcli:managed tag
  IAM role:      Leave role DevOpsAgentRole-WebappAdmin unchanged because it has no ldcli:managed tag
  Agent space:   Reuse agent space cchu-test-skill (dc00f70b-461c-496c-a320-b3ad0071798d)
  Association:   Keep the AWS account 506919356135 association
  Operator app:  Keep the operator app and its sign-in
  MCP server:    Reuse MCP server LaunchDarkly (9f4fc339-80ea-42a9-8b0c-daa49b9b420a)
  Skill:         Keep skill experiment-orchestration (ki-a4a4922b-0a7e-409b-a731-05907da92975)

AWS DevOps Agent skill panel:

[nhironaka]

is this skill aws specific?
we've defined a lot of public facing skills in our ai-tooling repo
wondering if there's a way we can consolidate

[cchu-ld]

It's not AWS specific, but it does more orchestration than either of the experimentation skills in ai-tooling. For example this skill covers implementation, deploy, and rollout, whereas the other skill stop after experiment setup.

Edit: Also ai-tooling assumes it's run in something like claude or cursor, but AWS DevOps doesn't make code changes themselves and has to hand it that part back to the user.

[nhironaka]

I guess my question, can this be defined in our ai-tooling repo, and imported in? or does it have to be defined in this repo

[nhironaka]

Also some robot generated feedback on this skill:

1. Report template headings still collide with the skill's own structure. ## Experiment Report: [Goal] and ### Hypothesis… sit at the same level as ## Step 6: Report and ## Safety rules, and the skill now has # Start an experiment / # Check on experiments H1s too. Put the whole template in a fenced ```markdown block so the skill's own outline isn't interleaved with the report's.
2. "Metric change: [before] → [after]" in the report template is still before/after. An A/B result is control vs treatment — before/after invites confounding. Rename the line: Metric change: [control] → [treatment] ([+/-]%).
3. State-machine gap: "design shown, awaiting confirmation" is not a record state. Step 4.3 ends the turn showing the design and waiting for confirmation, but the record still says "waiting for deployment". If that chat dies, the next check-in has no way to know a design was already drafted and needs confirming, not restarting. Add that record state (or fold design-confirmation into Step 4's entry condition in Step 5).
4. Sequential-testing tension. Step 4 enables sequentialTestingEnabled precisely so early reads don't inflate false positives — then Step 5 says never act on significance before 2 full weeks. The 2-week floor is defensible (weekday/weekly cycles), but say that's why, or you've enabled a feature and then forbidden its main benefit. One sentence fixes it.
5. Winner cleanup is missing. Next steps only recommends removing the flag for non-winners. After a rollout reaches 100%, the flag is dead weight too — recommend the cleanup PR (remove flag, keep the winning path) for winners as well.

[cchu-ld]

Re: ai-tooling - In this sense it is a AWS-specific issue - the AWS DevOps Agent has to have a GitHub connection to import a file from a Github URL. However we don't control what connections our users would have in their space. So a local copy is the only guaranteed way for us to script the skill install.

We could also make ai-tooling the source of truth and sync it into ldcli at build time - that could be a follow-up PR.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[nhironaka]

I guess my question, can this be defined in our ai-tooling repo, and imported in? or does it have to be defined in this repo

[nhironaka]

Also some robot generated feedback on this skill:

1. Report template headings still collide with the skill's own structure. ## Experiment Report: [Goal] and ### Hypothesis… sit at the same level as ## Step 6: Report and ## Safety rules, and the skill now has # Start an experiment / # Check on experiments H1s too. Put the whole template in a fenced ```markdown block so the skill's own outline isn't interleaved with the report's.
2. "Metric change: [before] → [after]" in the report template is still before/after. An A/B result is control vs treatment — before/after invites confounding. Rename the line: Metric change: [control] → [treatment] ([+/-]%).
3. State-machine gap: "design shown, awaiting confirmation" is not a record state. Step 4.3 ends the turn showing the design and waiting for confirmation, but the record still says "waiting for deployment". If that chat dies, the next check-in has no way to know a design was already drafted and needs confirming, not restarting. Add that record state (or fold design-confirmation into Step 4's entry condition in Step 5).
4. Sequential-testing tension. Step 4 enables sequentialTestingEnabled precisely so early reads don't inflate false positives — then Step 5 says never act on significance before 2 full weeks. The 2-week floor is defensible (weekday/weekly cycles), but say that's why, or you've enabled a feature and then forbidden its main benefit. One sentence fixes it.
5. Winner cleanup is missing. Next steps only recommends removing the flag for non-winners. After a rollout reaches 100%, the flag is dead weight too — recommend the cleanup PR (remove flag, keep the winning path) for winners as well.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 58b1055. Configure here.}