feat: improve agent experience of the CLI and public skills #4492

User Story

I use OpenShell through coding agents that load the public OpenShell skills. I need those agents to find the right command, avoid invalid flag combinations, and check policy files without trial runs against a gateway, so that agent sessions are reliable and cheap.

Problem Statement

This follows up #2736 and #2899, which moved the user-facing skills out of .agents/skills/ into the installable skills/ collection. Those skills now ship to users, so how well agents use them matters.

Coding agents operate OpenShell through the public skills in skills/ and the openshell CLI. Three gaps make that slower and less reliable than it needs to be:

• Agents cannot see the whole CLI. --help is nested, so finding a command takes several calls, and the skills show only some commands. Neither --help nor the skills say which flags conflict.
• Skills are large and drift. When an agent uses a skill, it reads the whole SKILL.md, even when the task needs a small part of it. openshell-cli is 51 KB and debug-openshell-cluster is 78 KB. Both repeat command syntax by hand, and two openshell-cli examples are rejected by the current CLI.
• Policies cannot be checked before they are applied. No command checks a policy file without a gateway. policy set parses the file locally, but if it parses, the policy is applied, and safety violations come back from the gateway.

Impact / Why This Matters

We measured these gaps in more than 1,600 subagent trial sessions on Claude Haiku, Sonnet, and Opus (method and results under Agent Investigation). On main:

• In four-task CLI sessions, agents made 11 to 14 CLI calls, most of them --help, and about one rejected command per session. They found sandbox get --policy-only, which the skill does not show, in 1 of the 9 sessions that needed it.
• Agents read debug-openshell-cluster in two chunks, and they tried both stale openshell-cli examples.
• Agents writing policies looked for a validator anyway: in 23 of 27 policy-writing sessions, they ran openshell policy validate, which does not exist.

Every extra call is another turn, and every turn resends the whole conversation, so these gaps make agent sessions slower and more expensive and leave agents guessing at flag combinations. With the changes proposed below, agents on all three models answered every task correctly and cost 15% to 24% less on CLI and debugging tasks than with main. They made 3 rejected commands in 324 sessions. On main, agents made 0.7 to 1.5 per session on CLI and policy tasks.

Proposed Design

Add two CLI commands and restructure the public skills around them.

1. openshell agent context prints a map of the installed CLI, generated from its command definitions so it cannot drift: every command with its usage line and summary, and every flag with its allowed values, default, environment variable, and the flags it cannot be combined with. Flag descriptions stay in --help. The map is about 20 KB, small enough to read in one agent tool call.
2. openshell policy validate --policy FILE runs the schema and safety validation the gateway also applies to a submitted policy, prints every violation, and exits nonzero. It needs no gateway and changes nothing. The gateway still runs its other checks, such as endpoint ambiguity, provider composition, and registered middleware, when the policy is applied.
3. Skills start from the map and load only what a task needs:
   • openshell-cli: a short SKILL.md with the bootstrap, rules that apply to every command, core concepts, and a task index, plus reference files for each workflow. Fix the two rejected examples.
   • debug-openshell-cluster: a triage SKILL.md, plus reference files per platform (Docker, Podman, Kubernetes, VM) and for gateway startup.
   • generate-sandbox-policy: validate every generated policy with openshell policy validate. The skill stays one file.
   • debug-inference: unchanged; it is already short.
   • Tell agents to read what they need in one step, one call per file, and keep every file small enough for one tool result.

Suggested UX

$ openshell agent context
...
openshell sandbox create [OPTIONS] [-- <COMMAND>...]  # Create a sandbox
  --upload <UPLOAD>  [repeatable] [conflicts: <COMMAND>...]
  --no-keep  [conflicts: --editor, --forward, --expose, --output]
  --restart-policy <never|on-failure|always>  [default: never]
...

$ openshell policy validate --policy policy.yaml
  network policy 'github_api': endpoint 0 has invalid L7 configuration: rules and access are mutually exclusive
Error:   × policy.yaml has 1 policy violation(s)

Acceptance Criteria

• openshell agent context lists every visible command and flag with allowed values, defaults, environment variables, and conflicts, in under 25,000 bytes.
• openshell policy validate reports schema and safety violations in a policy file without contacting a gateway.
• Public skills bootstrap from openshell agent context, and every openshell example they contain parses on the current CLI.
• In controlled agent trials on more than one model, agents using the updated skills and CLI are at least as accurate as with main and make fewer rejected commands.

Alternatives Considered

• Nested --help only: several calls per command, no conflict information, and missed commands the skills do not show.
• A complete single-call reference with every flag description: at 43 KB it exceeded Claude Code's 30,000-character tool output limit. In design trials, agents received a 2 KB preview and paged through a saved file, using 49% more input tokens and 2.5 more turns per session than main.
• Reading the map and references in one command: the combined output passed the same limit. Agents then paged through saved output, which added turns at medium reasoning effort until the skills asked for one call per file.
• Splitting every skill: in design trials, splitting generate-sandbox-policy cost 9% to 12% more tokens, because writing a policy needs most of the skill and agents read every reference file.
• Hand-maintained command references in skills: they drift; two current examples already fail.

Agent Investigation

We measured every change with controlled subagent trials before keeping it. Design trials compared candidate versions, including the alternatives above. We then compared the final changes against main on Claude Haiku, Sonnet, and Opus at two reasoning efforts, running every task batch three times per model.

Method. Each session received one version of a skill and one build of the CLI, then worked through a batch of realistic requests. Arms differed only in the skill and the CLI. Agents worked in isolated run directories that did not reveal which version they had, with an unreachable gateway, and every CLI call went through a logging wrapper. Answers were graded automatically: commands had to parse on the real CLI and meet a per-task rubric, and policy files had to pass openshell policy validate and content checks. Token counts, turns, and time came from the session transcripts. The task sets covered requests the skills already show, requests they never show, and traps such as flag conflicts and stale examples.

Results against main. Cost weights tokens by price. Changes are the mean change per task batch, pooled across the three models, with 95% bootstrap intervals.

What the trials showed:

• The map fixed discovery. On main, agents found sandbox get --policy-only in 1 of 9 sessions that needed it; with the map and a one-line pointer in the skill, every session found it.
• Round trips drive cost. Every turn resends the whole conversation, so the first skill versions read less text but took more turns and saved little. Asking agents to read everything they need in one step cut turns and raised the savings on debugging tasks at high effort from 16% to 22%.
• Every read has to fit in one tool result. When the map and references were printed together, or a reference was larger than about 30 KB, agents saw a truncated preview and paged through the rest. Reading one file per call fixed it: at medium effort, CLI sessions went from 5% more turns than main to 8% fewer, and Haiku debugging sessions from 27% more turns to 2% more. In the final CLI and debugging sessions, agents hit the limit in 2 of 270 sessions with the updated skills, compared with 76 of 270 with main.
• The few remaining rejected commands were deliberate. All 3 came from Sonnet sessions running sandbox create --forward --no-keep to confirm a conflict that the map already lists. Each of those sessions then gave the correct answer.
• Agents expect a policy validator. On main, agents ran openshell policy validate in 23 of 27 policy-writing sessions, and the CLI rejected every call. With the command, they validated every file they wrote, made no rejected commands, and at medium effort also answered the 4 tasks that agents on main got wrong.