Repository navigation
feat(skills): improve agent experience of the CLI and public skills - #4493
Open
johnnygreco wants to merge 15 commits into
Open
johnnygreco wants to merge 15 commits into
johnnygreco wants to merge 15 commits into
Conversation
Add `openshell agent context [COMMAND]...`, which prints every visible command, argument, and flag as one plain-text reference for coding agents. The text is projected from the clap command tree at runtime, so it always matches the installed binary instead of drifting like hand-written docs. Each command lists its usage, aliases, arguments, flags with defaults, environment variables, allowed values, and conflicts, plus the examples from its help text. Global flags appear once. Hidden commands and flags are omitted, ANSI styling is stripped, and output is deterministic. A command path such as `sandbox` or `sb create` narrows the output. Point the public skills at the new command, keeping nested `--help` as the fallback for older CLIs, and note it in the README and installation docs. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
`openshell agent context` printed the whole CLI at once, about 55 KB. Without arguments it now prints global flags and an overview that lists every command, one line each, grouped into sections of commands that are used together. `openshell agent context <section>` prints full usage, flags, and examples for one section, and a command path still prints one command or group. The section table lives beside the CLI definition. A command path can move a subtree into another section, so `sandbox provider` sits with providers. Commands that no section claims still appear in an `other` section, and a unit test fails until every visible command belongs to exactly one section. Integration tests check that every overview entry renders in its section and keep the overview and sections under size budgets. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Agents should bootstrap from one call. Replace the overview and sections with one compact reference: every command with its usage line and description, and every flag on one line with its description, allowed values, default, and environment variable. Descriptions keep their full text folded onto one line, which costs little and carries value formats such as `--upload <LOCAL_PATH>[:<SANDBOX_PATH>]`. Flag conflicts are not listed. `agent context` no longer takes arguments. A unit test checks that every visible command appears, and an integration test keeps the reference under a size budget. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Agent harnesses cap a single tool result. Claude Code moves output over 30,000 characters to a file and shows a 2 KB preview, so agents spent extra turns paging through the 43 KB reference. Print a map instead: every command with its usage line and summary, and every flag with its allowed values, default, environment variable, and the flags it cannot be combined with. Allowed values replace the value placeholder, and conflicts come from clap's conflict data, which `--help` does not show. Flag descriptions stay in `--help`. The map is about 20 KB and 444 lines, and an integration test keeps it under 25,000 bytes. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Whenever an agent used the skill, all 51 KB loaded, even when the task needed a small part of it. Keep a short SKILL.md with the `openshell agent context` bootstrap, rules that apply to every command, core concepts, and a task index. Move the workflow sections into reference files under references/ that agents read only when a task needs them. Replace the create flag list, which the command map now generates, with notes on behavior the map does not show. Fix two examples the CLI rejects: `sandbox create` cannot combine `--upload` with a trailing command, and `gateway info` takes the global `--gateway` flag rather than `--name`. In subagent trials across 24 tasks, agents with the split skill and the command map answered 48 of 48 correctly against 46 of 48 with the previous skill, using 19% fewer input tokens and 46% fewer CLI calls. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Whenever an agent used the skill, all 78 KB loaded, including the Docker, Podman, Kubernetes, and VM diagnostics a task did not need, and agents read it in two chunks. Keep triage in SKILL.md: CLI reachability, platform identification, platform-independent failure patterns, reporting, and configuration preflight. Move gateway startup dependencies and each platform's checks into references/ without rewording them, so agents read only the platform they are debugging. Kubernetes and OpenShift failure patterns move with the Kubernetes reference. In subagent trials across 12 diagnosis tasks, both versions answered every task correctly. Single-platform sessions used 18% to 24% fewer input tokens, and single-task sessions used 31% fewer. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
`policy set` only parses a policy file locally; the gateway reports violations after the file is sent. Add `openshell policy validate --policy FILE`, which parses the file and runs the schema and safety validation the gateway also applies to a submitted policy, then prints every violation and exits nonzero. It needs no gateway, so it runs before gateway resolution. The gateway runs more checks when the policy is applied, including endpoint ambiguity, provider composition, and registered middleware. Coding agents looked for this command: in subagent trials, agents writing policies without it tried `openshell policy validate --help` five times in six sessions. The manage-policies guide and the openshell-cli skill now check a file locally before `policy set`. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Before presenting a policy, have generate-sandbox-policy write it to a file and run `openshell policy validate`, fixing each reported violation until the file is valid. Keep the manual checklist for checks that need the gateway and for CLIs without the command. In subagent trials across 9 policy-writing tasks, agents with this step validated every policy they wrote and made no rejected CLI calls, against 1.8 per session with the previous skill, at the same token cost and with the same accuracy. Splitting this skill into references was also tried and cost 9% to 12% more tokens, because authoring a policy needs most of the skill, so the skill stays a single file. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Agent trials showed the remaining cost of these skills comes from extra steps, not from content. The openshell-cli skill now tells agents to read the map and the references they need in one step and to run --help only when a flag's meaning is unclear. The debugging skill now scopes the CLI map to openshell commands, since agents searched it for openshell-gateway commands it does not list, and asks agents to read several references together. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
The policy reference showed only the policy get export pipeline, so agents asked to print a sandbox's policy copied that pipeline instead of using sandbox get --policy-only. Name the purpose-built flag in the skill and the policy reference, and keep the export pipeline for editing. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Agents that printed the CLI map and several references in one command, or printed the 35 KB Kubernetes reference, received a truncated preview and spent extra turns paging through the saved output. Ask for one call per file and split the TLS and multi-replica checks into their own Kubernetes reference. Signed-off-by: Johnny Greco <jogreco@nvidia.com>
|
🌿 Preview your docs: https://nvidia-preview-pr-4493.docs.buildwithfern.com/openshell |
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improve the experience of coding agents that drive OpenShell through its public skills. Agents start from a generated CLI map, skills load only what a task needs, and agents can check policy files locally. This follows up #2736 and #2899, which moved the user-facing skills out of
.agents/skills/intoskills/. Each change was kept only when controlled subagent trials showed agents doing at least as well as withmain.Related Issue
Closes #4492
Changes
openshell agent context: prints a map of every command and flag generated from the clap definitions, including allowed values, defaults, environment variables, and conflicting flags, which--helpdoes not show. About 20 KB, small enough for one agent tool result.openshell policy validate --policy FILE: runs the schema and safety validation the gateway also applies to a submitted policy, without contacting a gateway, and prints every violation.openshell-cliskill: a 7 KBSKILL.mdwith the bootstrap, rules for every command, concepts, and a task index, plus five on-demand references. Tells agents to read the map and references in one step, one call per file. Fixes two examples the CLI rejects. Recentmainadditions to this skill (provider labels, upload streaming, andvalidation unavailablefor rule approval) now live in the matching references.debug-openshell-clusterskill: a triageSKILL.mdplus per-platform references for Docker, Podman, Kubernetes, VM, and gateway startup. Every reference fits in one tool result.generate-sandbox-policyskill: validates generated policies withpolicy validate. It stays one file; splitting it cost more.openshell <command> --helpfor flag descriptions wherever they mentionagent context, in place of separate instructions for CLIs without the new commands.sync-agent-infraskill describe theagent contextbootstrap. README and CONTRIBUTING show how to install the skills from a specific release withnpx skills add NVIDIA/OpenShell#<release-tag>.Compatibility. The CLI changes are additive: two new commands, and no change to any existing command, flag, output, or exit code. The skills are reorganized rather than extended. On a release without the new commands,
--helpstill works, and users can install the skills from their release tag.Agent Trial Results
Each run gave a subagent one skill version and one CLI build and a set of realistic tasks. Runs used Claude Haiku, Sonnet, and Opus at two reasoning efforts, comparing this branch against
mainon the same tasks. Answers were graded against the real CLI or by validating the policy files. Cost weights tokens by price. Changes are pooled across models, with 95% intervals.main→ branch)openshell-cliopenshell-clidebug-openshell-clusterdebug-openshell-clustergenerate-sandbox-policygenerate-sandbox-policyThe trials ran on this branch before its rebase onto current
main. The rebase added onlymain's own changes and moved the three skill passages noted above. Later commits only reworded the notes about older CLIs and skill installs. Across all models and both efforts, agents made 3 rejected commands in 324 runs. All 3 came from Sonnet confirming a flag conflict that the map already lists, and each of those runs answered correctly. Skill reads exceeded Claude Code's 30,000-character tool output limit in 2 of 270 CLI and debugging runs, compared with 76 of 270 onmain.Testing
cargo fmt --all -- --checkandcargo clippy -p openshell-cli --all-targets -- -D warningsopenshell-clilib (380), bin (121), and every integration test target exceptpolicy_only_process_identity_integration, which needslibz3and was not run locallymarkdownlint-cli2on the 20 changed Markdown files, the SPDX license header check, and a parse check of all 128 placeholder-freeopenshellexamples in the public skillsopenshell agent contextprints 19,914 bytes and includes the new provider--labelflagsChecklist