Skip to content

process_binary_aware_network_policy: approved + reloaded draft chunks still reject matching traffic and keep re-drafting the same rule #2752

Closed
Closed
process_binary_aware_network_policy: approved + reloaded draft chunks still reject matching traffic and keep re-drafting the same rule#2752
Assignees
johntmyers

Description

@thoraxe
thoraxe
opened on Aug 14, 2026

Summary

process_binary_aware_network_policy's learn-and-approve workflow doesn't actually gate live traffic: approving a draft chunk (via ApproveAllDraftChunks) and confirming the sandbox reloaded the new policy version (ReportPolicyStatus: status=loaded) does not unblock matching connections. A fresh attempt from the exact same binary to the exact same already-approved host:port is still rejected, and — confusingly — generates a brand-new duplicate draft chunk for the identical rule instead of being recognized as already covered.

Reproduction

  1. Kubernetes driver, topology = "sidecar", process_binary_aware_network_policy = true (under [openshell.drivers.kubernetes.sidecar]).
  2. Start a workload inside the sandbox (via ExecSandbox) that makes an outbound HTTP request to an in-cluster service, e.g. http://some-service.ns.svc:8000/v1/..., using the sandbox's own HTTP_PROXY=http://127.0.0.1:3128 (confirmed listening/reachable) and NODE_USE_ENV_PROXY=1.
  3. First attempt fails: fetch failed / Request was cancelled. Server-side, SubmitPolicyAnalysis logs a new draft chunk, e.g. allow_some_service_ns_svc_8000, scoped to the exact resolved binary path (confirmed via readlink -f $(which node) matching exactly what's recorded in binaries=[...] on the rule).
  4. Call ApproveAllDraftChunks({name: "<sandbox-short-name>", include_security_flagged: true}). Server logs CONFIG:APPROVED ... add-rule allow_some_service_ns_svc_8000 ... binaries=[/usr/local/bin/node] and CONFIG:MERGED, returning a new policy_version.
  5. Server logs ReportPolicyStatus: sandbox reported policy load result ... status=loaded for that same version shortly after — confirming the sandbox side has fetched and applied the new policy.
  6. Retry the exact same request (same binary, same host, same port):
    • Same failure: fetch failed / Request was cancelled.
    • A brand-new SubmitPolicyAnalysis draft chunk gets submitted for the identical rule (draft_version incremented), as if the already-approved rule were never consulted.

We tried, in order, to rule out staleness/propagation-timing explanations — none changed the outcome:

  • Relaunching the workload process after approval (same pod): still fails.
  • Not relaunching at all (same live process, same TCP stack): still fails.
  • Deleting the pod entirely so the driver's reconciler recreates it fresh (new containers, policy fetched from scratch at startup rather than live-reloaded): still fails, and still re-drafts the same already-approved rule.

This suggests the enforcement decision path (in the network-mode supervisor process) isn't actually consulting the same rule set that SubmitPolicyAnalysis/ApproveAllDraftChunks operate on — or there's a scoping key mismatch (e.g. hostname normalization, binary hash vs. path, or a workspace parameter defaulting differently between the two paths) that we couldn't identify from the client side.

Impact

With process_binary_aware_network_policy = true, no amount of legitimate approval workflow usage unblocks real traffic, making the feature effectively non-functional for any egress beyond whatever ships pre-approved. We also tried process_binary_aware_network_policy = false as a way to fall back to plain L3/L4 enforcement — that made things strictly worse: even loopback-adjacent DNS-via-the-local-proxy path broke (getaddrinfo EAI_AGAIN when bypassing the proxy directly, and the proxy path itself started failing more baseline requests, including ones that previously depended on nothing but plain NetworkPolicy L3/L4 matching that we'd already correctly scoped at the Kubernetes level).

Environment

  • OpenShell built from dev branch, commit c4b500a (2026-08-14)
  • Kubernetes driver, topology = "sidecar", sa_token_ttl_secs = 3600, enable_user_namespaces = true
  • Workaround we're evaluating: bypass sandbox-originated outbound network calls entirely for this specific traffic (file-based request/response bridge polled via ExecSandbox, actual HTTP call made from outside the sandbox).

Activity

  1. github-actions
    added
    state:triage-neededOpened without agent diagnostics and needs triage
    on Aug 14, 2026

  2. johntmyers commented on Aug 14, 2026

    @johntmyers
    johntmyers
    on Aug 14, 2026
    Collaborator

    @thoraxe You’re still getting connections denied because the approved chunk targets an endpoint that resolves to private IP space, so the connection is subsequently blocked by the SSRF checks.

    Policy-advisor-generated rules deliberately cannot authorize private-address access, even after human approval.

    What isn’t clear from the issue is this:

    A brand-new SubmitPolicyAnalysis draft chunk gets submitted for the identical rule (draft_version incremented), as if the already-approved rule were never consulted.

    Are you seeing an actual second draft-rule entry with a different chunk ID, or merely a second SubmitPolicyAnalysis RPC call?

    If it is only another RPC call, that is expected:

    1. The initial network-policy check denies the connection, producing a proposed rule.
    2. You approve the rule.
    3. On retry, the approved rule passes the host/port/binary policy check.
    4. The subsequent SSRF check rejects the destination because the .svc hostname resolves to private IP space.
    5. That new SSRF denial generates another SubmitPolicyAnalysis call for the same host, port, and binary.

    The second submission should be deduplicated against the existing chunk and should not create a new pending rule in the TUI. Please verify whether the returned chunk ID changes and whether a new pending rule actually appears. The server calculates/logs a new draft_version before the persistence upsert, so the incremented version alone does not prove that a second row was created.

    Please also check the OCSF logs. I expect the retry to show the network-policy decision as allowed, followed by an engine:ssrf denial indicating that the hostname resolved to a private/internal address.

    If the workload needs to reach private services, declare the .svc hostname directly in the sandbox policy instead of relying on a policy-advisor proposal. An exact hostname explicitly declared by the operator may resolve to private IP space without requiring allowed_ips. You can also use host + allowed_ips if you want to constrain the permitted private addresses. Loopback, link-local, unspecified addresses, and protected control-plane ports remain blocked.

  3. thoraxe commented on Aug 14, 2026

    @thoraxe
    thoraxe
    on Aug 14, 2026
    Author

    Thanks for the tips -- I'll dig in and follow-up.

  4. thoraxe commented on Aug 17, 2026

    @thoraxe
    thoraxe
    on Aug 17, 2026
    Author

    Reproduced this directly against the gateway's real API to check whether this is SSRF rejection-after-approval vs. non-functional enforcement.

    Setup: gateway/chart 0.0.104, relevant config:

    [openshell.gateway]
policy_validation_failure_mode = "fail_closed"

[openshell.drivers.kubernetes]
topology = "sidecar"   # "combined" hits the CAP_SETPCAP bug from #2751

[openshell.drivers.kubernetes.sidecar]
proxy_uid = 1337
process_binary_aware_network_policy = true

    Repro script (Python, via the generated openshell_pb2/openshell_pb2_grpc stubs, mTLS + a Keycloak-issued bearer JWT with aud: [forge-api, openshell-cli]):

    import grpc
from proto import openshell_pb2, openshell_pb2_grpc

creds = grpc.ssl_channel_credentials(root_certificates=ca, private_key=key, certificate_chain=cert)
channel = grpc.secure_channel("localhost:18080", creds)
stub = openshell_pb2_grpc.OpenShellStub(channel)
md = (("authorization", f"Bearer {TOKEN}"),)

# CreateSandbox: minimal template (image + env only), no policy attached in the request
req = openshell_pb2.CreateSandboxRequest(
    name="ssrf-repro-1",
    spec=openshell_pb2.SandboxSpec(
        template=openshell_pb2.SandboxTemplate(image="quay.io/thoraxe/openclaw-forge:latest", environment={}),
    ),
)
resp = stub.CreateSandbox(req, metadata=md)
sandbox_id = resp.sandbox.metadata.id
# ... WatchSandbox until phase == SANDBOX_PHASE_READY ...

# ExecSandbox: attempt egress to a private, undeclared .svc target, 3x
cmd = ["node", "-e",
    "require('http').get('http://forge-analyst-external-vllm-proxy.forge-system.svc.cluster.local:8000/', "
    "r=>{let d='';r.on('data',c=>d+=c);r.on('end',()=>console.log('STATUS',r.statusCode))}).on('error',e=>console.log('ERR',e.message));"]

def run_exec():
    exec_req = openshell_pb2.ExecSandboxRequest(sandbox_id=sandbox_id, command=cmd, timeout_seconds=10)
    for event in stub.ExecSandbox(exec_req, metadata=md, timeout=15):
        which = event.WhichOneof("payload")
        if which == "stdout": print("STDOUT:", event.stdout.data)
        elif which == "stderr": print("STDERR:", event.stderr.data)
        elif which == "exit": print("EXIT:", event.exit.exit_code)

run_exec(); run_exec(); run_exec()  # 3 identical attempts

    Output, all 3 attempts identical:

    STDOUT: b'STATUS 403\n'
EXIT: 0

    (403 is the vLLM proxy's own auth rejection — proof of a completed TCP+TLS+HTTP round trip, not a network-level block. No ERR, no timeout, ever.)

    Gateway logs (ANSI stripped, noise from unrelated RPCs filtered out):

    2026-08-17T13:36:39.454516Z  INFO CreateSandbox: minted sandbox JWT sandbox_id=632f85c3...
2026-08-17T13:36:39.455892Z  INFO driver.create_sandbox: Creating sandbox in Kubernetes sandbox_id=632f85c3... sandbox_name=ssrf-repro-1
2026-08-17T13:36:39.458535Z  INFO driver.create_sandbox: Resolved namespace annotations uid_range=1000800000/10000
2026-08-17T13:36:39.458552Z  INFO driver.create_sandbox: Resolved sandbox identity uid=1000800000 gid=1000800000
2026-08-17T13:36:39.464797Z  INFO driver.create_sandbox: Sandbox created in Kubernetes successfully sandbox_id=632f85c3...
2026-08-17T13:36:39.464926Z  INFO CreateSandbox request completed successfully sandbox_id=632f85c3...
2026-08-17T13:36:43.900313Z  INFO UpdateConfig: backfilled spec.policy from sandbox-discovered policy sandbox_id=632f85c3...
2026-08-17T13:36:43.924624Z  INFO ReportPolicyStatus: sandbox reported policy load result sandbox_id=632f85c3... version=1 status=loaded
2026-08-17T13:36:43.978935Z  INFO ExecSandbox RPC accepted, relay opened
2026-08-17T13:36:43.979428Z  INFO ExecSandbox (relay): command started sandbox_id=632f85c3... command_preview=node -e '...forge-analyst-external-vllm-proxy.forge-system.svc.cluster.local:8000/...'
2026-08-17T13:36:53.921814Z  INFO SubmitPolicyAnalysis RPC accepted
2026-08-17T13:36:53.929419Z  INFO SubmitPolicyAnalysis: persisted draft chunks sandbox_id=632f85c3... accepted=1 rejected=0 draft_version=1 summaries=1

--- attempts 2 & 3 (back-to-back) ---
2026-08-17T13:37:41.288825Z  INFO ExecSandbox RPC accepted, relay opened  (attempt 2)
2026-08-17T13:37:41.289383Z  INFO ExecSandbox (relay): command started channel_id=023700a6...
2026-08-17T13:37:41.363049Z  INFO ExecSandbox RPC accepted, relay opened  (attempt 3)
2026-08-17T13:37:41.363522Z  INFO ExecSandbox (relay): command started channel_id=b4db2710...
2026-08-17T13:37:43.922747Z  INFO SubmitPolicyAnalysis RPC accepted
2026-08-17T13:37:43.931129Z  INFO SubmitPolicyAnalysis: persisted draft chunks sandbox_id=632f85c3... accepted=1 rejected=0 draft_version=2 summaries=1

    A policy was confirmed loaded on the sandbox before the first attempt, and SubmitPolicyAnalysis clearly runs (drafting chunks, draft_version incrementing 1→2) — but it only fires after each connection has already completed successfully, and never blocked, delayed past ~10s, or required approval for any of the three identical attempts to a private/undeclared endpoint. This doesn't match the SSRF-rejection-after-approval pattern (which predicts a deny on retry); we saw zero denials.

  5. johntmyers commented on Aug 18, 2026

    @johntmyers
    johntmyers
    on Aug 18, 2026
    Collaborator

    @thoraxe can you provide the Sandbox logs as well?

  6. johntmyers
    self-assigned this
    on Aug 18, 2026

  7. thoraxe commented on Aug 18, 2026

    @thoraxe
    thoraxe
    on Aug 18, 2026
    Author

    @johntmyers Here are the sandbox-side logs, pulled via GetSandboxLogs (the openshell-supervisor-network sidecar doesn't write to stdout, so kubectl/oc logs on that container is empty — everything goes through the structured/OCSF buffer instead).

    Fresh repro, same gateway/chart 0.0.104 and config as before (topology = "sidecar", process_binary_aware_network_policy = true, policy_validation_failure_mode = "fail_closed"), same 3x identical ExecSandbox attempts against the same undeclared private .svc target:

    Sandbox logs (GetSandboxLogs, sandbox_id=caa43e1b-33f3-4791-a627-9074fca0493b, name ssrf-repro-2):

    [1787057219157] INFO gateway openshell_server::grpc::sandbox: minted sandbox JWT
[1787057219158] INFO gateway openshell_driver_kubernetes::driver: Creating sandbox in Kubernetes
[1787057219168] INFO gateway openshell_driver_kubernetes::driver: Sandbox created in Kubernetes successfully
[1787057219168] INFO gateway openshell_server::grpc::sandbox: CreateSandbox request completed successfully
[1787057223547] INFO gateway openshell_server::grpc::auth_rpc: issued gateway sandbox JWT
[1787057223525] INFO sandbox openshell_sandbox: Starting sandbox {command: '["sleep", "infinity"]'}
[1787057223525] INFO sandbox openshell_sandbox: Fetching sandbox policy via gRPC {endpoint: https://openshell.forge-system.svc.cluster.local:8080, sandbox_id: caa43e1b-...}
[1787057223533] INFO sandbox openshell_core::grpc_client: exchanging K8s ServiceAccount token for sandbox JWT
[1787057223554] INFO gateway openshell_server::grpc::policy: UpdateConfig: backfilled spec.policy from sandbox-discovered policy
[1787057223561] INFO gateway openshell_server::grpc::policy: GetSandboxProviderEnvironment request completed successfully
[1787057223577] INFO gateway openshell_server::grpc::policy: ReportPolicyStatus: sandbox reported policy load result
[1787057223666] INFO gateway openshell_server::supervisor_session: supervisor session: accepted
[1787057223548] OCSF sandbox ocsf: CONFIG:DISCOVERY [INFO] Server returned no policy; attempting local discovery
[1787057223548] OCSF sandbox ocsf: CONFIG:DEFAULT [INFO] No policy file on disk, using restrictive default [path:/etc/openshell/policy.yaml]
[1787057223555] INFO sandbox openshell_sandbox: Creating OPA engine from proto policy data
[1787057223555] INFO sandbox openshell_supervisor_network::opa: Configured OPA runtime binary identity mode {source: proto, require_binary_identity: true}
[1787057223555] OCSF sandbox ocsf: CONFIG:CONFIGURED [INFO] OPA runtime binary identity mode configured [source:proto require_binary_identity:true]
[1787057223562] OCSF sandbox ocsf: CONFIG:LOADED [INFO] Fetched provider environment [env_count:0]
[1787057223564] OCSF sandbox ocsf: CONFIG:ENABLED [INFO] TLS termination enabled: ephemeral CA generated
[1787057223564] INFO sandbox openshell_supervisor_network::inference_routes: Fetching inference route bundle from gateway
[1787057223568] OCSF sandbox ocsf: CONFIG:LOADED [INFO] Loaded inference route bundle [route_count:0 revision:d1fba762150c532c]
[1787057223568] OCSF sandbox ocsf: CONFIG:WAITING [INFO] Inference route bundle is empty; keeping routing enabled and waiting for refresh
[1787057223568] OCSF sandbox ocsf: CONFIG:ENABLED [INFO] Inference routing enabled with local execution [route_count:0]
[1787057223570] OCSF sandbox ocsf: NET:LISTEN [INFO] 127.0.0.1:3128
[1787057223570] INFO sandbox openshell_sandbox::sidecar_control: Sidecar control socket listening {path: /run/openshell-sidecar/control.sock}
[1787057223577] OCSF sandbox ocsf: CONFIG:LOADED [INFO] Acknowledged initial policy revision as loaded [version:1] [hash:ee85e91efa86de46c3624782393974c66347f9819fc8fda3e332a1de2e4a928a]
[1787057223656] INFO sandbox openshell_sandbox: Sidecar process supervisor reported initial process anchor {pid: 18}
[1787057223656] INFO sandbox openshell_supervisor_network::opa: Configured OPA runtime binary identity mode {require_binary_identity: true, source: proto}
[1787057223656] OCSF sandbox ocsf: CONFIG:CONFIGURED [INFO] OPA runtime binary identity mode configured [source:proto require_binary_identity:true]
[1787057223658] INFO sandbox openshell_sandbox: Policy binary symlink resolution complete for sidecar process anchor {pid: 18}
[1787057223660] INFO sandbox openshell_sandbox: Sidecar process supervisor reported entrypoint start {pid: 48, ssh_socket: @openshell-sidecar-ssh}
[1787057223661] INFO sandbox openshell_supervisor_network::opa: Configured OPA runtime binary identity mode {require_binary_identity: true, source: proto}
[1787057223661] OCSF sandbox ocsf: CONFIG:CONFIGURED [INFO] OPA runtime binary identity mode configured [source:proto require_binary_identity:true]
[1787057223663] INFO sandbox openshell_sandbox: Policy binary symlink resolution complete for sidecar process anchor {pid: 48}
[1787057223663] INFO sandbox openshell_sandbox: sidecar supervisor session task spawned
[1787057223668] OCSF sandbox ocsf: NET:OPEN [INFO] openshell.forge-system.svc.cluster.local:8080
[1787057247681] INFO gateway openshell_driver_kubernetes::driver: Fetching sandbox from Kubernetes
[1787057281514] INFO gateway openshell_server::supervisor_session: supervisor session: relay opened successfully
[1787057281514] INFO gateway openshell_server::supervisor_session: relay stream: claimed pending relay, bridging
[1787057281514] INFO gateway openshell_server::grpc::sandbox: ExecSandbox (relay): command started
[1787057281513] OCSF sandbox ocsf: NET:OPEN [INFO] [msg:ssh relay open (channel_id=790ff390-7ed4-4b33-8056-8945b5870977, target=unix:@openshell-sidecar-ssh)]
[1787057282142] INFO gateway openshell_server::supervisor_session: supervisor session: relay opened successfully
[1787057282142] INFO gateway openshell_server::supervisor_session: relay stream: claimed pending relay, bridging
[1787057282142] INFO gateway openshell_server::grpc::sandbox: ExecSandbox (relay): command started
[1787057282209] INFO gateway openshell_server::supervisor_session: supervisor session: relay opened successfully
[1787057282209] INFO gateway openshell_server::supervisor_session: relay stream: claimed pending relay, bridging
[1787057282209] INFO gateway openshell_server::grpc::sandbox: ExecSandbox (relay): command started
[1787057282134] OCSF sandbox ocsf: HTTP:GET [MED] DENIED /usr/local/bin/node(49) -> GET http://forge-analyst-external-vllm-proxy.forge-system.svc.cluster.local:8000/ [policy:- engine:opa] [reason:network connections not allowed by policy]
[1787057282141] OCSF sandbox ocsf: NET:CLOSE [INFO] [msg:ssh relay closed (channel_id=790ff390-7ed4-4b33-8056-8945b5870977, target=unix:@openshell-sidecar-ssh)]
[1787057282142] OCSF sandbox ocsf: NET:OPEN [INFO] [msg:ssh relay open (channel_id=b07798ad-4122-4adf-bdf8-98e85922e3ca, target=unix:@openshell-sidecar-ssh)]
[1787057282201] OCSF sandbox ocsf: HTTP:GET [MED] DENIED /usr/local/bin/node(62) -> GET http://forge-analyst-external-vllm-proxy.forge-system.svc.cluster.local:8000/ [policy:- engine:opa] [reason:network connections not allowed by policy]
[1787057282207] OCSF sandbox ocsf: NET:CLOSE [INFO] [msg:ssh relay closed (channel_id=b07798ad-4122-4adf-bdf8-98e85922e3ca, target=unix:@openshell-sidecar-ssh)]
[1787057282208] OCSF sandbox ocsf: NET:OPEN [INFO] [msg:ssh relay open (channel_id=57284b07-d317-4c9e-8247-bdf3738ac7d9, target=unix:@openshell-sidecar-ssh)]
[1787057282271] OCSF sandbox ocsf: HTTP:GET [MED] DENIED /usr/local/bin/node(75) -> GET http://forge-analyst-external-vllm-proxy.forge-system.svc.cluster.local:8000/ [policy:- engine:opa] [reason:network connections not allowed by policy]
[1787057282278] OCSF sandbox ocsf: NET:CLOSE [INFO] [msg:ssh relay closed (channel_id=57284b07-d317-4c9e-8247-bdf3738ac7d9, target=unix:@openshell-sidecar-ssh)]
[1787057283579] INFO gateway openshell_server::grpc::policy: SubmitPolicyAnalysis: persisted draft chunks
[1787057283575] INFO sandbox openshell_sandbox: Flushed activity summary to gateway {denied_action_count: 3, sandbox_name: ssrf-repro-2, network_activity_count: 3}
[1787057283575] INFO sandbox openshell_sandbox: Flushed denial analysis to gateway {proposals: 1, summaries: 1, sandbox_name: ssrf-repro-2}
[1787057307696] INFO gateway openshell_driver_kubernetes::driver: Fetching sandbox from Kubernetes

    Gateway logs, same window (ANSI stripped):

    2026-08-18T12:46:59.157274Z  INFO ... openshell_server::grpc::sandbox: minted sandbox JWT sandbox_id=caa43e1b-33f3-4791-a627-9074fca0493b
2026-08-18T12:46:59.158482Z  INFO ... openshell_driver_kubernetes::driver: Creating sandbox in Kubernetes sandbox_id=caa43e1b-... sandbox_name=ssrf-repro-2 namespace=forge-system
2026-08-18T12:46:59.161340Z  INFO ... openshell_driver_kubernetes::driver: Resolved namespace annotations for sandbox identity namespace=forge-system uid_range=Some("1000800000/10000") sup_groups=Some("1000800000/10000")
2026-08-18T12:46:59.161355Z  INFO ... openshell_driver_kubernetes::driver: Resolved sandbox identity uid=1000800000 gid=1000800000
2026-08-18T12:46:59.168667Z  INFO ... openshell_driver_kubernetes::driver: Sandbox created in Kubernetes successfully sandbox_id=caa43e1b-... sandbox_name=ssrf-repro-2
2026-08-18T12:46:59.168813Z  INFO ... openshell_server::grpc::sandbox: CreateSandbox request completed successfully sandbox_id=caa43e1b-... sandbox_name=ssrf-repro-2
2026-08-18T12:47:03.547272Z  INFO ... openshell_server::grpc::auth_rpc: issued gateway sandbox JWT sandbox_id=caa43e1b-...
2026-08-18T12:47:03.554287Z  INFO ... openshell_server::grpc::policy: UpdateConfig: backfilled spec.policy from sandbox-discovered policy sandbox_id=caa43e1b-...
2026-08-18T12:47:03.561801Z  INFO ... openshell_server::grpc::policy: GetSandboxProviderEnvironment request completed successfully sandbox_id=caa43e1b-... provider_count=0 env_count=0 provider_env_revision=16248441118584987786
2026-08-18T12:47:03.577706Z  INFO ... openshell_server::grpc::policy: ReportPolicyStatus: sandbox reported policy load result sandbox_id=caa43e1b-... version=1 status=loaded
2026-08-18T12:47:03.666638Z  INFO ... openshell_server::supervisor_session: supervisor session: accepted sandbox_id=caa43e1b-... session_id=54da2574-789f-46d8-8160-8436d7561cb2 instance_id=a77dcd83-6eb3-4de2-a1f4-0d0b423e6fb4
2026-08-18T12:47:27.681340Z  INFO reconcile{...}: openshell_driver_kubernetes::driver: Fetching sandbox from Kubernetes sandbox_id=caa43e1b-... namespace=forge-system
2026-08-18T12:48:01.514050Z  INFO ... openshell_server::supervisor_session: supervisor session: relay opened successfully sandbox_id=caa43e1b-... session_id=54da2574-... channel_id=790ff390-7ed4-4b33-8056-8945b5870977
2026-08-18T12:48:01.514206Z  INFO ... openshell_server::supervisor_session: relay stream: claimed pending relay, bridging channel_id=790ff390-... sandbox_id=caa43e1b-...
2026-08-18T12:48:01.514215Z  INFO openshell_server::grpc::sandbox: ExecSandbox (relay): command started sandbox_id=caa43e1b-... channel_id=790ff390-... command_len=306 stdin_len=0 command_preview=node -e '...forge-analyst-external-vllm-proxy.forge-system.svc.cluster.local:8000/...'
2026-08-18T12:48:02.142623Z  INFO ... supervisor session: relay opened successfully ... channel_id=b07798ad-4122-4adf-bdf8-98e85922e3ca  (attempt 2)
2026-08-18T12:48:02.142810Z  INFO openshell_server::grpc::sandbox: ExecSandbox (relay): command started ... channel_id=b07798ad-...
2026-08-18T12:48:02.209100Z  INFO ... supervisor session: relay opened successfully ... channel_id=57284b07-d317-4c9e-8247-bdf3738ac7d9  (attempt 3)
2026-08-18T12:48:02.209186Z  INFO openshell_server::grpc::sandbox: ExecSandbox (relay): command started ... channel_id=57284b07-...
2026-08-18T12:48:03.579838Z  INFO ... openshell_server::grpc::policy: SubmitPolicyAnalysis: persisted draft chunks sandbox_id=caa43e1b-... accepted=1 rejected=0 draft_version=1 summaries=1
2026-08-18T12:48:27.696788Z  INFO reconcile{...}: Fetching sandbox from Kubernetes sandbox_id=caa43e1b-...
2026-08-18T12:49:27.707946Z  INFO reconcile{...}: Fetching sandbox from Kubernetes sandbox_id=caa43e1b-...
2026-08-18T12:49:33.090497Z  INFO ... openshell_driver_kubernetes::driver: Deleting sandbox from Kubernetes sandbox_id=caa43e1b-... namespace=forge-system
2026-08-18T12:49:33.098346Z  INFO ... openshell_driver_kubernetes::driver: Sandbox deleted from Kubernetes sandbox_id=caa43e1b-...
2026-08-18T12:49:33.125613Z  INFO openshell_server::supervisor_session: supervisor session: expected transport close during teardown sandbox_id=caa43e1b-... error=... "peer closed connection without sending TLS close_notify"
2026-08-18T12:49:33.125684Z  INFO openshell_server::supervisor_session: supervisor session: ended sandbox_id=caa43e1b-... session_id=54da2574-...

    This changes the diagnosis. The sandbox's own OCSF log shows the OPA engine evaluating and logging DENIED on all 3 attempts:

    HTTP:GET [MED] DENIED /usr/local/bin/node(49) -> GET http://forge-analyst-external-vllm-proxy.forge-system.svc.cluster.local:8000/ [policy:- engine:opa] [reason:network connections not allowed by policy]

    But the corresponding ExecSandbox output I posted earlier for these exact same 3 attempts got a real 403 from the target every time — a completed TCP+TLS+HTTP round trip, never ERR/timeout. So this isn't the "approved-rule-never-consulted" or "SSRF-rejection-after-approval" pattern I originally suspected — the policy engine is correctly identifying the connection as disallowed and logging a deny for it, but that decision is not actually being enforced at the network layer on this sidecar topology + process_binary_aware_network_policy config. The log and the real network outcome disagree.

    One more thing worth noting: GetSandboxLogs was the only way to get this — the openshell-supervisor-network sidecar container doesn't emit anything to stdout/stderr, so plain kubectl/oc logs -c openshell-supervisor-network on the sandbox pod returns nothing. If that's expected, it might be worth documenting (or exposing via oc logs too) since it's not discoverable otherwise.

    Happy to pull anything else — pcap from inside the sandbox netns, the OPA policy bundle/decision trace, etc. — if it'd help narrow this down further.

  8. johntmyers commented on Aug 18, 2026

    @johntmyers
    johntmyers
    on Aug 18, 2026
    Collaborator

    @thoraxe Thanks for collecting the sandbox logs. The latest script verifies that an undeclared endpoint is denied, but it stops before exercising the approve/reload/retry workflow described in the original issue.

    Could you extend the script to perform the complete sequence in one run?

    1. Create a sandbox without the target endpoint in its policy.
    2. Attempt the request once.
      • Print the complete HTTP status, headers, and body.
      • Capture the corresponding OCSF decision.
    3. Wait for SubmitPolicyAnalysis.
    4. Call GetDraftPolicy.
      • Record the proposed rule, chunk ID, status, and draft version.
    5. Call ApproveAllDraftChunks.
      • Record the returned policy version and hash.
    6. Wait until that exact policy version reports status=loaded.
    7. Attempt the identical request again.
      • Print the complete HTTP status, headers, and body.
      • Capture the corresponding OCSF decision.
    8. Call GetDraftPolicy again.
      • Compare the chunk IDs and statuses with the pre-approval result.
      • Confirm whether a genuinely new pending rule exists.
    9. If possible, capture a target-side access log for both attempts.

    Please use a unique request path or correlation value for each attempt so the client output, OCSF events, gateway logs, and target logs can be matched unambiguously.

    The current STATUS 403 result is consistent with OpenShell enforcing the pre-approval OPA denial: the forward proxy itself returns HTTP 403 with an "error":"policy_denied" JSON body. Printing the body will distinguish that response from a 403 generated by the vLLM service.

    The complete sequence should tell us:

    • Whether approval changes the OPA decision from denied to allowed.
    • Whether the post-approval request is subsequently rejected by SSRF.
    • Whether the approved policy revision is actually active for the retry.
    • Whether the second SubmitPolicyAnalysis reuses the existing chunk ID or creates a new pending row.
    • Whether either request reaches the upstream service.

    Without the approval, reload wait, and post-approval retry in the same script, the latest reproduction does not exercise the behavior reported in the original issue.

  9. thoraxe commented on Aug 19, 2026

    @thoraxe
    thoraxe
    on Aug 19, 2026
    Author

    Ran the exact sequence you asked for (@johntmyers) — full round trip against a live gateway (v0.0.106, topology = "sidecar"), via the real CreateSandbox/ExecSandbox/GetDraftPolicy/ApproveAllDraftChunks/GetSandboxPolicyStatus RPCs, mTLS + a Keycloak JWT with aud: openshell-cli and the user realm role.

    Setup: CreateSandbox with no spec.policy at all (undeclared endpoint), so the first request is denied outright.

    1. Attempt #1 (unique correlation value): curl from inside the sandbox to a private, undeclared .svc target:

    HTTP_CODE=403
{"detail":"GET forge-analyst-external-vllm-proxy.forge-system.svc:8000/v1/models not permitted by policy","error":"policy_denied"}

    2. GetDraftPolicy (after waiting for the automatic SubmitPolicyAnalysis):

    chunks {
  id: "66d1a2cc-f37f-480a-b6a3-1515e3b2504a"
  status: "pending"
  rule_name: "allow_forge_analyst_external_vllm_proxy_forge_system_svc_8000"
  proposed_rule { endpoints { host: "...svc" port: 8000 } binaries { path: "/usr/bin/curl" } }
  binary: "/usr/bin/curl"
}
draft_version: 1

    3. ApproveAllDraftChunks → policy_version: 2, chunks_approved: 1.

    4. Polled GetSandboxPolicyStatus until status: POLICY_STATUS_LOADED, active_version: 2 — confirmed the approved rule is present verbatim in revision.policy.network_policies.

    5. Attempt #2 (identical request, new unique correlation value):

    HTTP_CODE=403
{"detail":"GET forge-analyst-external-vllm-proxy.forge-system.svc:8000 blocked: internal address","error":"ssrf_denied"}

    Note the error changed from policy_denied to ssrf_denied — the policy-level check now passes; a separate SSRF check on the private/internal-IP destination is what's rejecting the retry.

    6. GetDraftPolicy again: same chunk id (66d1a2cc-...), status now "approved", draft_version still 1 — no new duplicate draft chunk was created on the retry.

    This matches your alternative explanation from the first comment exactly: the approved rule is genuinely consulted and enforced (visible directly in the policy revision, and the denial reason changes), and the retry fails at the orthogonal SSRF layer, not because enforcement ignores the approval. It also doesn't reproduce the "brand-new draft chunk" symptom from the original report — same chunk id and draft_version both times.

    I can't speak to whether this was already true on the v0.0.104/commit c4b500a build the original report used, or whether something changed since — but on current v0.0.106 I cannot reproduce "approval doesn't unblock traffic" as originally reported. Given this, and that the SSRF-rejection explanation is now directly confirmed rather than theorized, I think this can be closed. Thanks for the persistence in tracking down the real mechanism here — happy to re-open with a fresh repro if this surfaces again on a non-private target.

  10. thoraxe commented on Aug 19, 2026

    @thoraxe
    thoraxe
    on Aug 19, 2026
    Author

    Confirmed via live repro above: this doesn't reproduce as reported on v0.0.106. Closing.

  11. thoraxe
    closed this as completedon Aug 19, 2026
  12. purp
    removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions