Investigate process-agent throughput degradation on high-density Kubernetes nodes #354

Activity

1. LouisLotter

   added

   kind/bugSomething isn't workingSomething isn't working

   squad/marvin

   on Oct 23, 2026
2. dottorblaster commented on Oct 23, 2026

   

   dottorblaster

   on Oct 23, 2026

   Member

   More actions

   Source-level analysis of process-agent CPU/tracer cost (from a separate investigation)

   Context: this came out of a different case (reported high CPU from the SUSE Observability agent), not the throughput regression in this issue. The attached diagnostics for that case turned out to be a backend storage/network benchmark with no agent telemetry, so I went through stackstate-process-agent (HEAD 6fb3bf18) and the 7.78.x-based stackstate-agent tree instead. Several of the findings below are directly relevant to the >3,000-connection / throughput problem, in particular the default-on protocol classification path that this issue already flags as the main lead. No claim here that CPU and throughput share a root cause — but the same code paths are the suspects, and the instrumentation below will settle it.

   What the tracer is actually configured to do by default

   From config/config.go (NewDefaultAgentConfig) and config/tracer_config.go (TracerConfig):

   Setting	Default	Note
   protocol_inspection_enabled → ProtocolClassificationEnabled	true	confirms the classification path is on by default, including for unrecognized TCP payloads
   EnableHTTPMonitoring / EnableHTTP2Monitoring	true
   EnableMongoMonitoring / EnableAMQPMonitoring / EnablePostgresMonitoring	true
   EnableNativeTLSMonitoring	true	uprobes on OpenSSL/GnuTLS/BoringSSL
   EnableKafkaMonitoring	false	hard-coded off
   MaxTrackedHTTPConnections	1024	relevant if a large share of the >3,000 connections are HTTP
   HTTPNotificationThreshold	512
   HTTPMapCleanerInterval / HTTPIdleConnectionTTL	300 s / 30 s
   EnableGatewayLookup	true	per-connection route/gateway resolution
   EnableConntrack / EnableEbpfConntracker	true / true
   EnableConntrackAllNamespaces	true	conntrack across all netns
   ConntrackMaxStateSize / ConntrackRateLimit	131072 / 500 per 3 s
   MaxTrackedConnections	10000	3,000 connections is ~30 % of the map, not a hard cap
   MaxConnectionsStateBuffered / MaxClosedConnectionsBuffered	75000 / 10000
   TCPConnTimeout / ClientStateExpiry	2 min / 2 min
   EnableShortLivedNetworkRelationFilter	true, 60 s	connections shorter than 60 s are filtered after the fact, they are still observed
   EnableMonotonicCount	true
   EnableRootNetNs	true

   Process check (checks/process.go): runs every 30 s (default), enumerates every PID via process.AllProcesses(), scrubs command lines, and calls the shared container provider with a hard 2*time.Second timeout; it emits an increment every cycle and a full snapshot every 60 s (IncrementalPublishingRefreshInterval). In the node agent, if process/container collection is enabled there instead, the intervals are 10 s (2 s realtime) for both.

   Concrete leads worth testing against the reproduction

   1. Protocol classification of unrecognized payloads (ProtocolClassificationEnabled=true). This matches the lead already in the issue. Test with network_tracer_config.protocol_inspection_enabled: false and, separately, disabled_protocols: ["http","http2","mongo","amqp","postgres","tls"] to see whether the cost is classification in general or a specific parser.
   2. MaxTrackedHTTPConnections: 1024. On a node with >3,000 connections, if a large fraction are HTTP/HTTP2, the HTTP map is permanently saturated: notification/eviction churn every cleaner interval. Worth instrumenting before/after raising it to e.g. 4096 in a canary build.
   3. EnableGatewayLookup: true. Route-table lookups per connection; this is a known CPU-heavy path on hosts with large routing tables / many interfaces (bonded 100 GbE setups are exactly where this shows up). A/B it independently of protocol inspection.
   4. Conntrack: EnableConntrackAllNamespaces=true + ConntrackRateLimit=500/3s. Under high connection churn, registrations can hit the rate limit and fall back, adding work. Compare conntrack -S / nf_conntrack_count with the agent on/off.
   5. Per-cycle volume, not per-connection: 30 s process cycle × full container enumeration (2 s timeout) + 60 s full snapshot is a fixed cost that grows with pod/process count, independent of the network regression.

   Instrumentation (already built in — no collection scripts needed)

   The process agent always serves pprof/expvar on :6062 (cmd/agent/main_common.go, profile_process_agent.sh):

   kubectl -n <ns> exec <node-agent-pod> -c process-agent -- \
     curl -sS 'http://localhost:6062/debug/pprof/profile?seconds=30' -o /tmp/pa-cpu.pprof
   go tool pprof -top -nodecount=40 /tmp/pa-cpu.pprof

   Look for pkg/network/tracer, cilium/ebpf, protocols/*, conntrack, gopsutil/process, containers, encoding/json. A 30 s profile taken during the iperf/throughput run is the single most decisive artifact — it will show whether time is in the eBPF datapath, protocol classification, conntrack, or userspace collection/serialization.

   For the rest of the agent stack:

   # node agent: flare + CPU/heap profile (profiles core, process, trace, security, system-probe sub-agents)
   kubectl -n <ns> exec <node-agent-pod> -c node-agent -- agent flare --profile 30 --local /tmp/flare.zip
   
   # cluster agent
   kubectl -n <ns> exec <cluster-agent-pod> -- datadog-cluster-agent flare --profile 30 --local /tmp/dca-flare.zip
   
   # per-check cost (AverageExecutionTime) and effective config
   kubectl -n <ns> exec <pod> -- agent status
   kubectl -n <ns> exec <pod> -- agent config get <key>

   agent status reports AverageExecutionTime per check (pkg/status/types.go, pkg/collector/check/stats/stats.go), which separates “one expensive check” from “many cheap ones”. For JMX, agent jmx list collected shows the effective bean/attribute count.

   Ranked suspects for agent-side CPU (same paths that matter for throughput)

   #	Suspect	Source evidence	Confirm with	Mitigation
   1	eBPF tracer: protocol classification / HTTP map saturation / gateway lookup	config/tracer_config.go:53-70; MaxTrackedHTTPConnections: 1024, EnableGatewayLookup: true	:6062 pprof during load; A/B each flag	protocol_inspection_enabled: false, disabled_protocols, raise HTTP map cap
   2	Process/container collection volume	checks/process.go:120,165,186; 30 s cycle, 60 s snapshot, 2 s container timeout	pprof (gopsutil/process, containers); collected N processes and M containers in …	raise intervals; disable process or container collection independently
   3	Log level / payload logging	log_payloads default false; guarded hot-path logs in check_handler_transactional.go:136-180	agent config get log_level, log_payloads; pprof (encoding/json, log I/O)	DD_LOG_LEVEL=INFO, log_payloads: false
   4	Cluster-agent topology snapshots	every component is JSON-marshalled at least twice (TRACE in collectors, DEBUG in kubernetes_topology_config.go:122) plus payload; secrets SHA-256 + x509 parse per snapshot	pprof (topologycollectors, encoding/json); #secrets/#configmaps	disable unneeded resources.*, lower snapshot rate
   5	JMX / Python checks	JMX-Fetch is a JVM inside the agent; 25 % RAM default, 3 restarts, 3-thread pool	ps shows java; agent jmx list collected; per-check time	narrow bean scope; move JMX to its own container
   6	Disk/container corechecks on dense nodes	diskv2 enumerates mounts per cycle; cgroup reads per container	pprof (corechecks/system/disk, containers)	exclude CSI mounts; raise interval

   Not a suspect in this build: GOMAXPROCS vs CPU limit. Go 1.26 sizes GOMAXPROCS from the cgroup quota and the agent also calls automaxprocs (pkg/runtime/runtime.go); only an explicit GOMAXPROCS env override breaks it.

   Suggested measurement order for the reproduction

   1. Baseline agent-on, 30 s pprof on the process-agent container + agent status + host softirq/system CPU split.
   2. protocol_inspection_enabled=false only → re-measure. This isolates the issue's stated lead without touching anything else.
   3. Re-enable, then EnableGatewayLookup=false in a canary build (config-only via a patched image, since it is not exposed as a chart value).
   4. Raise MaxTrackedHTTPConnections in a canary if step 1 shows HTTP map eviction/notification churn.
   5. Only then disable process/container collection, to separate network-datapath cost from userspace collection cost.
   6. Measure per-NIC counters, retransmits, nf_conntrack_count, and CFS throttling at each step, so a throughput change can be attributed rather than inferred.

   Happy to turn steps 1–6 into a scripted canary runbook if useful. Note: the original CPU case also had a backend I/O benchmark attached (HDFS/Kafka dd/nc/producer-perf) that showed a ~4.3–4.7× buffered-vs-direct disk ratio and a ~35 % cross-node Kafka leader throughput drop; that data is de-identified here and can be shared separately if it is relevant to this issue.
3. LouisParkin commented on Oct 23, 2026

   

   LouisParkin

   on Oct 23, 2026

   More actions

   Reproduced locally with the customer's 2.10.0 process-agent image (stackstate-k8s-process-agent:4898d630). The chart 1.3.22 DaemonSet spec was run under Docker, measuring TCP bulk throughput between two container netns over veth.

   State	0 conns	3000 idle	3000 active
   Agent stopped	41.8	42.4	43.5 Gbps
   Chart defaults	26.7	26.3	26.7
   protocolInspection.enabled=false	41.7	41.7	42.7
   httpTracing.enabled=false			25.8
   only http protocol enabled			27.8

   • Cause: USM protocol classification. socket__classifier_entry (~1 µs) and the net_dev_queue tracepoint (~0.8 µs) run on every transmitted packet. On top of that, socket__protocol_dispatcher runs per tap. protocol_classifier_entrypoint never records a negative result, so unknown-protocol flows (iperf, storage, replication) are re-classified on every packet for their whole lifetime.
   • Density: Idle connections don't matter. Active connections raise node-wide BPF CPU from ~0.7 to ~3.2 cores at ~150k pps background, and per-run cost doubles. That cost lands in softirq/sender context, not the process-agent cgroup (~12% CPU), so container metrics hide it.
   • Mitigation: nodeAgent.protocolInspection.enabled=false recovers throughput to within ~2% of agent-off. Disabling HTTP tracing or individual protocols does not help.
   • Fix direction: cache a negative classification per connection, or cap classification attempts, so unclassified flows stop paying per-packet cost.

   Side findings:

   • Start() doesn't seed NetNsMonitor pids, so exec/exit churn repeatedly drops and re-attaches netns dispatchers.
   • The image HEALTHCHECK (process-agent health) starts a full agent that never exits. This is Docker-only: k8s ignores it.

   Details: official-plans/process-agent-throughput-2026-10-23/README.md in mission-control. Caveat: this was measured on a veth path, not the 2×100G bond, so the size of the effect on the customer's NICs needs their paired A/B measurement.
4. LouisParkin commented on Oct 26, 2026

   

   LouisParkin

   on Oct 26, 2026

   More actions

   Design notes for the classification cap, following Bram's three points (upstream, false negatives, in-flight connections).

   Upstream DataDog: main has no negative result or attempt cap for flows whose protocol is never recognised. DataDog/datadog-agent#56780 (merged 2026-10-01, from a customer performance report) stops per-packet re-classification of long-lived TLS flows and moves early exits ahead of the payload copy. It does not cover unknown protocols, but is worth backporting. Earlier TLS-only attempts DataDog/datadog-agent#52578 and DataDog/datadog-agent#52581 were closed unmerged.

   False negatives: classifiers are stateless per packet and look at the first CLASSIFICATION_MAX_BUFFER (24) bytes of the current segment. The budget should therefore count data segments, not bytes: what matters is how many segments pass before one starts on a message boundary.

   • HTTP, Postgres, MySQL, AMQP, Redis, Kafka and TLS records can be recognised mid-stream at a message boundary.
   • HTTP/2 is only recognised from the preface or server SETTINGS frame, so mid-stream HTTP/2 is never classified today.
   • Mongo already caps attempts per connection (MONGO_MAX_CLASSIFICATION_TRIES, mongo_connection_classification_tries LRU map) to avoid false positives — an in-tree precedent.

   In-flight connections: initial_tcp_seq records whether the handshake was seen, but the socket-filter classifier cannot read it. Options: record SYN sightings in the socket filter; use the small budget only when the first observed data segment starts the stream; or use one larger budget for all.

   Proposal

   • Per-connection attempt counter alongside the existing protocol state (protocol_stack_wrapper_t or a separate LRU map); never in protocol_stack_t, which is embedded in conn_stats_t.
   • On exhaustion set a "classification gave up" flag (bit 7 is free) so later packets skip classification.
   • Check it in both protocol_classifier_entrypoint and the dispatcher; both re-classify unknown flows per packet.
   • Check before usm_context_init/payload copy, or the saving is lost.
   • Separate, configurable budgets for handshake-seen and in-flight connections.
   • Telemetry: give-ups per handshake state, so false negatives are debuggable.
   • New entries for unknown flows grow connection_protocol; the stale-entry GC must cover them.

   Validation: rerun the local throughput reproduction; USM classification tests; a false-negative harness with real HTTP/Postgres/Kafka/Redis clients started mid-stream, measuring segments-to-classification per protocol.
5. craffit

   assigned

   LouisParkin

   on Oct 26, 2026
6. LouisParkin commented on Oct 27, 2026

   

   LouisParkin

   on Oct 27, 2026

   More actions

   Proposed fix: StackVista/datadog-agent-upstream-for-process-agent#854 and #376.

   Each connection now has a budget of classification attempts: 64 packet inspections when the handshake was seen, 1024 when we joined mid-stream. A connection that uses up its budget skips classification for 5 minutes.

   Local reproduction (iperf3, 3000 active connections, Gbps):

   Agent off	Before	With cap
   42	27.5	31–33

   • In-kernel classification time dropped from 49 s to 21 s per 15 s run.
   • The remaining gap is the fixed per-packet hook cost, which this change does not address.
   • In telemetry from the live agent, every connection that was classified needed at most 15 inspections.
   • A new regression test checks that a protocol appearing within the budget is still classified and one appearing after it is not.

   This lowers the overhead; it does not remove it. For throughput benchmarking, the main advice still stands: use representative load, or disable protocol inspection.
7. LouisParkin

   closed this as completedin #376on Oct 30, 2026