Skip to content

fix: close silently dead http2 edge connections with ping health checks - #1773

Open
jyong-jobber wants to merge 1 commit into
cloudflare:masterfrom
jyong-jobber:http2-ping-health-check
Open

jyong-jobber wants to merge 1 commit into
cloudflare:masterfrom
jyong-jobber:http2-ping-health-check

Conversation

@jyong-jobber

@jyong-jobber jyong-jobber commented Oct 19, 2026 •

Copy link
Copy Markdown

Fixes #1772

Problem

With --protocol http2, an edge connection can die without the local socket reporting an error. This happens across laptop sleep, or behind a corporate proxy that keeps the local socket open. When it does, cloudflared never notices:

  • /ready keeps reporting readyConnections: 4, and cloudflared_tunnel_ha_connections stays at 4.
  • cloudflared_tunnel_tunnel_register_success never increases, because nothing reconnects.
  • Visitors get error 1033 until cloudflared is restarted. I observed it stuck for over 3 hours across several wake cycles.

A goroutine dump (/debug/pprof/goroutine?debug=2) taken while it was stuck shows each connection's frame reader blocked in a read, with nothing to time it out:

goroutine 194 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait
crypto/tls.(*Conn).Read
net/http/internal/http2.(*serverConn).readFrames
created by net/http/internal/http2.(*serverConn).serve
  ... github.com/cloudflare/cloudflared/connection.(*HTTP2Connection).Serve  connection/http2.go:83

HTTP2Connection builds its http2.Server with only MaxConcurrentStreams, so no liveness check runs. h2mux used to provide one (--heartbeat-interval and --heartbeat-count), but those flags have been unused since h2mux was removed in TUN-7252, and http2 never got an equivalent. Server-side HTTP/2 pings were added to golang.org/x/net in May 2024 (golang/go#67812), after the http2 transport was written.

Change

Enable the HTTP/2 PING health check on edge connections, through http.HTTP2Config passed via ServeConnOpts.BaseConfig:

  • SendPingTimeout: 30s: after 30s without receiving a frame, send a PING.
  • PingTimeout: 15s: if it isn't acknowledged within 15s, close the connection. Serve then returns errEdgeConnectionClosed, and the supervisor reconnects as it does for any lost connection.
    Note: above values can be changed as needed if they are too aggressive.

A healthy idle connection costs one PING and one ACK every 30 seconds. The timeouts are fields on HTTP2Connection so the tests can shorten them. I'm happy to move them to flags, or to wire them to the dormant --heartbeat-* flags, if you'd prefer.

Testing

  • New TestHTTP2ClosesUnresponsiveEdgeConnection: a fake edge completes the handshake, then never acknowledges PINGs. The connection is closed with errEdgeConnectionClosed. Without this change it stays open, and the test fails after 5s.
  • New TestHTTP2KeepsResponsiveEdgeConnection: a fake edge acknowledges PINGs, and the connection stays open across several health checks.
  • go test ./connection/: passes. The two new tests also pass three runs with -race.
  • Manual, on macOS (2026.9.1): an unpatched build stayed stuck on error 1033 after every sleep tested, until restarted. A build with the same ping settings recovered by itself after sleeps of 2 and 34 minutes, logging Lost connection with the edge followed by 4 new Registered tunnel connection lines.

🤖 Generated with Claude Code
Co-Authored-By: Amplify 3.3.0 amplify@getjobber.com

The http2 transport serves edge connections with an http2.Server that has
no health check. If a connection dies without the socket reporting an
error (for example across laptop sleep, or behind a proxy that keeps the
local socket open), the frame reader blocks in a read forever. cloudflared
keeps reporting the connection as ready and never redials it, while the
edge has already dropped it and returns error 1033.

h2mux had heartbeats (--heartbeat-interval/--heartbeat-count) for this,
but they were lost when h2mux was removed and http2 never gained an
equivalent. Enable the HTTP/2 PING health check: after 30s without
receiving a frame a PING is sent, and the connection is closed if it is
not acknowledged within 15s, so the supervisor reconnects it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Amplify 3.3.0 <amplify@getjobber.com>

@auditbenchai auditbenchai left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open in Audit Bench Ai

No actionable comments were generated in this review. 🎉

Merge risk: 🟢 Low · up to ed63280

Connections on high-latency or temporarily sluggish networks might experience premature timeouts if 15 seconds is insufficient for ping ACKs, though default values are standard.

Merge readiness: 🟡 Ready to merge, with notes

Pre-merge checks: ✅ 9 · ⚠️ 1

⚠️ Checks to review: 1 warning

Check name Status Explanation Resolution
CI checks ⚠️ No CI checks ran on this commit. Add CI so every PR is built and tested.
✅ Passed checks: 9 passed
Check name Status Explanation
Merge conflicts ✅ Merges cleanly into master.
Secrets ✅ No keys, passwords or tokens in the changed code.
Blocking issues ✅ No critical or high-severity issues in the changed lines.
Tests ✅ Tests were changed alongside the code.
PR size ✅ 98 lines changed.
Title check ✅ The title clearly and concisely describes the fix applied to HTTP/2 edge connections.
Description check ✅ The description thoroughly details the problem, proposed solution, testing methodology, and timeout values.
Linked issues check ✅ The PR links to #1772 and directly addresses the problem of hung HTTP/2 edge connections described.
Out of scope changes check ✅ All changes are strictly scoped to introducing and testing the HTTP/2 ping health checks.
📝 Walkthrough

This pull request configures HTTP/2 PING health checks on edge connections to detect and close silently dead connections, such as those stalled across laptop sleep or behind corporate proxies. When an idle connection misses ping acknowledgments, it will now terminate and trigger cloudflared's reconnection logic instead of hanging indefinitely. Unit tests are added to verify that unresponsive connections are terminated and responsive ones remain active.

File Changes
connection/http2.go Configured SendPingTimeout and PingTimeout via http.HTTP2Config on ServeConnOpts for edge connections.
connection/http2_test.go Added unit tests verifying that edge connections close when unprompted PINGs are unacknowledged and stay open when acknowledged.
ℹ️ Review info
  • Commit: ed63280
  • Verdict: ✅ Pass
  • Findings: none
  • Review: local checks plus AI review of the changed lines
  • Scope: Reviewed 2/2 files (of 2 total) — 0 from cache, 0 needed no AI review. Found 0 finding(s), 0 potential secret(s).

💬 Reply to any inline comment and Audit Bench Ai will recheck that code and answer in the thread. Mention @auditbench in the conversation to ask about the whole PR.

🛠️ Open the full report and fix with AI

Thanks for using Audit Bench Ai! Free pull request reviews for open source: paste any public PR at auditbenchai.com/scan.

Reviewed by Audit Bench Ai · AI code review for security, logic and performance · Review any public PR free

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 http2: edge connections that die silently are never redialed; /ready still reports them as healthy

2 participants