Repository navigation
fix: close silently dead http2 edge connections with ping health checks - #1773
jyong-jobber wants to merge 1 commit into
Conversation
The http2 transport serves edge connections with an http2.Server that has no health check. If a connection dies without the socket reporting an error (for example across laptop sleep, or behind a proxy that keeps the local socket open), the frame reader blocks in a read forever. cloudflared keeps reporting the connection as ready and never redials it, while the edge has already dropped it and returns error 1033. h2mux had heartbeats (--heartbeat-interval/--heartbeat-count) for this, but they were lost when h2mux was removed and http2 never gained an equivalent. Enable the HTTP/2 PING health check: after 30s without receiving a frame a PING is sent, and the connection is closed if it is not acknowledged within 15s, so the supervisor reconnects it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Amplify 3.3.0 <amplify@getjobber.com>
auditbenchai
left a comment
There was a problem hiding this comment.
No actionable comments were generated in this review. 🎉
Merge risk: 🟢 Low · up to ed63280
Connections on high-latency or temporarily sluggish networks might experience premature timeouts if 15 seconds is insufficient for ping ACKs, though default values are standard.
Merge readiness: 🟡 Ready to merge, with notes
Pre-merge checks: ✅ 9 · ⚠️ 1
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| CI checks | No CI checks ran on this commit. | Add CI so every PR is built and tested. |
✅ Passed checks: 9 passed
| Check name | Status | Explanation |
|---|---|---|
| Merge conflicts | ✅ | Merges cleanly into master. |
| Secrets | ✅ | No keys, passwords or tokens in the changed code. |
| Blocking issues | ✅ | No critical or high-severity issues in the changed lines. |
| Tests | ✅ | Tests were changed alongside the code. |
| PR size | ✅ | 98 lines changed. |
| Title check | ✅ | The title clearly and concisely describes the fix applied to HTTP/2 edge connections. |
| Description check | ✅ | The description thoroughly details the problem, proposed solution, testing methodology, and timeout values. |
| Linked issues check | ✅ | The PR links to #1772 and directly addresses the problem of hung HTTP/2 edge connections described. |
| Out of scope changes check | ✅ | All changes are strictly scoped to introducing and testing the HTTP/2 ping health checks. |
📝 Walkthrough
This pull request configures HTTP/2 PING health checks on edge connections to detect and close silently dead connections, such as those stalled across laptop sleep or behind corporate proxies. When an idle connection misses ping acknowledgments, it will now terminate and trigger cloudflared's reconnection logic instead of hanging indefinitely. Unit tests are added to verify that unresponsive connections are terminated and responsive ones remain active.
| File | Changes |
|---|---|
connection/http2.go |
Configured SendPingTimeout and PingTimeout via http.HTTP2Config on ServeConnOpts for edge connections. |
connection/http2_test.go |
Added unit tests verifying that edge connections close when unprompted PINGs are unacknowledged and stay open when acknowledged. |
ℹ️ Review info
- Commit:
ed63280 - Verdict: ✅ Pass
- Findings: none
- Review: local checks plus AI review of the changed lines
- Scope: Reviewed 2/2 files (of 2 total) — 0 from cache, 0 needed no AI review. Found 0 finding(s), 0 potential secret(s).
💬 Reply to any inline comment and Audit Bench Ai will recheck that code and answer in the thread. Mention @auditbench in the conversation to ask about the whole PR.
🛠️ Open the full report and fix with AI
Thanks for using Audit Bench Ai! Free pull request reviews for open source: paste any public PR at auditbenchai.com/scan.
Reviewed by Audit Bench Ai · AI code review for security, logic and performance · Review any public PR free
Fixes #1772
Problem
With
--protocol http2, an edge connection can die without the local socket reporting an error. This happens across laptop sleep, or behind a corporate proxy that keeps the local socket open. When it does,cloudflarednever notices:/readykeeps reportingreadyConnections: 4, andcloudflared_tunnel_ha_connectionsstays at 4.cloudflared_tunnel_tunnel_register_successnever increases, because nothing reconnects.cloudflaredis restarted. I observed it stuck for over 3 hours across several wake cycles.A goroutine dump (
/debug/pprof/goroutine?debug=2) taken while it was stuck shows each connection's frame reader blocked in a read, with nothing to time it out:HTTP2Connectionbuilds itshttp2.Serverwith onlyMaxConcurrentStreams, so no liveness check runs. h2mux used to provide one (--heartbeat-intervaland--heartbeat-count), but those flags have been unused since h2mux was removed in TUN-7252, and http2 never got an equivalent. Server-side HTTP/2 pings were added togolang.org/x/netin May 2024 (golang/go#67812), after the http2 transport was written.Change
Enable the HTTP/2 PING health check on edge connections, through
http.HTTP2Configpassed viaServeConnOpts.BaseConfig:SendPingTimeout: 30s: after 30s without receiving a frame, send a PING.PingTimeout: 15s: if it isn't acknowledged within 15s, close the connection.Servethen returnserrEdgeConnectionClosed, and the supervisor reconnects as it does for any lost connection.Note: above values can be changed as needed if they are too aggressive.
A healthy idle connection costs one PING and one ACK every 30 seconds. The timeouts are fields on
HTTP2Connectionso the tests can shorten them. I'm happy to move them to flags, or to wire them to the dormant--heartbeat-*flags, if you'd prefer.Testing
TestHTTP2ClosesUnresponsiveEdgeConnection: a fake edge completes the handshake, then never acknowledges PINGs. The connection is closed witherrEdgeConnectionClosed. Without this change it stays open, and the test fails after 5s.TestHTTP2KeepsResponsiveEdgeConnection: a fake edge acknowledges PINGs, and the connection stays open across several health checks.go test ./connection/: passes. The two new tests also pass three runs with-race.Lost connection with the edgefollowed by 4 newRegistered tunnel connectionlines.🤖 Generated with Claude Code
Co-Authored-By: Amplify 3.3.0 amplify@getjobber.com