StreamErrors when making PUT requests with cached, persistent connections via Async::HTTP::Faraday::PersistentClients.

using async-http-faraday (but i suspect this lib is more relevant), we are seeing a not-insignificant number of Protocol::HTTP::StreamErrors when making a PUT request to the google sheets API (updating a cell in a spreadsheet) when using cached, persistent connections via Async::HTTP::Faraday::PersistentClients. i was wondering if you might have a suggested direction to solve for it. my current thinking is that critical, relatively-infrequent, non-idempotent requests should perhaps use their own non-persistent client. here is the stack trace we are seeing for the error:

Protocol::HTTP2::StreamError: Stream closed! (Protocol::HTTP2::StreamError)
    from async/http/protocol/http2/response.rb:113:in 'wait'
    from async/http/protocol/http2/response.rb:155:in 'wait'
    from async/http/protocol/http2/client.rb:46:in 'read_response'
    from async/http/protocol/http2/client.rb:36:in 'call'
    from protocol/http/request.rb:87:in 'call'
    from async/http/client.rb:143:in 'make_response'
    from async/http/client.rb:101:in 'call'
    from async/http/faraday/adapter.rb:216:in 'block (2 levels) in Async::HTTP::Faraday::Adapter#perform_request'
    from async/http/faraday/adapter.rb:253:in 'block in Async::HTTP::Faraday::Adapter#with_timeout'
    from async/scheduler.rb:631:in 'with_timeout'
    from async/task.rb:153:in 'with_timeout'
    from async/http/faraday/adapter.rb:252:in 'with_timeout'
    from async/http/faraday/adapter.rb:202:in 'block in Async::HTTP::Faraday::Adapter#perform_request'
    from async/http/faraday/adapter.rb:244:in 'block (2 levels) in Async::HTTP::Faraday::Adapter#with_client'
    from async/http/faraday/clients.rb:107:in 'with_client'
    from async/http/faraday/adapter.rb:243:in 'block in Async::HTTP::Faraday::Adapter#with_client'
    from kernel/sync.rb:25:in 'Sync'
    from async/http/faraday/adapter.rb:233:in 'with_client'
    from async/http/faraday/adapter.rb:176:in 'perform_request'
    from async/http/faraday/adapter.rb:167:in 'call'
    from faraday/follow_redirects/middleware.rb:77:in 'perform_with_redirection'
    from faraday/follow_redirects/middleware.rb:65:in 'call'
    from faraday/middleware.rb:56:in 'call'
    from faraday/rack_builder.rb:153:in 'build_response'
    from faraday/connection.rb:452:in 'run_request'
    from faraday/connection.rb:280:in 'put'

async-http-faraday 0.22.1
async-http 0.94.2

Can you try v0.95.0 and see if it fixes the issue?

it does not 😞

Protocol::HTTP2::StreamError: Stream closed! (Protocol::HTTP2::StreamError)
    from async/http/protocol/http2/response.rb:123:in 'wait'
    from async/http/protocol/http2/response.rb:173:in 'wait'
    from async/http/protocol/http2/client.rb:56:in 'read_response'
    from async/http/protocol/http2/client.rb:41:in 'call'
    from protocol/http/request.rb:87:in 'call'
    from async/http/client.rb:155:in 'make_response'
    from async/http/client.rb:112:in 'call'
    from async/http/faraday/adapter.rb:216:in 'block (2 levels) in Async::HTTP::Faraday::Adapter#perform_request'

@ioquatix

@ioquatix any further thoughts on this?

I will take a look again, thanks for reporting back.

have you had a chance to take a look at this? it's now also an issue with the Hue API's PUT endpoints.

I have released socketry/protocol-http2#29 in v0.26.1 - do you mind capturing the same error so we can see exactly what error code it's failing with?

Protocol error!

Oh no, that's probably the worst case lol

It means something is wrong with my implementation of HTTP/2.

Okay, I need to get more debugging information from you if possible.

bundle open protocol-http2 and enable these lines:

			def read_frame(maximum_frame_size = MAXIMUM_ALLOWED_FRAME_SIZE)
				Console.info(self) {"read_frame: length=#{length} type=#{type} flags=#{flags} stream_id=#{stream_id} -> klass=#{@frames[type].inspect}"}

			def write_frame(frame)
				Console.info(self, name: "write") {frame.inspect}

Enable the Console lines and also change debug to info.

Can you then capture a session that ends in protocol error.

We captured the debugging info you asked for — and I believe the frame traces pinpoint the root cause: Async::HTTP::Client#call's idempotent retry resends a request whose body was already consumed on the first attempt, without rewinding it.

Versions: async-http 0.95.1, protocol-http2 0.26.1, async-http-faraday 0.22.2, async 2.42.0, Ruby 4.0.5. I enabled the framer logging via a Module#prepend shim (at warn level to pass our production log filtering; output is otherwise identical) and additionally logged Stream#close! and Async::Pool::Controller#reuse/#retire, all keyed by framer instance so full sessions are extractable from our log aggregator.

The failing request is a PUT https://api.meethue.com/v2/bridges/.../lights/7/state with a 19-byte JSON body, over a pooled persistent connection. In our fleet, every occurrence of this "Protocol error!" flavor is a PUT.

Attempt 1 — connection A (a warm pooled connection). The request is fully written, but the server sends GOAWAY (max_age connection cycling) and closes without responding to it:

00:28:22 write  #<HeadersFrame stream_id=311 flags=4 104b>
00:28:22 write  #<DataFrame stream_id=311 flags=0 19b>    # Writing #<Protocol::HTTP::Body::Buffered:0x0000ffff52ca4b38>
00:28:22 write  #<DataFrame stream_id=311 flags=1 0b>     # body fully consumed
00:28:22 read   #<PingFrame stream_id=0 flags=0 ...>
00:28:22 write  #<PingFrame stream_id=0 flags=1 ...>
00:28:22 read   #<GoawayFrame stream_id=0 flags=0 payload=[511, 0, "max_age"]>
         # connection closes; stream 311 never receives a response

Attempt 2 — the client transparently retries (PUT is idempotent?) on a brand-new connection. Note the HEADERS go out, but the body write produces only the zero-length END_STREAM frame — and the annotation shows it is the same Buffered object (0x0000ffff52ca4b38), already at EOF:

00:28:22 write  #<SettingsFrame stream_id=0 flags=0 payload=[[2, 0], [5, 1048576], [4, 8388608], [9, 1]]>
00:28:22 read   #<SettingsFrame stream_id=0 flags=0 payload=[[3, 100], [4, 1048576], [6, 65536]]>
00:28:22 write  #<SettingsFrame stream_id=0 flags=1 payload=[]>
00:28:22 read   #<WindowUpdateFrame stream_id=0 flags=0 payload=983041>
00:28:22 read   #<SettingsFrame stream_id=0 flags=1 payload=[]>
00:28:22 write  #<HeadersFrame stream_id=1 flags=4 156b>
00:28:22 write  #<DataFrame stream_id=1 flags=1 0b>       # Writing #<Protocol::HTTP::Body::Buffered:0x0000ffff52ca4b38> — same object, no bytes left
00:28:22 read   #<HeadersFrame stream_id=1 flags=4 73b>
00:28:22 read   #<ResetStreamFrame stream_id=1 flags=0 payload=1>
00:28:22 close! stream_id=1 error_code=1                  # PROTOCOL_ERROR -> StreamError: Protocol error!

So the server received a PUT declaring a content-length, followed by END_STREAM after zero body bytes — a malformed message per RFC 9113 §8.1.1 — and correctly answered with error headers plus RST_STREAM(PROTOCOL_ERROR). The connection itself stayed healthy; it served three more requests normally afterward.

The retry is definitely Client#call's internal one (not our Faraday-level retry, which would rebuild the body from the source string — the identical body object address rules that out). Looking at async-http 0.95.1, the rescue path in Client#call retries idempotent requests but never rewinds request.body; the only rewind call site in the gem is in the redirect middleware.

That likely also explains the original report here: Google's frontends do the same max_age GOAWAY connection cycling as whatever fronts the Hue API, and a PUT (idempotent, body-bearing) is exactly the request shape that hits this. Requests without bodies retry fine, which is why it's sporadic and method-dependent.

Happy to test a fix — this fires ~1.5 times per hour across our fleet, so we can confirm quickly against production traffic.

🤖 Reply via Claude

Ah, that makes sense, it should be a straight forward fix, I'll take a look.

Thanks for the detailed frame trace. That pins down the root cause clearly.

What is happening:

The first PUT attempt sends the full body on a warm HTTP/2 connection.
The server sends GOAWAY for connection cycling before responding to that stream.
protocol-http2 correctly treats streams above last_stream_id as not processed and raises Protocol::HTTP::RefusedError.
async-http retries the request on a new connection, but the original request body has already been consumed.
The retry therefore sends the same content-length but only an empty END_STREAM DATA frame.
The server rejects that malformed request with RST_STREAM(PROTOCOL_ERROR), which is the Protocol error! you observed.

I have split the fix into two PRs:

Add request retry preparation protocol-http#108 adds request-level retry preparation APIs: Request#rewind! for known-unprocessed retries and Request#retry! for method-safe ambiguous retries.
Rewind request bodies before retry #229 uses those APIs before retrying, so consumed request bodies are rewound before being resent.

The async-http PR depends on protocol-http#108 being merged and released first. I tested async-http against the protocol-http PR checkout, including a focused regression test for this body-consumption retry case.

One correction/nuance after looking more carefully at the GOAWAY trace and the protocol-http2 implementation:

The logged GOAWAY payload appears to be [last_stream_id, error_code, debug_data]:

request stream_id=311
GOAWAY payload=[511, 0, "max_age"]

Since 311 <= 511, that stream is not above the GOAWAY last_stream_id. So GOAWAY alone does not prove stream 311 was unprocessed, and protocol-http2 should not convert that stream to RefusedError based on the GOAWAY. I confirmed the implementation uses the right comparison (stream.id > last_stream_id) for marking streams as not processed.

That means the retry in this trace was most likely from the later connection close/EOF path for an idempotent request, not from the GOAWAY unprocessed-stream path. The core bug is still the same: once async-http retries, it must prepare/rewind the request body before resending it. Otherwise the second attempt can send the original content-length with an already-consumed body.

So the current fix direction still stands:

Add request retry preparation protocol-http#108 adds Request#rewind! and Request#retry!.
Rewind request bodies before retry #229 uses rewind! for known-unprocessed retries and retry! for ambiguous idempotent retries.

While checking this, I noticed a small protocol-http2 correctness issue: GOAWAY last_stream_id is a 31-bit field and the reserved high bit should be ignored when decoding. I opened socketry/protocol-http2#30 for that. It does not affect the logged 511 case, but it is the correct RFC behavior.

Fixed in release v0.98.0 - could you please try it?

0.98.1 fixes the issue, thank you 🥳

we're also getting some "Internal error!"s. do you think those might be unavoidable and deserve retries, or should we open a new issue about that?

Open a new issue with the same debugging we did here, it will be helpful.

Followed up on the "Internal error!"s in #231 — turned out not to be an async-http fault: Facebook sends complete 200 headers and then resets the stream before any DATA frame. Framed there as a question about whether the existing retry mechanism should cover that case.

🤖 Reply via Claude

Navigation Menu

Repository navigation

`StreamErrors` when making PUT requests with cached, persistent connections via `Async::HTTP::Faraday::PersistentClients`. #220

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

StreamErrors when making PUT requests with cached, persistent connections via Async::HTTP::Faraday::PersistentClients. #220

Description

Activity

ioquatix commented on Apr 11, 2026

jakeonfire commented on Apr 15, 2026

jakeonfire commented on Apr 23, 2026

ioquatix commented on Apr 24, 2026

jakeonfire commented on Jul 15, 2026

ioquatix commented on Jul 19, 2026

jakeonfire commented on Jul 21, 2026

ioquatix commented on Jul 21, 2026

ioquatix commented on Jul 21, 2026

jakeonfire commented on Jul 24, 2026

ioquatix commented on Jul 24, 2026

ioquatix commented on Jul 24, 2026

ioquatix commented on Jul 24, 2026

ioquatix commented on Jul 24, 2026

jakeonfire commented on Jul 28, 2026

ioquatix commented on Jul 28, 2026

jakeonfire commented on Jul 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`StreamErrors` when making PUT requests with cached, persistent connections via `Async::HTTP::Faraday::PersistentClients`. #220