Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces client-side EOF tracking in google-cloud-storage to prevent infinite read loops on doubly-gzipped files, alongside corresponding test updates. It also configures 100% code coverage requirements and updates the Nox sessions for google-crc32c. Feedback on the changes points out an issue in the check session of noxfile.py, where running coverage against the installed wheel instead of the source directory will result in 0% coverage and cause the coverage check to fail; it is recommended to remove coverage flags from this session.
ohmayr
force-pushed
the
fix-storage-doubly-gzipped-eof
branch
from
September 20, 2026 09:08
6f60e2a to
6891704
Compare
…18423) Fixes #18423. When reading doubly-gzipped blobs (or objects with Content-Encoding: gzip), out-of-bounds byte range requests may return full payload bytes instead of HTTP 416 RequestRangeNotSatisfiable. This change adds client-side _eof tracking to BlobReader. When a range download returns fewer bytes than requested (or raises 416), BlobReader sets _eof = True so subsequent read() calls immediately return b"" without initiating redundant HTTP range requests.
ohmayr
force-pushed
the
fix-storage-doubly-gzipped-eof
branch
from
September 20, 2026 09:14
6891704 to
94c3c1b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When reading doubly-gzipped blobs (or objects stored with
Content-Encoding: gzip), GCS decompressive transcoding can cause out-of-bounds byte range requests to returnHTTP 200 OKwith full body payload instead ofHTTP 416 RequestRangeNotSatisfiable. BecauseBlobReaderpreviously relied exclusively on catching HTTP 416 to recognize EOF, out-of-bounds reads resulted in infinite read loops.Fix
Added client-side
self._eoftracking toBlobReader:len(downloaded) < fetch_end - fetch_start), or raisesRequestRangeNotSatisfiable,BlobReadersetsself._eof = True.read()calls returnb""immediately without making unnecessary HTTP requests.seek()resetsself._eof = Falseto support repositioning and re-reading.Verification
test_read_doubly_gzipped_eofinpackages/google-cloud-storage/tests/unit/test_fileio.py.pytest: 39/39 unit tests pass.gs://apache-beam-samplesacross all 9 test files from Issue google-cloud-storage: Failed to handle doubly-gzipped files #18423 in bothraw_download=Falseandraw_download=Truemodes.Fixes #18423