Skip to content

AVRO-4361: [java] Size collection guards from the writer's schema - #4047

Open
rayokota wants to merge 1 commit into
apache:mainfrom
rayokota:AVRO-collection-guard-writer-schema
Open

rayokota wants to merge 1 commit into
apache:mainfrom
rayokota:AVRO-collection-guard-writer-schema

Conversation

@rayokota

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This PR fixes AVRO-4361: the collection size guard (AVRO-4241, AVRO-4300,
AVRO-4324) rejects valid data when the reader's array element or map value
schema has a larger minimum encoded size than the writer's. Examples are a
field added with a default, int promoted to double, or null read as a
nullable union. The guard computed the minimum from the reader's schema, but
the bytes were written with the writer's schema.

  • Fast reader (FastReaderBuilder.createArrayReader): the writer's array
    schema is available as action.writer, so the guard now uses the writer's
    element type. The guard stays exact.

  • Classic and reflect readers (GenericDatumReader, ReflectDatumReader):
    only the reader's element schema is available while reading. A new
    minBytesToRead(Schema) keeps the existing exact bound
    (minBytesPerElement) when the writer's and reader's schemas are equal. That
    check is computed once per reader and reset by setSchema / setExpected.
    When the schemas differ, it uses a new minResolvedBytesPerElement: the
    minimum size of any writer's schema that resolves to the reader's.

    • A field with a default counts 0.
    • float and double count 1, since they may be promoted from int or long.
    • A union counts the smaller of 1 and its smallest branch.
    • fixed keeps its size.

    This bound is never larger than the exact one, so a type that was capped as
    zero-byte is still capped. And it is 0 whenever a writer's value could be 0
    bytes, so valid data is never rejected. The zero-byte allocation cap is now
    chosen from the same bound, for arrays and maps.

  • When the schemas differ, the classic and reflect guards are looser than the
    exact bound, but still bound a declared count by the bytes remaining.

  • ensureAvailableCollectionBytes(Decoder, long, Schema) is unchanged; its
    Javadoc now says to pass the writer's element schema. A protected
    ensureAvailableCollectionBytes(Decoder, long, int) overload is added for
    ReflectDatumReader.

Verifying this change

New tests, each run on both the fast and classic readers where applicable:

  • TestGenericDatumReader
    • testMinResolvedBytesPerElement: the resolved bound for promotions,
      unions, fixed, and records with defaulted fields.
    • arrayOfRecordsWithAddedDefaultFieldDecodes: the reported case.
    • arrayWithPromotedElementTypeDecodes: int, long and float read as
      double.
    • mapWithPromotedValueTypeDecodes
    • arrayOfNullsReadAsNullableUnionDecodes
    • resolvedArrayRejectsHugeCount: the guard still rejects a huge declared
      count when resolving.
    • equalSchemasKeepExactBound: equal but distinct schema instances keep the
      exact bound.
  • TestReflectDatumReader.read_arrayWithPromotedElementType

Without the fix, the five decode tests fail with the reported
EOFException: Collection claims ... bytes are available.

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

@github-actions github-actions Bot added the Java Pull Requests for Java binding label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Java Pull Requests for Java binding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant