Repository navigation
Null map<K, struct<…>> is read back as an empty map #4114
Description
Apache Iceberg version
0.11.1 (also present in 0.12.0 and on main)
Please describe the bug 🐞
ArrowProjectionVisitor.map() rebuilds a map whose values are structs with pa.MapArray.from_arrays(map_array.offsets, key_result, value_result, arrow_field). The offsets array carries no nulls, so the map's validity bitmap is dropped and a null map reads back as an empty one. Maps with non-struct values go through map_array.cast() and keep their nulls. This is the map counterpart of #3833; #4065 fixes list() only.
import pyarrow as pa
from pyiceberg.io.pyarrow import _to_requested_schema
from pyiceberg.schema import Schema
from pyiceberg.types import MapType, NestedField, StringType, StructType
value = StructType(NestedField(3, "source", StringType(), required=False))
schema = Schema(
NestedField(1, "m", MapType(4, StringType(), 5, value, value_required=False), required=False)
)
source = pa.struct([("source", pa.string())])
batch = pa.record_batch({"m": pa.array([[("d", {"source": "x"})], None], type=pa.map_(pa.string(), source))})
print(_to_requested_schema(schema, schema, batch).column(0).to_pylist())
# [[('d', {'source': 'x'})], []] expected: [[('d', {'source': 'x'})], None]Keeping the original validity and offsets buffers, as #4065 does for lists, fixes it:
projected = pa.MapArray.from_arrays(map_array.offsets, key_result, value_result, arrow_field)
return pa.Array.from_buffers(
projected.type, len(map_array), map_array.buffers()[:2],
map_array.null_count, map_array.offset, [projected.values],
)🧑💻🤖 — posted via Claude Code
Activity