Arrow Pushdowns fail when *any* column is a string_view #227

Activity

1. paultiq

   added

   needs triage

   on Dec 15, 2025
2. paultiq commented on Dec 15, 2025

   

   paultiq

   on Dec 15, 2025

   Last edited by paultiq

   ContributorAuthor

   More actions

   This is an Arrow issue as shown below, but DuckDB could/should not attempt a pushdown if any column is a string_view (or perhaps any View).

   apache/arrow#43010

   import pyarrow as pa
   import pyarrow.dataset as ds
   
   table = pa.table({"id": [1, 2, 3], "value": ["a", "b", "c"]}).cast(pa.schema([("id", pa.int64()), ("value", pa.string_view())]))
   scanner = ds.dataset(table).scanner(filter=ds.field("id") > 1)
   scanner.to_table()
3. Tishj commented on Dec 18, 2025

   

   Tishj

   on Dec 18, 2025

   Member

   More actions

   This was merged upstream to fix the problem: duckdb/duckdb#20165
4. paultiq commented on Dec 18, 2025

   

   paultiq

   on Dec 18, 2025

   Last edited by paultiq

   ContributorAuthor

   More actions

   That PR doesn't fix this issue. This issue was tested after that was merged.

   That PR handles cases where the expression compares a string_view column.

   This issue occurs when any column is a string_view.

   Example:

   import polars as pl
   import duckdb
   print(duckdb.__version__)
   
   df = pl.DataFrame({"sv_col": ["abc"], "int_col": [1]})
   arrow_table = df.to_arrow(compat_level=pl.CompatLevel.newest())
   
   
   # Passes, because of duckdb/duckdb#20165
   duckdb.execute("select * from arrow_table where sv_col = 'abc'").df()
   
   # Still fails
   duckdb.execute("select * from arrow_table where int_col = 1").df()
5. paultiq commented on Dec 18, 2025

   

   paultiq

   on Dec 18, 2025

   Last edited by paultiq

   ContributorAuthor

   More actions

   In digging into this, and the related Vortex issue, there's a messy set of problems bigger than just this issue:

   Arrow backends (Vortex, Arrow Compute, etc) support different predicates, but arrow.cpp:CanPushdown assumes coverage is universal:

   • Arrow Compute's array_filter doesn't support tables that contain string_views (this issue), but Vortex DataSets do
   • After Map view types when converting arrow_expressions to vortex expressions vortex-data/vortex#5765 is merged, Vortex DataSets will support equality / inequalities on string_views, but Arrow Compute will not
   • Arrow Compute supports comparisons of timestamps, but Vortex DataSets do not Epic: vortex-compute compute functions vortex-data/vortex#5781. This is a new issue that raises an exception in duckdb.

   In other words, duckdb makes the false assumption that all Arrow datasets support the same predicates / comparisons for pushdown.

   Possible Solutions

   • duckdb should be pessimistic, and deny pushdown when any backend doesn't support a particular predicate on a particular datatype
   • duckdb should be optimistic, attempting to pushdown, but duckdb-python would gracefully handle an exception during pushdown - falling back to a no-pushdown case. Currently, duckdb-python will reraise the exception.
   • (too complicated) duckdb would need to add rules to handle backend specific cases, handling (for instance) vortex datasets differently.
   • somehow check at runtime which predicates are supported: this would involve some change in Arrow.
6. danking commented on Jan 8, 2026

   

   danking

   on Jan 8, 2026

   Last edited by danking

   More actions

   FWIW, after noodling on this a bit, I've come to the conclusion that the Arrow Dataset API is pretty awkward. I suppose there's a reasonable argument that Vortex (and other Dataset implementors) should just support every Arrow expression (we could always just evaluate the expression ourselves before yielding each record batch). However, that means that duckdb will push expressions down to us that might be faster/cheaper to evaluate in duckdb. It seems like, at least for Vortex, we should push folks away from the Arrow Dataset API and towards the Vortex duckdb extension.
7. paultiq commented on Jan 8, 2026

   

   paultiq

   on Jan 8, 2026

   Last edited by paultiq

   ContributorAuthor

   More actions

   @danking while I think I agree with your conclusion, I don't think it's related to -this- issue. This issue is arrow compute raising when array_filter is called when there's a string_view in the Arrow table.

   It won't actually raise with a Vortex dataset, which makes it an opposite case where Vortex supports the pushdown where Arrow doesn't.

   Just to be clear: even if users never hit the vortex dataset api, this issue still needs to be resolved for Arrow

   👍React with 👍1Reacted by Dan King
8. evertlammerts commented on Jan 27, 2026

   

   evertlammerts

   on Jan 27, 2026

   Member

   More actions

   This PR adds a config option that gates pushdown on whether the schema contains view types. I'm not sure whether this is the way to go, feedback very welcome.
9. pdet

   added a commit that references this issue on Feb 3, 2026

   Disable pushdown for arrow tables that contain view columns (#20703)

   ...

   584c07d
10. evertlammerts

    closed this as completedon Feb 3, 2026