Repository navigation
polars.LazyFrame don't work with from_arrow, even tough it work with raw sql query #308
Description
What happens?
ouput of the code example:
PS C:\Users\stett\Documents\python\pql> uv run t.py
Query (sql on LazyFrame):
WITH lf AS (SELECT * FROM arrow_scan(0x15c661746e0, 0x7ffbf06c5ab0, 0x7ffbf06c5420))SELECT * FROM lf
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1 ┆ x │
│ 2 ┆ y │
│ 3 ┆ z │
└─────┴─────┘
Query (sql on DataFrame):
WITH df AS (SELECT * FROM arrow_scan(0x15c66515a90, 0x7ffbf06c5ab0, 0x7ffbf06c5420))SELECT * FROM df
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1 ┆ x │
│ 2 ┆ y │
│ 3 ┆ z │
└─────┴─────┘
Query (from Arrow on DataFrame):
SELECT * FROM arrow_scan(0x15c67bb85e0, 0x7ffbf06c5ab0, 0x7ffbf06c5420) AS arrow_object_8756061e3d8cb73a
shape: (3, 2)
┌─────┬─────┐
│ a ┆ b │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1 ┆ x │
│ 2 ┆ y │
│ 3 ┆ z │
└─────┴─────┘
Traceback (most recent call last):
File "C:\Users\stett\Documents\python\pql\t.py", line 19, in <module>
_qry_on(duckdb.from_arrow(lf), "from Arrow on LazyFrame")
~~~~~~~~~~~~~~~~~^^^^
_duckdb.InvalidInputException: Invalid Input Error: Python Object Type LazyFrame is not an accepted Arrow Object.
PS C:\Users\stett\Documents\python\pql> I saw that in all cases the scan_arrow function was called, and concluded that it's more likely to be a runtime type check issue rather than a real limitation with LazyFrame, hence this is why I raise this as an issue.
To Reproduce
import duckdb
import polars as pl
def _qry_on(qry: duckdb.DuckDBPyRelation, name: str) -> None:
print(f"Query ({name}):\n{qry.sql_query()}\n")
print(qry.pl())
if __name__ == "__main__":
data = {"a": [1, 2, 3], "b": ["x", "y", "z"]}
lf = pl.LazyFrame(data)
df = pl.DataFrame(data)
_qry_on(duckdb.sql("""SELECT * FROM lf"""), "sql on LazyFrame")
_qry_on(duckdb.sql("""SELECT * FROM df"""), "sql on DataFrame")
_qry_on(duckdb.from_arrow(df), "from Arrow on DataFrame")
_qry_on(duckdb.from_arrow(lf), "from Arrow on LazyFrame")OS:
Windows
DuckDB Package Version:
1.4.4
Python Version:
3.13.7
Full Name:
Stettler Thibaud
Affiliation:
None
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have not tested with any build
Did you include all relevant data sets for reproducing the issue?
Yes
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration to reproduce the issue?
- Yes, I have
Activity
paultiq commentedon Feb 11, 2026
The first case
select * from lfpasses because replacement scans explicitly check for LazyFrames, and calls .collect() and .to_arrow() on them.The second case fails because a LazyFrame isn't an arrow type nor does it implement the capsule interface arrow_c_stream. If you changed that to
from_arrow(lf.collect()), it'd work.Implementation Difference
The difference is because from_arrow uses the following function to determine if something is Arrow:
duckdb-python/src/duckdb_py/pyconnection.cpp
Line 2312 in 89ed9a1
But, replacement scans have explicit code for various types, including LazyFrames:
duckdb-python/src/duckdb_py/python_replacement_scan.cpp
Lines 149 to 150 in 694922b
Possible Improvement
The replacement scan behavior could be reimplement/refactored so that the "turn LazyFrames into Arrow" code is shared with the from_arrow path. The code seems somewhat redundant and inconsistent, as is.
I discussed this flow a bit here - #132 (reply in thread)
cmdlineluser commentedon Feb 11, 2026
@paultiq There seems to have been some recent arrow related PRs:
(Not sure how relevant they are here?)
paultiq commentedon Feb 11, 2026
Oh that is interesting: Looks like that'll add
__arrow_c_stream__to LazyFrame.If LazyFrame implements
__arrow_c_stream__, then the from_arrow path would work. The downside is that arrow pushdowns would be bypassed (as they would with any capsule).OutSquareCapital commentedon Feb 12, 2026
Thank you for the answers.
It's not a critical issue anyway, more like a surprising inconsistency.
After reading #132, a tool I haven't seen mentionned is the excellent narhwals.
I would suggest taking a look at it. maybe this could be of some help if the internal conversion/check layer is moved at least partially into python (or not if I'm entirely wrong about what is done there).
And, as an unrelated side-point, would definitely be of help for the public python API -> one entry point for {data, lazy}frames instead of from_df, from_arrow, etc...
evertlammerts commentedon Feb 24, 2026
#334 added somewhat better support for lazyframes. It seems lazyframes are not exporting streams yet though, as of the latest version (1.38.2).