Inscrutable error when reading pyarrow string type columns in a dataframe

Discussed in #36

^{Originally posted by vlulla September 5, 2025}
Pandas recently, i think, got an option called future.infer_string, currently set to False, which will read the sequence of str objects as pyarrow string dtype. Since this is going to be the default in Pandas version 3.0, I have pandas.options.future.infer_string=True at the top of my script. With this option set i get an inscrutable error during some of my interactive sessions. A minimum reproducible example of my repl interaction is listed:

In [1]: import duckdb, pandas as pd
In [2]: s = ["string" for _ in range(5)]
In [3]: pd.options.future.infer_string = True ## changed to be prepared for upcoming update of pandas
In [4]: df = pd.DataFrame({"s":s})
In [5]: duckdb.sql("select * from df")
---------------------------------------------------------------------------
NotImplementedException                   Traceback (most recent call last)
Cell In[5], line 1
----> 1 duckdb.sql("select * from df")

NotImplementedException: Not implemented Error: Data type 'str' not recognized

In [6]: pd.options.future.infer_string = False ## default behavior
In [7]: df1 = pd.DataFrame({"s":s})
In [8]: duckdb.sql("select * from df1")
Out[8]:
┌─────────┐
│    s    │
│ varchar │
├─────────┤
│ string  │
│ string  │
│ string  │
│ string  │
│ string  │
└─────────┘
In [9]:

I have two questions:

Since pyarrow strings are going to become the default for collection of strings, is it possible to be make replacement scan not raise this error with str dtype?
Regardless of how this pandas dtype evolution is addressed, can this error message be improved? I would've never been able to distill this error had my table been bigger than just a few columns!

I love using duckdb from within the python repl and i believe that replacement scans are just so brilliant!

Thanks in advance!

Reproduced with Pandas 3.0, installed via pip install git+https://github.com/pandas-dev/pandas.git

import pandas as pd
import duckdb
print(pd.__version__)
print(duckdb.__version__)

df = pd.DataFrame({"col1": ["Hi"]})

duckdb.sql("select * from df")

3.0.0.dev0+2382.g29ce48952a
1.3.2
Traceback (most recent call last):
File "/Users/extra/git/scratch/mre37.py", line 9, in
duckdb.sql("select * from df")
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
duckdb.duckdb.NotImplementedException: Not implemented Error: Data type 'str' not recognized

Fix is adding the following to numpy/type.cpp, around line 61. Happy to open a PR:

	if (col_type_str == "str") {
		return NumpyNullableType::STRING;
	}

This fix is consistent with guidance in https://pandas.pydata.org/docs/user_guide/migration-3-strings.html:

The new default string dtype is an instance of the pandas.StringDtype class. The dtype can be constructed as pd.StringDtype(na_value=np.nan), but for general usage we recommend to use the shorter "str" alias.

Looks like pandas default version updated to 3.0.0 and I'm now getting this duckdb error.

Same for me.

Workarounded by fixing previous version in build script by python -m pip install duckdb pandas==2.3.3

This error will be gone after the next bugfix release, next week

Support for Pandas 3.0.0 (#277)

v1.4.4 is released, with support for Pandas 3! pip install --upgrade duckdb should do it.

Navigation Menu

Repository navigation

Inscrutable error when reading pyarrow string type columns in a dataframe #37

Discussed in #36

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Inscrutable error when reading pyarrow string type columns in a dataframe #37

Description

Discussed in #36

Activity

paultiq commented on Sep 9, 2025

valankar commented on Jan 22, 2026

justdmitry commented on Jan 22, 2026

evertlammerts commented on Jan 22, 2026

evertlammerts commented on Jan 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions