Skip to content

[1.4.0] Queries don’t terminate since type change of .arrow() #85

Closed
Closed
[1.4.0] Queries don’t terminate since type change of .arrow()#85
Labels
bugSomething isn't workingreproduced

Description

@Dino-Burger
Dino-Burger
opened on Sep 25, 2025

What happens?

Duckdb changed the return type of arrow() from a pyarrow table to a RecordBatchReader.
The consequence of this is that some queries don’t terminate.

import duckdb

xxx = duckdb.sql("From range(5) T(a)")
yyy = xxx.arrow()

# works fine
duckdb.sql("From xxx")

# never terminates
duckdb.sql("From yyy")

# works fine
zzz = xxx.fetch_arrow_table()
duckdb.sql("From zzz")

It would be great if the query From yyy would throw an error instead of not terminating, for example

The table yyy is a RecordBatchReader which is not supported form replacement scans. Please use fetch_arrow_table() instead of fetch_record_batch() (or .arrow()).

To Reproduce

Execute above code in a venv with duckdb and pyarrow installed.

OS:

Windows

DuckDB Package Version:

1.4.0

Python Version:

3.13.2

Full Name:

Dino Burger

Affiliation:

None

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a nightly build

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

  • Yes, I have

Did you include all relevant configuration to reproduce the issue?

  • Yes, I have

Activity

  1. evertlammerts
    added
    bugSomething isn't working
    reproduced
    on Sep 25, 2025

  2. Tishj commented on Sep 26, 2025

    @Tishj
    Tishj
    on Sep 26, 2025
    Member

    That's an interesting issue 🤔
    What I imagine happens is:
    yyy is created, this creates a streamqueryresult, which starts an execution pipeline but blocks immediately, to be resumed when the stream is fetched from.

    The scan of xxx likely kills this previous execution, starting a new one and materializing the entire result

    Somehow when yyy is scanned it doesn't detect that the execution was canceled (???)
    I would expect this to throw an error or produce an empty result.
    But instead it seems to block indefinitely

  3. szarnyasg
    added
    reproduced
    and removed
    reproduced
    on Oct 2, 2025
  4. duckdblabs-bot
    added
    reproduced
    and removed
    reproduced
    on Oct 2, 2025

  5. paultiq commented on Oct 3, 2025

    @paultiq
    paultiq
    on Oct 3, 2025
    Contributor

    I thought this was related to something I was working on (it's not), so took a look...

    You've got two threads deadlocked, with one holding the ClientContextLock and the other waiting on it.

    One on ResultArrowArrayStreamWrapper::MyStreamGetNext, and the other at :ArrowArrayStreamWrapper::GetNextChunk()

     	[Inline Frame] _duckdb.cp313-win_amd64.pyd!std::_Mutex_base::lock() Line 52	C++
 	[Inline Frame] _duckdb.cp313-win_amd64.pyd!std::lock_guard<std::mutex>::{ctor}(std::mutex &) Line 445	C++
 	[Inline Frame] _duckdb.cp313-win_amd64.pyd!duckdb::ClientContextLock::{ctor}(std::mutex &) Line 322	C++
 	[Inline Frame] _duckdb.cp313-win_amd64.pyd!duckdb::make_uniq(std::mutex &) Line 65	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::ClientContext::LockContext() Line 169	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::StreamQueryResult::LockContext() Line 44	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::StreamQueryResult::IsOpen() Line 193	C++
>	_duckdb.cp313-win_amd64.pyd!duckdb::ResultArrowArrayStreamWrapper::MyStreamGetNext(ArrowArrayStream * stream, ArrowArray * out) Line 123	C++

    And the other:

     	_duckdb.cp313-win_amd64.pyd!duckdb::ArrowArrayStreamWrapper::GetNextChunk() Line 56	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::ArrowTableFunction::ArrowScanParallelStateNext(duckdb::ClientContext & context, const duckdb::FunctionData * bind_data_p, duckdb::ArrowScanLocalState & state, duckdb::ArrowScanGlobalState & parallel_state) Line 124	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::ArrowTableFunction::ArrowScanInitLocalInternal(duckdb::ClientContext & context, duckdb::TableFunctionInitInput & input, duckdb::GlobalTableFunctionState * global_state_p) Line 171	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::ArrowTableFunction::ArrowScanInitLocal(duckdb::ExecutionContext & context, duckdb::TableFunctionInitInput & input, duckdb::GlobalTableFunctionState * global_state_p) Line 180	C++
>	[Inline Frame] _duckdb.cp313-win_amd64.pyd!duckdb::TableScanLocalSourceState::{ctor}(duckdb::ExecutionContext &) Line 80	C++
 	[Inline Frame] _duckdb.cp313-win_amd64.pyd!duckdb::make_uniq(duckdb::ExecutionContext &) Line 65	C++

  6. bodacious-bill commented on Jan 27, 2026

    @bodacious-bill
    bodacious-bill
    on Jan 27, 2026

    This is still happening in 1.4.4. I'm starting with con = duckdb.connect() and then running sql queries with result = con.execute(sql).arrow(). Using .fetch_arrow_table() instead as a workaround appears to work so far.

    It would be nice if this type change on the return value for .arrow() were documented anywhere. According to the documentation, "fetch_arrow_table" is an alias of "arrow":

    https://duckdb.org/docs/stable/clients/python/conversion#apache-arrow

    Is this behavior intended? Is fetch_arrow_table() no longer an alias of arrow()? Previously I just downgraded to 1.3.2 to avoid this, but I want to run 1.4.4 for the pandas 3.0.0 support.

  7. paultiq commented on Jan 27, 2026

    @paultiq
    paultiq
    on Jan 27, 2026
    Contributor

    That's definitely a doc error, see #97 (comment) for a discussion of current state

  8. bodacious-bill commented on Jan 27, 2026

    @bodacious-bill
    bodacious-bill
    on Jan 27, 2026

    Are the docs on github here someplace? I can update it and submit a PR

  9. paultiq commented on Jan 27, 2026

    @paultiq
    paultiq
    on Jan 27, 2026
    Contributor

    https://github.com/duckdb/duckdb-web

  10. evertlammerts commented on Feb 10, 2026

    @evertlammerts
    evertlammerts
    on Feb 10, 2026
    Member

    I've created a few pr's to update docs. Closing this issue.

  11. evertlammerts
    closed this as completedon Feb 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingreproduced

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions