Repository navigation
Adding parquet files error: Failed to create checkpoint because of error: Could not read enough bytes from file #129
Description
What happens?
Hi,
I have been experiencing some issues inserting data from parquet files into duckdb. In most cases this works well, but I have a few cases where I have consistent errors, that look something like this:
TransactionContext Error: Failed to commit: Failed to create checkpoint because of error: Could not read enough bytes from file "/tmp/.db": attempted to read 262144 bytes from location 4869745866884263936
In this particular case I am exporting a large table from BigQuery as parquet files (5000, totally ~125GB). Some time around loading the 2000th file into duckdb I will get an error thrown like the above.
In order to add the files, I have tried both of the following statements:
COPY db.{table}
FROM '{file}' (FORMAT parquet)
and
INSERT INTO db.{table}
SELECT {select_clause} FROM read_parquet('{file}')
I've also tried v1.3.2 and v1.4.1 of the python package, but the error still happens.
To Reproduce
N/a
OS:
linux
DuckDB Package Version:
1.3.2
Python Version:
3.9
Full Name:
Alicia Peters
Affiliation:
Benchling
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have not tested with any build
Did you include all relevant data sets for reproducing the issue?
No - Other reason (please specify in the issue body)
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration to reproduce the issue?
- Yes, I have
Activity
taniabogatsch commentedon Oct 20, 2025
Hi @alicia-peters-dev - the easiest way to look into this would be by sharing the data (up to the 2000ish file). See my other reply.
"/tmp/.db"- that file name looks odd, how are you attaching/opening the file?Besides that, it's very hard to say why the location passed to the file read function is uninitialized/"out of bounds". It's a bug, and maybe with a stack trace we could know more, is it possible to run this with lldb and get that, maybe?
As for workarounds, maybe you could try running manual checkpoints more aggressively. I.e., I assume you're running
in a loop for each file? We auto-checkpoint every 16mb, but maybe you can decrease that (
SET checkpoint_threshold = ...), or run manual checkpoints viaCHECKPOINT db. However, that would likely just hide the bug, if it does anything...Another idea to work around it is to disable different compression methods - depending on your data, you could only run with subsets and see if that makes a difference, to eliminate the code path with the bug, if its compression-related. Available compression functions are
"rle", "dictionary", "pfor", "bitpacking", "fsst", "chimp", "patas", "zstd", "alp", "alprd", "roaring", "dict_fsst". You can disable them via:SET disabled_compression_methods='fsst,rle';.evertlammerts commentedon Nov 1, 2025
Fix has landed in duckdb/duckdb#19475.