Avro reader memory leak

Apache Iceberg version

version = "0.9.1"

Please describe the bug 🐞

It seems like there is a memory leak in the avro/reader.py
I have a long running service that keeps crashing. I tried to replicate the issue locally and it seems it also has this issue.

The following code creates a Memory catalog and generates some random data for ingestion into iceberg.

from pyiceberg.catalog.memory import InMemoryCatalog
import tracemalloc
from datetime import datetime, timezone
import polars as pl

def generate_df():
    df = pl.DataFrame(
        {
            "event_type": ["playback"] * 1000,
            "event_origin": ["origin1"] * 1000,
            "event_send_at": [datetime.now(timezone.utc)] * 1000,
            "event_saved_at": [datetime.now(timezone.utc)] * 1000,
            "data": [
                {
                    "calendarKey": "calendarKey",
                    "id": str(i),
                    "referenceId": f"ref-{i}",
                }
                for i in range(1000)
            ],
        }
    )
    return df

df = generate_df()
catalog = InMemoryCatalog("default", warehouse="/tmp/iceberg")
catalog.create_namespace("default")

df = generate_df()
catalog = InMemoryCatalog("default", warehouse="/tmp/iceberg")
catalog.create_namespace("default")
table = iceberg_table = catalog.create_table(
    "default.leak", schema=df.to_arrow().schema, location="/tmp/iceberg/leak"
)

df = pl.DataFrame()

tracemalloc.start()
for i in range(1000):
    df = generate_df()
    df.write_iceberg(table, mode="append")
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics("lineno")
    for stat in top_stats[:10]:
        print(stat)

Slowly but steadily the outputs for the avro reader memory size increases

/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:330: size=370 KiB, count=3782, average=100 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:190: size=222 KiB, count=1891, average=120 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:133: size=184 KiB, count=5673, average=33 B

After some more writes the output looks like this

/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:330: size=420 KiB, count=4290, average=100 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:190: size=251 KiB, count=2145, average=120 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:133: size=208 KiB, count=6435, average=33 B

If we take a look at the AvroFile class it uses the enter and exit dunder methods. The enter method assigns the reader to a variable on the instance but it seems like the different reader classes sticks around.
https://github.com/apache/iceberg-python/blob/main/pyiceberg/avro/file.py#L192

Willingness to contribute

I can contribute a fix for this bug independently
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
I cannot contribute a fix for this bug at this time

I think I found the root cause.
If we only print the avro/reader

    for i in range(1000):
        df = generate_df()
        df.write_iceberg(table, mode="append")
        snapshot = tracemalloc.take_snapshot()
        top_stats = snapshot.statistics("lineno")
        print(i)
        for stat in top_stats:
            if "avro/reader" in str(stat):
                print(stat)

In manifest.py a cache is set https://github.com/apache/iceberg-python/blob/main/pyiceberg/manifest.py#L878
If I set this to 1 the following will always increase but by a very tiny amount 200 byte per write.

/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:322: size=1464 B, count=23, average=64 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:372: size=728 B, count=13, average=56 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:295: size=620 B, count=1, average=620 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:453: size=590 B, count=1, average=590 B

The higher the max cache value the faster the memory increases.

If I remove the cache altogether the memory will be released correctly no matter the amount of iterations.

@Declow

Hi @Declow — thanks for reporting this issue. If your observation is correct, it’s definitely a serious matter, and we should address it promptly. I appreciate your initiative in digging into the root cause.

/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:322: size=1464 B, count=23, average=64 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:372: size=728 B, count=13, average=56 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:295: size=620 B, count=1, average=620 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:453: size=590 B, count=1, average=590 B

Even with the cache set to 1, it still stores a tuple of ManifestFile objects, which grows with each new commit. Since each ManifestFile has the base overhead of a Python object instance plus some attributes (e.g., file path, partition), an increase of around ~100 B to ~500 B per commit seems reasonable.

Did you notice any memory usage growth beyond this range that might indicate a more substantial leak?

For my toy example here it was not that bad. When I used our production setup with a lot more manifest files the service went from using 800MB RAM to reaching it max of 1556MB and crashing within 5 minutes. It was using almost 10MB for each commit maybe more.
When I fixed the issue it on the service by removing the cache everything was stable.

I'm wondering if that just maybe the size of a large manifest list * 128. How many manifests are you materializing in a single manifest-list path? Do you have an estimate of the memory used for a single return of _manifests, i.e. tuple(read_manifest_list(file)) for your Iceberg table?

Thanks for creating the issue. The repro script is super helpful.

This is my current hypothesis:
I think what you're observing is the side affect of the write pattern and its interaction with the _manifests LRU cache.

append the dataframe here will always create a new manifest entry since we use the "fast append" strategy.
This means each write will generate a new manifest list. _manifests, with its LRU cache, will cache the new manifest list's path along with the tuple of ManifestFile objects.
Each cached manifest list path will have overlapping ManifestFile objects since we're constantly adding new manifest files through append.
This means in a LRU cache of size 128, we cache 128 manifest file paths. the first cached item will have 1 tuple of ManifestFile, and the 128th cached item will have 128 tuples of ManifestFile. Most of these ManifestFile are referring to the same underlying manifest file but in a python new object.

# cache key: value, str: tuple[ManifestFile]
1: (ManifestFile1)
2: (ManifestFile1, ManifestFile2)
3: (ManifestFile1, ManifestFile2, ManifestFile3)
...
n: (ManifestFile1, ..., ManifestFilen-1, ManifestFilen)

Each of these ManifestFile objects will refer to the data read using the avro reader.
This is the top allocation, even above the avro reader objects. I believe these are just ManifestFile objects

/Users/kevinliu/repos/iceberg-python/pyiceberg/typedef.py:183: size=1146 KiB, count=12750, average=92 B

Heres a modified repro script:

from pyiceberg.catalog.memory import InMemoryCatalog
import tracemalloc
from datetime import datetime, timezone
import polars as pl

def generate_df():
    df = pl.DataFrame(
        {
            "event_type": ["playback"] * 1000,
            "event_origin": ["origin1"] * 1000,
            "event_send_at": [datetime.now(timezone.utc)] * 1000,
            "event_saved_at": [datetime.now(timezone.utc)] * 1000,
            "data": [
                {
                    "calendarKey": "calendarKey",
                    "id": str(i),
                    "referenceId": f"ref-{i}",
                }
                for i in range(1000)
            ],
        }
    )
    return df

df = generate_df()
catalog = InMemoryCatalog("default", warehouse="/tmp/iceberg")
catalog.create_namespace("default")

df = generate_df()
catalog = InMemoryCatalog("default", warehouse="/tmp/iceberg")
catalog.create_namespace("default")
table = iceberg_table = catalog.create_table(
    "default.leak", schema=df.to_arrow().schema, location="/tmp/iceberg/leak"
)

df = pl.DataFrame()
import gc, objgraph

from pyiceberg.manifest import _manifests

def get_max_value_size(manifest_cache):
    max_key, max_size = max(
        ((key, len(value)) for key, value in manifest_cache.items()),
        key=lambda item: item[1],
        default=(None, -1)
    )
    return max_key, max_size


tracemalloc.start()
for i in range(1000):
    df = generate_df()
    df.write_iceberg(table, mode="append")

    # debug
    manifest_cache = _manifests.cache
    print("ManifestFile instances alive:", objgraph.count("ManifestFile"))
    print(f"size: {len(manifest_cache)}, keys: {len(manifest_cache)}, values: {len(manifest_cache)}")
    max_key, max_size = get_max_value_size(manifest_cache)
    print(f"Max size of the cache value tuple: {max_size}, key: {max_key}")
    # gc.collect()
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics("lineno")
    for stat in top_stats[:10]:
        print(stat)

    print()

Let me know if this makes sense. I'd like to hear what others think

@kevinjqliu

@kevinjqliu It makes sense that the list of manifest files keeps increasing as the write continues but it is not root cause.

As long as the function is cached the AvroFileHeader is never GC'ed. These objects are created each time a AvroFile is read inside of read_manifest_list.
If you run the script with any cache size it will continue to increase the memory usage.
/avro/reader.py:330: size=116 KiB, count=1190, average=100 B
If you run the script for more than 50 iterations.

Flame graph and memory graph after running the script with 50 iterations

After 100 iterations

@Declow

We're experiencing this exact same memory leak in production AWS Lambda functions using pyiceberg. Our data shows the same avro/reader.py line growth:

Memory progression across Lambda invocations (same 50MB file):

1st run: reader.py:330: 52.8 MiB
2nd run: reader.py:330: 88.0 MiB
3rd run: reader.py:330: 106 MiB

Memory grows from 1.4GB to 2GB+ after just 5 invocations processing identical files. Despite aggressive cleanup attempts with gc.collect() and libc.malloc_trim(0), the avro reader allocations persist between Lambda executions.

Our logs confirm @Declow's findings. This is causing production Lambda functions to hit memory limits and crash.

Happy to test any proposed fixes in our production environment with concurrent Lambda executions.

@kris-gaudel

thanks for all the great discussion here. I want to also mention #2004, where we're looking to replace the cython avro reader with one from iceberg-rust. this might help but i think the main issue is still with the cache.

As long as the function is cached the AvroFileHeader is never GC'ed. These objects are created each time a AvroFile is read inside of read_manifest_list.

this is also what im seeing. but it looks like it we manually clear the cache, gc will clean up the objects

@kris-gaudel 's graph shows that at least this is bounded... :)

@andormarkus could you try changing the cache size or manually clear the cache here?

iceberg-python/pyiceberg/manifest.py

Line 878 in 19efd2d

    
           @cached(cache=LRUCache(maxsize=128), key=lambda io, manifest_list: hashkey(manifest_list))

I have implemented manual clearing of the _manifests cache but the memory leak persists in AWS Lambda.

Environment: AWS Lambda, PyIceberg with Glue catalog

Cache Clearing Implementation:

def cleanup(self):
    """Clear PyIceberg cachetools LRUCache to prevent memory leaks"""
    try:
        # Clear the specific cachetools LRUCache that's causing the memory leak
        import pyiceberg.manifest as manifest_module
        
        # The _manifests function has @cached(cache=LRUCache(maxsize=128))
        if hasattr(manifest_module, '_manifests'):
            manifests_func = getattr(manifest_module, '_manifests')
            # cachetools @cached decorator stores cache in the function
            if hasattr(manifests_func, 'cache'):
                cache_size = len(manifests_func.cache)
                manifests_func.cache.clear()
                logger.info(f"Cleared PyIceberg manifest cache ({cache_size} entries)")
                        
    except Exception as e:
        logger.warning(f"Could not clear PyIceberg cache: {e}")
    
    # Clean up objects
    if hasattr(self, 'catalog'):
        del self.catalog
    if hasattr(self, 'client'):
        del self.client
    gc.collect()

Results: Memory still grows from 1438MB → 1858MB across invocations. The manifest cache clearing had minimal impact.

@andormarkus

Well there is a trend up. This is after 300. It does not seem bounded.

@andormarkus Clearing the cache does help

But even if I remove the cache completely it still increases. The caching just makes it way worse.

I was running memray on 1000 iterations and the usage increases over time significantly (detailed via the logs). I couldn't generate a flame graph bc of the binary size being too large

iceberg-python/pyiceberg/avro/file.py

Line 192 in a33cd38

"""Perform cleanup when exiting the scope of a 'with' statement."""

@andormarkus

We ran into this issue (like @andormarkus' case) on AWS Lambda as well. We tried the above mentioned option with regularly clearing the cache but the results were similar with the test script and the memory usage in the real Lambda was still too high for our scenario, hence we opted to deactivated the cache completely:

from pyiceberg.manifest import _manifests

# unwrap the cached function to use the function directly
_manifests = _manifests.__wrapped__  # type: ignore[assignment]

(reg. __wrapped__ see https://cachetools.readthedocs.io/en/latest/#cachetools.cached)

So it's only a workaround and probably we leave some performance on the table, but we valued the stability and less retried Lambda executions above pure performance. Your milage may vary.

@thomas-pfeiffer

Hi @thomas-pfeiffer,

Thanks for sharing your workaround! I'm curious about your specific scenario that led you to disable the cache completely.

In my case, I implemented cache clearing at invocation boundaries:

Init step - Clear cache at the beginning of Lambda execution
Post-execution step - Clear cache after completing the operation

This approach gives me:

During execution: Full benefit of the cache (faster performance)
Between invocations: Memory growth is limited/bounded

With this strategy, my Lambda memory stabilizes at 1600-1800MB instead of continuously growing to OOM, while still maintaining cache performance benefits within each execution. I'm running with 2048MB memory allocation, so this 10-20% overhead is acceptable in our environment.

Could you share more details about your scenario?

What led you to disable the cache entirely rather than clearing at invocation boundaries?
Did you try clearing at execution start/end and find it insufficient?
Are your Lambda executions particularly long-running or processing very large manifest lists?
What memory allocation are you working with?

Understanding your use case would help the community find the right balance between performance and memory stability.

Thanks!

@andormarkus

@andormarkus

I ended up disabling the cache completely as well.

This approach gives me:
During execution: Full benefit of the cache (faster performance)
Between invocations: Memory growth is limited/bounded

It depends on your use-case if this is really true.
My use-case is I consume a stream of events and append the data to iceberg.
The way that the cache is right now does not give additional performance.
When data is appended to iceberg it creates a key in the cache like so.
[key1] -> list of manifests [manifest1]
A second append
[key2] -> list of manifests [manifest1, manifest2]

The first key is never activated and the cache is just updated with a new key. A lot of the same manifest files will just be referenced multiple times in the cache. This is why the overall memory increases so rapidly.
So if your use-case is just writing data to iceberg then you might as well disable the cache.

@andormarkus

@andormarkus reg. your questions:

What led you to disable the cache entirely rather than clearing at invocation boundaries?

I use the repro script from #2325 (comment) and when I disabled the cache completely, the memory was fully stable even after 2.000 executions. I guess the stability was much more a concern since our individual Lambda executions run 2-3 mins, hence repeating failed executions caused by out of memory errors are worse for us then losing a few seconds on a not-cached manifest. (Not taking the remark from #2325 (comment) into account.)

Did you try clearing at execution start/end and find it insufficient?

Not really. Disabling the cache was the simpler solution. And since we leverage some other things we felt eradicating this memory leak source completely seemed the better choice for us.

Are your Lambda executions particularly long-running or processing very large manifest lists?

Average duration was between 2-3 min in the last days; Depends a bit on the incoming data in our use case. Haven't checked regarding the manifest list tbh.

What memory allocation are you working with?

We allocated 2048MB to the AWS Lambda. On some days the usage goes up to 2022MB, but it's not very consistent e.g., yesterday it was mostly around ~1400MB. Depends highly on the incoming data for us, but no out of memory errors since the workaround so far.

Small remark / question reg. your approach:

Init step - Clear cache at the beginning of Lambda execution
Post-execution step - Clear cache after completing the operation

@andormarkus Doesn't that mean you clean your cache twice immediately one another? I would have assumed the post-execution cache cleaning to be enough.

Our use case seems similar to @Declow scenario, where we do only appends on the tables we write. I guess we would have no hits on the cache. (Haven't tried it tbh.)

Maybe 2 somewhat stupid questions for the round here:

For which use case / scenario does the cache work as expected or intended?
What is the intended purpose of this cache? What is it supposed to do?

Hey all, thanks for all the detailed discussion here.

Summarizing from what I've read above

the manifest cache is problematic; it's utilizing a lot of memory, esp in the case for fast append
theres a potential memory leak in Avro reader

I've opened #2951 to cache the ManifestFile
We should fix the avro reader leak as well

Its probably a good idea to use the script above and create a benchmark script to test for memory leaks

im happy with the current state of #2951
Please give it a try.

We should also make the cache size configurable, and be able to disable it too.

@kevinjqliu

Thanks for the follow-up @kevinjqliu, is there an issue to make the cache size configurable/choice to disable it? I'd love to work on that

Thanks for creating the issue! 😄

fix manifest cache (#2951)

#2951 is now merged. I tried to benchmark the memory leak in Avro reader, but found most of the memory usage was caused by the cache. I'm curious if the Avro reader issue is still significant after this fix.

Make manifest cache size configurable (#2993)

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'

Navigation Menu

Repository navigation

Avro reader memory leak #2325

Apache Iceberg version

Please describe the bug 🐞

Willingness to contribute

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Avro reader memory leak #2325

Description

Apache Iceberg version

Please describe the bug 🐞

Willingness to contribute

Activity

Declow commented on Aug 13, 2025

sungwy commented on Aug 15, 2025

Declow commented on Aug 15, 2025

sungwy commented on Aug 15, 2025

kevinjqliu commented on Aug 16, 2025

Declow commented on Aug 18, 2025

kris-gaudel commented on Aug 21, 2025

kris-gaudel commented on Aug 21, 2025

andormarkus commented on Aug 22, 2025

kevinjqliu commented on Aug 22, 2025

andormarkus commented on Aug 22, 2025

Declow commented on Aug 22, 2025

kris-gaudel commented on Aug 23, 2025

12 remaining items

thomas-pfeiffer commented on Dec 3, 2025

andormarkus commented on Dec 8, 2025

Declow commented on Dec 8, 2025

thomas-pfeiffer commented on Dec 8, 2025

kevinjqliu commented on Jan 25, 2026

kevinjqliu commented on Jan 25, 2026

kris-gaudel commented on Jan 25, 2026

kevinjqliu commented on Jan 25, 2026

kevinjqliu commented on Jan 29, 2026

github-actions commented on Jul 29, 2026

github-actions commented on Aug 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions