help please？！！！

[NierWinter]

need some clarification. When using llama-cpp-python to start the server API with options --cache true --cache_type disk, I noticed that every time I query the model, it triggers
LlamaDiskCache.getitem. Does this mean that when using LlamaDiskCache in this way, the KV cache read/write completely bypasses GPU memory and system RAM, directly accessing the KV cache on disk? Wouldn't that be much slower?

[NierWinter]

I originally thought that the disk was only used for persistence, and that upon startup, the KV cache would be paged back into memory or GPU memory. Isn't that how it works?

[kamalesh404]

Good question — the confusion is understandable, but LlamaDiskCache is not the KV cache, and it doesn't move anything off the GPU. Two separate things:

1. What LlamaDiskCache actually is: a prompt cache
It maps prompt text -> final completion text. __getitem__ firing on every query is completely normal — that's just the lookup ("have I seen this exact prompt before?"). On a miss, inference runs exactly as it would without any cache: weights + active KV cache in VRAM/RAM as usual.

So no, nothing "bypasses GPU memory and system RAM". The disk only stores past results so identical prompts can be served without re-running the model — and it persists across restarts, which is its whole point vs --cache_type ram.

2. Where the real KV cache lives
Unchanged by this setting: during generation the KV cache lives in RAM/VRAM like normal. The disk cache never holds KV tensors.

3. About performance
The per-request cost is one tiny disk lookup (a SQLite/hash lookup), negligible next to even one model forward pass. A cache hit saves the entire inference, so for repeated identical prompts (same system prompt, same prefix) it's a big net win. If your prompts basically never repeat, the cache just never hits — then either switch to --cache_type ram to avoid disk I/O entirely, or drop --cache altogether.

4. Your mental model, corrected
There's no "page KV back into memory on startup" step, because nothing is paged out: on startup the cache is just a cold on-disk lookup table that fills up as you query. First identical prompt = miss + inference + store; second identical prompt = hit, served from disk, no GPU work at all.

Rule of thumb: repeated prompts -> keep disk cache; unique prompts every time -> disable it.

[laulauraluo]

The lookup does not mean inference reads its active state directly from disk. An important correction to the earlier answer: this cache stores LlamaState objects, not completed response text.

In the current source, the completion path retrieves a state for a matching token prefix, compares it with the state already in memory, and calls load_state() only if the cached prefix is longer. It then proceeds into generation.

save_state() and load_state() copy native llama context state to bytes and restore it through llama_state_set_data. So your persistence-and-restore intuition is closer: restoration happens on a useful request-time hit, rather than eagerly loading every cached entry at startup. This setting does not replace the active inference context with disk-backed access, and a hit does not simply return an old answer.

Disk I/O and state-copy overhead can matter; whether caching helps depends on the prefix computation saved and the state size. Seeing __getitem__ alone proves neither a useful hit nor a speedup. With verbose logging, cache hit in this completion path indicates that a state was actually loaded.

This is a source-level check of the linked revision, not a GPU performance measurement; your installed version may differ.