especially because this is the most painfully glaring flaw in their plan. Their solution is for an inference provider to... store the KV cache (which they can compute!) on-premise, on their own disks, but pay some third party for it?
Well, it’s one flaw. I would argue that the bigger flaw, which you alluded to, is that the cost of computing the cache yourself maxes out in the single digit dollars even very large frontier models, and that’s a one-time cost. Even if you imagine all the logistics are free and all the transfers are instant, what are we even talking about here from an economic perspective?
KV caching is a super interesting engineering space, especially when you’re talking about local models where compute and memory bandwidth are highly constrained and you’re trying to trim fractions of a second everywhere you can by flipping between different ICL prefixes. But selling caches for specific documents just makes no sense at all.
The KV cache is order dependent and dependent on the context of tokens which exist before the KV cache.
There are some transformation approaches to re-use the kv cache across inferences, but none are in wide use due to accuracy concerns following the transformation.
Isn't it also, most fundamentally, dependent on the model weights?
My understanding was that what the KV cache stores is nothing else than the "activations" of the W_k and W_v matrices of an attention module for a given input sequence.
So I don't quite understand how this is supposed to work:
> Let a publisher precompute a document's KV cache, and let every other agent buy the right to load it and skip prefill.
Should a publisher precompute the cache for every popular model that is out there?
The paper has a section on "Reusing precomputed KV across queries" which talks about how other papers have tried to address this problem, but yeah, this paper adds nothing on its own besides a catchy title.
AsyncResoning[1] does a trick of that sort to give agents concurrent cache views.
You basically have two agents look at the same cache under different views. Say agent_0 gets [a_1, a_0] and agent_1 gets [a_0, a_1]. They also write to this cache concurrently while decoding. To solve positional embedding inconsistencies they rotate the query projections for each block (a_0 and a_1) separately.
The computations you get that way do not exactly match the setup where you would naively prefill on every step, but are close enough.
Same trick could be used for the setup discussed here, I guess: prefill the document cache separately (p), prepend the system prompt (s) and get a cache view [s, p] from which you can then decode.
Prefix caching is already widely deployed by all providers, right? llama.cpp does it. vLLM does it. I'm sure everyone hosting LLMs for a living does it. This paper seems to focus entirely on prefixes (i.e. the prefilled content is rooted at 0). This is... nothing.
The referenced CacheBlend paper (https://arxiv.org/pdf/2405.16444) which tries to stitch together multiple independent prefills looks more interesting and is new to me. The problem it's trying to solve is:
* KV projections for a given token at a given layer are a function of the residual at that layer,
* which is a function of the attention contribution of the previous layer,
* which is a (nonlinear) function of all earlier tokens' KV values at the previous layer.
This is what stops you from just pasting KV blocks together. Intuitively it might feel like you could do the equivalent of an MPEG deblocking filter to fix up the edges, but there's no guarantee the tokens that need fixing up are at the beginning of the KV block, so they have to be sneaky about it.
Unfortunately while that paper is quite verbose it's lacking in detail in the most important part: how they perform the approximate KV recomputation. It looks like the rough idea is that they fully recompute the KV for the first layer, and use the deviation between the recompute and the original cached KV as a heuristic for how important it is to recompute the full KV values (i.e. all remaining layers) for that token. They use that to derive a mask for the tokens which most strongly attend to the earlier context, then do a sparse update of those tokens.
What's still unclear is how this actually ends up being a performance win, given that the sparse update for each token still requires the exact KV for all the prior tokens in order to actually arrive at the correct value. It just kind of recurses the problem instead of fixing it. Maybe they just use the precomputed KV for the other tokens as input, and live with the approximation?
I think this is already somewhat pragmatically solved: just don't pull huge documents into context. Give the LLM tools to search them and retrieve the fragments that are actually relevant.
Yes, but presumably the authors are suggesting broader application than just caching a system prompt.
The paper's approach should work well if (a) you can calculate KV(A || B) as a function of KV(A) and KV(B) independently, (b) you can identify which documents A1, A2, A3, ... are used commonly enough to be worth caching, and (c) it is cheaper to buy and sell KV(A) on a market than to compute KV(A) when it is needed. Given the size of KV(A) I am not sure that (c) will become true even if people solve the open research problem represented by (a) and accept the state-of-the-art trade-offs known for (b).
This paper doesn't make any sense - for background, I've maintained an AI client that's cross-platform, cross-provider, and integrates llama.cpp since 2022. I don't know why they think "agents" don't share prefill work - paid providers cache on the prefill text, llama.cpp, same, and I specifically hooked up llama.cpp so it can do subsets as well. i.e. all the agents would reuse the cache
It reads like it started from an underspecification of "agents" x a strain of pop-wisdom about "KV cache" that I've seen enter mainstream discourse over the past 3 months that is Not Even Wrong, then, they solved a non-existent problem.
EDIT: based on the rest of comments either requesting a primer on terms, or, pointing out it makes errors in even more obvious ways, flagging.
I don't think Luoyuan Zhang is necessarily doing this, but I'm pretty sure lots of people are using arxiv as a glorified blog and hoping no one notices.
> Then the part that matters: where the KV lives
When your abstract was clearly generated by an LLM and not curated to at least make it sound human, it does not make me want to read your paper.
especially because this is the most painfully glaring flaw in their plan. Their solution is for an inference provider to... store the KV cache (which they can compute!) on-premise, on their own disks, but pay some third party for it?
Well, it’s one flaw. I would argue that the bigger flaw, which you alluded to, is that the cost of computing the cache yourself maxes out in the single digit dollars even very large frontier models, and that’s a one-time cost. Even if you imagine all the logistics are free and all the transfers are instant, what are we even talking about here from an economic perspective?
KV caching is a super interesting engineering space, especially when you’re talking about local models where compute and memory bandwidth are highly constrained and you’re trying to trim fractions of a second everywhere you can by flipping between different ICL prefixes. But selling caches for specific documents just makes no sense at all.
The KV cache is order dependent and dependent on the context of tokens which exist before the KV cache.
There are some transformation approaches to re-use the kv cache across inferences, but none are in wide use due to accuracy concerns following the transformation.
Isn't it also, most fundamentally, dependent on the model weights?
My understanding was that what the KV cache stores is nothing else than the "activations" of the W_k and W_v matrices of an attention module for a given input sequence.
So I don't quite understand how this is supposed to work:
> Let a publisher precompute a document's KV cache, and let every other agent buy the right to load it and skip prefill.
Should a publisher precompute the cache for every popular model that is out there?
The paper has a section on "Reusing precomputed KV across queries" which talks about how other papers have tried to address this problem, but yeah, this paper adds nothing on its own besides a catchy title.
Just curious, do you have links to read more about transformations or other techniques for KV cache reuse?
All major model providers offer prefix caching, which is this.
No, reusing segments of the kv cache for different purposes in an order independent manner is an active research area.
Any keyword or paper I can search for?
AsyncResoning[1] does a trick of that sort to give agents concurrent cache views.
You basically have two agents look at the same cache under different views. Say agent_0 gets [a_1, a_0] and agent_1 gets [a_0, a_1]. They also write to this cache concurrently while decoding. To solve positional embedding inconsistencies they rotate the query projections for each block (a_0 and a_1) separately.
The computations you get that way do not exactly match the setup where you would naively prefill on every step, but are close enough.
Same trick could be used for the setup discussed here, I guess: prefill the document cache separately (p), prepend the system prompt (s) and get a cache view [s, p] from which you can then decode.
1. https://arxiv.org/abs/2512.10931
Absolute slop paper. Replace document with text and you'll get it.
"People are asking the same questions and an answer is generated every time, what if we could like cache the questions and their answers..."
Sounds like someone was using chatgpt to understand how chatgpt works and then asked it to generate a paper based on his proposal to improve it.
At least it wasn't a patent.
Prefix caching is already widely deployed by all providers, right? llama.cpp does it. vLLM does it. I'm sure everyone hosting LLMs for a living does it. This paper seems to focus entirely on prefixes (i.e. the prefilled content is rooted at 0). This is... nothing.
The referenced CacheBlend paper (https://arxiv.org/pdf/2405.16444) which tries to stitch together multiple independent prefills looks more interesting and is new to me. The problem it's trying to solve is:
* KV projections for a given token at a given layer are a function of the residual at that layer,
* which is a function of the attention contribution of the previous layer,
* which is a (nonlinear) function of all earlier tokens' KV values at the previous layer.
This is what stops you from just pasting KV blocks together. Intuitively it might feel like you could do the equivalent of an MPEG deblocking filter to fix up the edges, but there's no guarantee the tokens that need fixing up are at the beginning of the KV block, so they have to be sneaky about it.
Unfortunately while that paper is quite verbose it's lacking in detail in the most important part: how they perform the approximate KV recomputation. It looks like the rough idea is that they fully recompute the KV for the first layer, and use the deviation between the recompute and the original cached KV as a heuristic for how important it is to recompute the full KV values (i.e. all remaining layers) for that token. They use that to derive a mask for the tokens which most strongly attend to the earlier context, then do a sparse update of those tokens.
What's still unclear is how this actually ends up being a performance win, given that the sparse update for each token still requires the exact KV for all the prior tokens in order to actually arrive at the correct value. It just kind of recurses the problem instead of fixing it. Maybe they just use the precomputed KV for the other tokens as input, and live with the approximation?
I think this is already somewhat pragmatically solved: just don't pull huge documents into context. Give the LLM tools to search them and retrieve the fragments that are actually relevant.
Yeah, I'm really not sure what the point of this paper is. Every non-toy environment does prefix caching.
Yes, but presumably the authors are suggesting broader application than just caching a system prompt.
The paper's approach should work well if (a) you can calculate KV(A || B) as a function of KV(A) and KV(B) independently, (b) you can identify which documents A1, A2, A3, ... are used commonly enough to be worth caching, and (c) it is cheaper to buy and sell KV(A) on a market than to compute KV(A) when it is needed. Given the size of KV(A) I am not sure that (c) will become true even if people solve the open research problem represented by (a) and accept the state-of-the-art trade-offs known for (b).
Seems Cloudflare is now doing this for scraping, so makes sense to continue down the pipeline!
Does anyone have a good recommendation for explaining or as a primer on KV cache?
If you want the actual maths instead of handwaving, I recommend: https://arxiv.org/abs/2207.09238
For something gentler, 3Blue1Brown: https://www.youtube.com/watch?v=eMlx5fFNoYc (this is part of a series)
convert this question to KV cache and give it to your agent
This paper doesn't make any sense - for background, I've maintained an AI client that's cross-platform, cross-provider, and integrates llama.cpp since 2022. I don't know why they think "agents" don't share prefill work - paid providers cache on the prefill text, llama.cpp, same, and I specifically hooked up llama.cpp so it can do subsets as well. i.e. all the agents would reuse the cache
It reads like it started from an underspecification of "agents" x a strain of pop-wisdom about "KV cache" that I've seen enter mainstream discourse over the past 3 months that is Not Even Wrong, then, they solved a non-existent problem.
EDIT: based on the rest of comments either requesting a primer on terms, or, pointing out it makes errors in even more obvious ways, flagging.
I don't think Luoyuan Zhang is necessarily doing this, but I'm pretty sure lots of people are using arxiv as a glorified blog and hoping no one notices.
A truly global singleton
Lambda computing for prompts?