Cache engineering: let agents drop KV entries instead of rewriting history

Every long-running agent I have put into production eventually hits the same wall. The trajectory grows, the context window fills, and the harness has to do something: trim old tool output, or summarize the first eighty turns into a paragraph. Both work. Both are also more expensive than they look, because editing the beginning of a transcript throws away the KV cache for everything after the edit.

Cache Engineering for Agents (Fu, Wang, et al., Tsinghua) takes that cost seriously and proposes a different lever. Leave the text alone. Let the agent operate on its KV cache directly, dropping selected entries from future attention and re-encoding the positions of what remains. The paper pairs this with a serving engine, AKV, that keeps cache reuse correct once the cache stops being a pure function of the token prefix. The code is at thu-nics/AKV.

Why context engineering is expensive

The paper's motivating number is blunt: in their accounting, KV cache reads make up 82.4% of total API cost for long agent trajectories. Agents re-read their own history every turn, so most of the bill goes to re-reading it.

Context engineering cuts the amount of history, but it changes the prefix. Prefix caching in engines like SGLang and vLLM only reuses state for an unchanged prefix. Edit message 5 and every cached state from message 6 onward is invalid and must be prefilled again. The earlier the edit, the larger the bill.

Harnesses respond by deferring the edit until a threshold (say 96K tokens), then cutting hard and keeping only a few recent rounds. That keeps re-prefill rare but loses intermediate information all at once. The paper quantifies the dilemma on 100 BrowseComp-Plus tasks with GPT-OSS-120B: trimming to the latest 12 tool-response rounds costs 1.85x the full-context baseline, because of re-prefill. Summarizing at 48K saves about 7% but costs 5 points of accuracy.

Three operations on the cache

Cache engineering keeps the logical token sequence intact and edits the active cache, meaning the set of KV entries that later tokens attend to. There are three operations:

• append: the ordinary decode step. Compute the new token's KV conditioned on the active cache and add it.
• drop(P): remove the entries at indices P from future attention. Retained entries and their position encodings are untouched, so nothing is recomputed.
• repos: re-encode retained keys to contiguous positions 0..r-1 and reset the next position to r. For RoPE this is a rotation, K ← R(q)R(p)⁻¹K, with values unchanged.

The repos operation is the less obvious half, and the ablations show it matters. Dropping entries does not lower position indices, so a long-running agent with a small active cache still marches toward the model's position limit. With GPT-OSS-120B on BrowseComp-Plus, tool-response dropping without repos scores 55%; adding repos lifts it to 62%, mostly because runs no longer halt at the position ceiling.

The default policy is deliberately simple: a tool-response rolling drop that keeps all reasoning and the latest 12 tool responses, and drops older tool-response cache. The authors base this on attention measurements. Older tokens get less attention than recent ones, and tool output gets less than assistant messages at comparable depth. The ablation backs it up: dropping reasoning cache, alone or with tool cache, lowers pass rates on both benchmarks tested. Raw tool output is cheap to forget; the model's own reasoning is not.

One subtlety I like: a retained entry was computed while the dropped entries were still visible, so it still carries some of their information. A drop is softer than deleting text.

The serving problem: history dependence

This is where the paper earns its systems half. Under standard caching, the token prefix fully determines the cache. Under cache engineering it does not. Take tokens A, B, C. If you append C and then drop A, C's KV was computed while attending to A. If you drop A and then append C, it was not. Same retained tokens, same positions, different cache. A plain prefix match would silently reuse the wrong state.

AKV handles this with a principle the authors call stateless interface, stateful serving. The client sends the full message list as usual, plus an optional cache field that lists drop and reposition operations by message index. Every request fully describes the cache it needs, so the server may evict anything between turns and still reconstruct it. Internally:

• An extended radix tree adds drop and reposition nodes, so matching uses both the token history and the operation history.
• Drop-aware eviction frees cache that every live request has dropped, and evicts dropped cache before cache from completed requests, since the latter is likely to be needed next turn.
• Segment-batch attention restores missing state in one prefill pass. Queries are split at operation boundaries so each segment attends to the right visible set, while the FFN runs over the full sequence once to avoid repeated weight reads.
• Repositioned copies are made by rotating keys and copying values, never by recomputing hidden states.

It is built on SGLang, which matters for adoption.

What the numbers say

Across three models (MiniMax-M2.7, GPT-OSS-120B, MiroThinker-1.7-Mini) and four benchmarks (BrowseComp-Plus, GAIA-Text, SWE-bench Verified, xbench-DeepSearch), the rolling-drop policy beat 96K-threshold summarization by 1.7 points on average and cut cost by 27.7%, or 21.9% against 48K summarization. The biggest single gain is GPT-OSS-120B on SWE-bench Verified: the pass rate rises from 40.0% to 43.2% while per-task cost falls from 20.0 to 9.7 cents.

On serving, comparing each method at its peak-throughput concurrency on eight A800s, AKV delivers 1.18x output throughput and a 1.51x speedup in P95 request latency. The gap widens at high concurrency, where KV pressure forces the baseline into more re-prefill and fewer requests can decode at once.

Two extensions are worth noting. Policies refined through recursive self-improvement cut held-out cost by a further 7.1% beyond the hand-designed policy. Exposing cache management as a tool the agent calls itself gave the best BrowseComp-Plus scores for both models tested. For MiniMax-M2.7, though, cost went up (5.4 to 5.9 cents) because the agent ran more turns.

My read

The gains are real but uneven. Several model-benchmark cells are flat or slightly worse, such as GPT-OSS-120B on xbench-DS. Costs are modeled from public token prices rather than measured bills, though the authors deliberately charge every repositioned token at the cache-hit rate. Numerically, rotating keys is not bitwise-identical to recomputing them, and the paper says so.

The larger constraint is access. Cache engineering needs an engine that accepts cache operations, so for now it is a self-hosted technique. Closed APIs expose prompt caching, not the cache. The authors also list linear-attention models as unsupported.

Still, I think the framing is right. We have treated the transcript as the only thing an agent can manage, and paid for it with re-prefill. Treating the active cache as a first-class, history-aware object, with a serving layer that keeps reuse correct, is the cleaner abstraction. If you run open-weight agents on your own serving stack, it is worth trying a rolling-drop policy on tool output before your next summarization threshold.