Context compression: cutting the tokens your agents re-pay for

The tokens you re-pay for

In a single-shot chat, the prompt is paid for once. In an agent loop, it isn't. A coding agent runs ten, twenty, fifty steps, and on each step the entire running transcript — the system prompt, the tool schemas, every tool result so far, every model turn — goes back over the wire. A 30,000-token tool output you saw on step three is still being billed on step thirty. Latency tracks the same curve: more input tokens mean more time to first token and a larger KV cache to manage. The model's quality is roughly fixed. The context is the part you actually engineer, and it's where the spend compounds.

What's filling the window

When I profile a bloated agent, the offenders are predictable. Tool outputs are the largest — a directory listing, a 500-row query result, a stack trace, a page of JSON from some API, most of it structural noise the model never reads closely. Then logs. Then RAG chunks retrieved with low precision: five passages fetched to use two sentences. Then the conversation history itself, which carries all of the above forward on every turn. The signal-to-token ratio in this material is low. A JSON array of a hundred near-identical records is mostly repeated keys and formatting.

Compression is not summarization

The reflex is to summarize — hand the context to a cheap model and ask it to shrink. That trades tokens for risk. A summarizer is lossy in ways you can't predict, and it will quietly drop the one line that mattered. The work worth doing treats compression as structure-aware and, ideally, reversible. JSON has structure you can exploit with little or no loss. Code has an abstract syntax tree. Prose compresses differently again. And if you keep the originals, the model can ask for the full version when it genuinely needs it — so "compression" stops meaning "permanent deletion."

Headroom is a recent open-source take on this, and it's a useful map of the design space whether or not you adopt it. It sits between the agent and the provider and compresses everything the agent reads — tool outputs, logs, RAG chunks, files, and history — before it reaches the model, caching the originals so they can be retrieved on demand.

What a compression layer does

A compression layer sits between the agent and the provider: route by content type, compress, align the cache, keep the originals retrievable.

The shape is consistent across serious implementations. A router inspects each piece of content and classifies it as JSON, source code, or free text. Each type goes to a compressor built for it — a structural pass for JSON, an AST-aware pass for code, a learned model for prose. A cache-alignment step keeps the stable prefix of the prompt byte-identical across turns, so the provider's KV cache actually hits instead of missing on a one-character drift. A reversible store holds the originals locally and exposes a retrieval tool, so the model can pull back full content when it needs it. The agent's own code doesn't change; the layer runs as a proxy or wraps the SDK.

The detail I'd flag for production is that cache-alignment step. Provider prompt caching only pays off when the prefix is stable, and plenty of agent frameworks invalidate it by accident — injecting a timestamp, reordering fields, rewriting whitespace. A layer that stabilizes the prefix can save more than the compression does.

What it actually buys you

Savings track structure: repetitive JSON and logs compress around 92%; original source and prose far less. Figures from Headroom's published benchmarks.

Savings depend entirely on the workload, and the honest version is a range, not a headline. On Headroom's published numbers, structured and repetitive workloads compress hardest — a code search dropping from 17,765 to 1,408 tokens, an SRE debugging session from 65,694 to 5,118, both near 92%. Less structured work compresses less: a codebase exploration only 47%, because original prose and novel code lack the redundancy JSON has. That spread is the useful signal. If your agent mostly chews on API responses and logs, you'll see the high end; if it reads source and writes prose, plan for the low end.

Accuracy is the question that matters, and it's the one to verify on your own data. Headroom reports parity on standard benchmarks — GSM8K unchanged, TruthfulQA within noise, SQuAD and tool-calling preserved at high compression. I treat numbers like these as a reason to run an evaluation, not as a result I can quote. The failure mode of compression is quiet: it doesn't crash, it occasionally removes the one token the model needed, and you only see it in task success rate, never in the logs.

Where it fits, and where I'd skip it

This earns its place when your agents read a lot of machine-generated content, when you pay per token at volume, or when latency from oversized prompts is the bottleneck. It's also the natural home for cross-agent memory — a shared, deduplicated store across Claude, Codex, and the rest, so the same context isn't rebuilt per agent.

I'd skip it when context is already small, when a single provider's native compaction covers you, or when you can't run a local process in the request path — a sandboxed or tightly managed runtime. And I would not put it in production without an eval harness measuring task success before and after. Compression that's invisible when it works is just as invisible when it fails.

The reframe

The shift worth making is to stop treating the context window as free space to fill and start treating it as a budget you manage. The model is a fixed cost; the tokens you feed it are not. Most teams are paying full freight to ship the model structural noise it skims and discards. Compressing that — carefully, reversibly, with the originals one retrieval away — is one of the few optimizations that cuts cost and latency at once, without touching the model or the answer.