Is complex agentic memory always needed? A naive RAG baseline says often not

Over the last two years, agent memory has turned into its own engineering discipline. We have systems that extract atomic facts, mutate them with UPDATE and DELETE operations, consolidate them into hierarchies, build knowledge graphs, and run iterative search at recall time. The assumption underneath all of it is that plain retrieval over the raw transcript is not good enough.

A new paper from Bao, Zhang, and colleagues, Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought, tests that assumption directly, and for the conversation tasks it evaluates, the assumption mostly fails. Their baseline, Raw-RAG, does no LLM processing at ingestion at all. Across LoCoMo, LongMemEval, MEME, and MemoryAgentBench, it ranks first among the compared memory systems in six of seven benchmark-model settings, beating the strongest agentic system by 0.13 to 5.53 points.

I think this is one of the more useful memory papers of the year. That is not because the headline is surprising. It is because the diagnostics explain why the simple thing works.

What Raw-RAG actually is

The pipeline is deliberately boring:

• Serialize each conversation with message timestamps (and image captions where present).
• Split it into 256-token windows with 32-token overlap and embed them into a flat index.
• At query time, retrieve the 128 nearest windows by cosine similarity.
• Score each question-window pair with a cross-encoder reranker and keep the top 15.
• Expand those 15 windows to complete turns (the LoCoMo configuration) and hand the raw text to the reader LLM, which works out temporal order and state changes itself.

There is no extraction, no summarization, no graph, and no mutation. All the state reconstruction happens at read time, inside the reader.

The structural difference is where the work happens. Agentic systems spend LLM calls at write time. Raw-RAG defers everything to the reader. On the near-million-token BEAM-1M stream, Raw-RAG ingests in 0.33 hours, against 3.25 hours for the next-fastest completed system and 26.95 hours for Memora.

The mutation audit is the part I keep thinking about

Before the main comparison, the authors audit Mem0's write path on LoCoMo and LongMemEval. UPDATE and DELETE together make up only 4.39% of 33,902 write decisions on LoCoMo and 2.46% of 428,858 on LongMemEval. Many of those are duplicates or malformed. Mutations that actually persist a text change come to 1.90% and 1.34% of decisions.

Then they switch mutation off: UPDATE becomes append-only ADD, and DELETE becomes a no-op. LoCoMo accuracy rises, from 67.36% to 69.42%, and no category gets worse. In this configuration, the machinery that justifies agentic memory, explicit state maintenance, adds nothing measurable.

The obvious objection is that real conversations contain superseded facts, so something has to track current state. The paper answers that with MEME, a benchmark built around deletions, cascades, and invalidations. When readers get the phase-visible raw evidence in chronological order, DeepSeek-V4-Flash reaches 82.32% on cascades and 91.54% on invalidation. A capable reader can resolve state changes from raw history. The important caveat is in the same table: GPT-4o-mini scores 1.83% on cascades. Reader capability matters a lot.

Why the simple baseline wins: coverage, not cleverness

The most useful idea in the paper is splitting memory performance into two quantities:

• Coverage: the fraction of questions for which the retrieved context contains sufficient evidence.
• Utilization: accuracy given that sufficient evidence was supplied.

Raw-RAG is the bottom bar in each group. On MemoryAgentBench, systems differ far more in whether they surface the evidence than in what the reader does with it.

On MemoryAgentBench, coverage spans 19.07% to 45.29% across systems, with Raw-RAG highest. Utilization spans only 80.00% to 89.71%, and Raw-RAG is mid-pack at 84.43%. MEME shows the same pattern. Raw-RAG does not read better. It retrieves the right material more often, and that dominates.

Two further findings sharpen this:

1. Paths, not just values. On MEME tracking and cascade questions, retrieving the answer value is not enough. The reader also needs the chronology or dependency chain that makes that value current. Conditional on retrieving the value, Raw-RAG also retrieves the full supporting path 62.34% of the time. MemMachine, which also retains raw content, reaches 56.70%. MemU reaches 9.62%. Extraction keeps the value and drops the relation.
2. Better retrieval does not close the gap. With a gold-informed oracle selecting from each system's complete store, all seven stores still fall 7.58 to 29.38 points short of raw evidence once the packing cap is removed. The two source-preserving systems come closest. The authors are careful here: this residual is not a pure measure of ingestion loss, since even Raw-RAG falls short, and evidence assembly and positional effects such as lost-in-the-middle may also contribute.

As a physicist, I read this as a data-processing inequality applied to memory. Each lossy transform at write time can only reduce what is available downstream. It pays off only when the reader cannot use the raw signal, and modern readers mostly can.

The ablations tell you which knobs matter

The paper also works as a tuning guide for anyone running RAG over conversation logs:

• The reranker is required. Removing it costs 3.92 points on MemoryAgentBench.
• Retrieval depth has a broad sweet spot. Final k of 5 to 60 stays within 1.3 points of k=15. Returning a single item costs 11.73 points, and top-120 costs 14.57.
• Chunk size barely matters. Windows from 128 to 1,024 tokens show no significant change.
• Bolting memory management onto Raw-RAG hurts. Atomic-fact extraction costs 19.62 points, and test-time learning collapses from 47.68% to 15.37%. Summary augmentation costs 5.81 points and graph expansion 5.78. These additions also take 2.9 to 8.8 hours and 21.6M to 109.3M LLM tokens, against 12 minutes for Raw-RAG.

The authors also review 14 prior evaluations and find that the RAG baselines there often used coarse retrieval units, shallow depth, or no reranking. Part of the gap that has been reported between RAG and agentic memory looks like a weak-baseline artifact.

Where I would still add structure

The scope is clear, and the authors state it: offline conversation benchmarks, mostly strong readers. In production, I would still reach for something beyond Raw-RAG in a few cases:

• Deletion and privacy obligations. When a user asks you to forget something, an append-only raw log becomes a liability. You need provenance and real deletion, which is an engineering requirement rather than a retrieval one.
• Small or cheap readers. The MEME numbers show that weaker models cannot reliably resolve cascading state from raw text. If your reader is a small model, materialized state may be worth its cost.
• Repeated read-time reasoning. Raw-RAG re-derives current state on every query. For high-QPS agents that ask about the same entity repeatedly, caching derived state can win on latency and cost even when it does not win on accuracy.
• One benchmark-model setting already goes the other way. MemMachine leads on DeepSeek-V4-Flash MEME, and full context beats Raw-RAG there by nearly 20 points. Long, dependency-heavy histories remain open territory.

The practical takeaway

If you are building agent memory, start with Raw-RAG: timestamped raw history, dense top-128, cross-encoder rerank to about 15, and a capable reader. Measure coverage and utilization separately on your own traffic. Add extraction, mutation, or graphs only when you can show they raise coverage or path retention over that baseline, and write down what each addition costs at ingestion.

The authors close with the right standard: new memory systems should demonstrate gains over a well-specified naive RAG baseline. I would apply the same rule to internal designs. In the evaluated settings, raw history plus good retrieval was the stronger memory, and complexity has to beat that.