MemoryArena: Benchmarking Memory That Changes What Agents Do
An agent buys a camera body in one session. In the next session, it is asked to choose the highest-rated compatible lens. The new instruction lists the available lenses, but the right choice still depends on an earlier decision: which camera body was actually purchased.
The same dependency appears in other forms. A travel-planning agent must incorporate new preferences without breaking an itinerary created for previous travelers. A search agent must find an answer that satisfies every condition accumulated across several searches.
These tasks ask for more than recall. Remembering what happened is useful only if the remembered information changes what the agent does next.
MemoryArena evaluates this capability through interdependent, multi-session tasks in four environments: Bundled Web Shopping, Group Travel Planning, Progressive Web Search, and Sequential Formal Reasoning.
Two findings stand out. First, agents often make partial progress without completing the full dependency chain. Second, success generally becomes harder to sustain at later subtask depths.
These results preview the central challenge that MemoryArena is designed to examine: preserving past information is not the same as using it to sustain correct action across dependent sessions. Recall-oriented or single-session benchmarks do not directly test this full dependency chain. That gap is where MemoryArena begins.
1. Remembering Is Not Enough
1.1 The Gap Between Recall and Use
A memory system can preserve the answer to “What happened?” without helping an agent answer “What should I do now?”
Many memory benchmarks test whether information remains accessible across long conversations or documents. They ask models to retrieve facts, answer questions, or summarize information from an extended history. These evaluations provide a useful measure of recall, but they usually stop before the recalled information has to guide an action in a changing environment.
Interactive agent benchmarks test a different capability. They place models in environments where they must search, use tools, navigate interfaces, or make decisions based on feedback. These tasks evaluate action, but they are often organized as independent or single-session episodes. Persistent information from a previous session is not causally required for the next one.
What is missing is the intersection: an agent must act, observe the consequences, preserve the relevant state, and use that state in a later session whose correct solution depends on what happened before.
1.2 From Stored Facts to Future Decisions
MemoryArena treats recall as part of an ongoing decision process rather than as the endpoint of evaluation.
Simply sequencing independent tasks does not create a functional need for persistent memory. Later success must depend on state created earlier.
MemoryArena therefore makes later subtasks underspecified unless the agent retains and correctly uses information acquired during previous sessions. A lens cannot be selected correctly without knowing the camera body. A group itinerary cannot be extended correctly without preserving earlier plans and preferences. A final search answer cannot be verified without integrating conditions introduced by previous queries.
The question is therefore functional: did the system preserve the state that matters, make it available at the right time, and help the agent choose an action consistent with the full task?
2. Make the Past Change the Future
2.1 Interdependent, Not Merely Sequential
Each MemoryArena task is divided into subtasks that are executed in separate sessions. Within a session, the agent can take multiple actions and receive observations from the environment. When the session ends, however, its interaction trace is no longer directly available as ordinary session history.
Before acting on a later subtask, the agent retrieves relevant information from memory. Its actions produce new environment feedback, and the completed interaction updates the memory carried into the next session. Together, these steps form a Memory-Agent-Environment Loop that carries task-relevant state across otherwise isolated sessions.
2.2 Four Ways the Past Becomes a Constraint
MemoryArena instantiates cross-session dependence in four different forms:
Bundled Web Shopping: Earlier purchases determine which later products are compatible and which preferences remain satisfiable.
Group Travel Planning: New travelers add preferences and shared constraints that must remain consistent with plans already created for the group.
Progressive Web Search: Each subquery introduces another condition that the final answer must satisfy together with the previous conditions.
Sequential Formal Reasoning: Later mathematical or physical reasoning steps reuse definitions, intermediate results, or conclusions established in previous sessions.
The required state changes across these environments, but the evaluation principle is the same. Information from the past must remain precise and usable enough to constrain the current decision.
2.3 Memory Strategies Trade Preservation for Selection
These tasks generate long interaction histories. A MemoryArena task contains an average of 57 agent-environment action steps, the longest task settings reach 16 subtask sessions, and Progressive Web Search has an average task trace length of 122.4k tokens.
The benchmark evaluates three broad ways of carrying that history forward. Long-context configurations append prior interactions verbatim so the task agent can reason over the accumulated trace. External-memory systems use learned or curated mechanisms to abstract, consolidate, organize, and retrieve prior experience. Retrieval-augmented generation systems store previous information in an indexed collection and retrieve selected pieces for the current task.
Each approach makes a different tradeoff between preserving detail and selecting what is relevant. MemoryArena does not assume that any one of them should be universally best. Their usefulness depends on what state they preserve, what they return, and whether the task agent can use that representation effectively.
3. Agents Remember Pieces, but Lose the Task
3.1 Partial Progress Is Not Competion
Agents often completed individual subtasks without turning those local successes into a globally consistent solution.
Bundled Web Shopping makes this gap especially clear. Across the evaluated methods, the average Progress Score was 52%, while the average full-task Success Rate was only 2%. Agents could make valid purchases at individual stages, but completing the bundle required every accumulated compatibility and selection constraint to remain consistent.
Group Travel Planning was harder still. Every evaluated configuration recorded a 0% full-task Success Rate. Yet the across-method average soft Progress Score was 38%. This does not mean that 38% of complete itineraries succeeded. It means that agents frequently satisfied some of the many constraints inside individual planning subtasks while still failing to produce a fully valid group plan.
These results motivate three complementary measurements. Progress Score, or PS, measures the average fraction of ordered subtasks completed correctly within a task. Success Rate, or SR, measures whether the task reaches its full success criterion. For shopping and travel, that requires a completely valid bundle or group plan. For search and formal reasoning, it is determined by the correctness of the concluding subtask. Group Travel Planning additionally uses soft Progress Score, or sPS, to award partial credit for the fraction of itinerary constraints satisfied within each subtask.
3.2 No Memory Strategy Wins Everywhere
The results do not produce a simple ranking of memory strategies. The strongest single configuration in the evaluation, GPT-5.1-mini paired with Text-Embedding-3-Small retrieval, reached an all-task average Success Rate of 23%. Even this best overall result left most tasks unsolved.
The leading configurations differed across environments. Two long-context configurations tied for the highest Success Rate in Bundled Web Shopping, while retrieval-based configurations achieved the highest Success Rates in Progressive Web Search and in both Math and Physics Formal Reasoning.
These environment-level results do not identify a universally superior memory paradigm. Among the configurations and environments evaluated in MemoryArena, no strategy was consistently best.
3.3 Different Dependencies Create Different Failure Modes
The depth patterns previewed earlier add detail to this environment-level variation. In Progressive Web Search, where the average task trace reaches 122.4k tokens, long-context agents degraded more rapidly at later subtask positions. Configurations using external memory or retrieval generally declined more slowly as tasks progressed. This pattern suggests that selectively resurfacing prior information can help when a growing interaction history becomes difficult to use directly, but it does not establish a general advantage for either approach.
Other environments place greater emphasis on the precise reuse of previous state. Formal Reasoning may depend on exact intermediate results established in preceding sessions, while Group Travel Planning may depend on specific activities, time slots, and relationships between travelers. In these settings, retrieval-based configurations were more robust deeper into the dependency chain than external-memory systems that relied on heavier consolidation or abstraction. These are environment-specific observations, not evidence that retrieval is universally superior.
The authors interpret the broader decline at later subtask positions as possible belief drift. Small errors in an agent’s estimate of the evolving task state may accumulate across sessions and eventually affect later decisions. MemoryArena does not separately isolate belief drift as a causal mechanism, so this remains an interpretation of the observed trajectories rather than an independently verified finding.
Together, these patterns suggest that memory failures depend on the form of cross-session state each task requires. Aggregate results show where performance breaks down; a concrete trajectory can show how one missing detail changes the next action.
4. Memory Should Be Evaluated by What the Agent Does Next
4.1 When One Missing Detail Changes the Action
A Bundled Web Shopping case shows how an apparently small retrieval omission can directly change an agent’s behavior.
Earlier in the task, the agent had purchased a Sony soundbar described as Compact. In a later session, it needed to buy a TV wall mount. The task rules stated that Compact soundbars pair well with Articulating mounts and should avoid Low Profile mounts.
A long-context configuration retained the purchase history, identified the Compact attribute, and selected an Articulating mount. In the BM25 case, the retrieved context contained available products and compatibility notes but omitted the previous purchase details that identified the soundbar as Compact.
This example should not be generalized into a claim about BM25’s overall performance. BM25 achieved the highest Success Rate in Progressive Web Search among the reported configurations. The case illustrates a narrower point: retrieved context can look relevant while still omitting the specific piece of state that the next decision requires.
4.2 Memory and Agents Must Work Together
Failures in MemoryArena cannot always be assigned cleanly to the memory system or the task agent.
On the memory side, systems optimized for generic recall, compression, or semantic similarity may fail to preserve and update the task-relevant state needed for a particular dependency chain. A summary can remove an exact attribute. A retrieval system can return related material without returning the decisive event. A structured representation can preserve relationships that the task agent does not know how to use.
On the agent side, the model may formulate an ineffective retrieval query, misinterpret the returned information, or fail to incorporate it into its reasoning. Even when useful evidence is available, the agent still has to recognize how that evidence changes its current view of the task.
This coordination problem motivates closer co-design of memory representations and task agents. It does not establish that a particular joint-training method or memory architecture will solve the problem.
Beyond functional integration, practical systems must also account for execution cost: external-memory agents had the highest latency in these experiments, long-context agents the lowest, and RAG systems fell between them, while memory-structure complexity alone did not consistently predict latency within the external-memory and RAG groups.
4.3 The Right Lens Is the Real Test
Return to the camera bundle.
Remembering which camera body was purchased is necessary, but it is only the beginning. A useful memory system must preserve the attribute that matters for compatibility. It must surface that attribute when the lens decision is made. The agent must then use it to change the action and select a lens that is both preferred and compatible.
This is why memory for acting agents cannot be evaluated only by inspecting what they recall. Its value appears in the decisions that follow.
For an acting agent, remembering the camera is not the finish line. Choosing the right lens is the test.