Video walkthrough

Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks

Overview 3:59 CC AI

paperi.ai
0:00 / 0:00

Haoyu Zhao, Zhengxu Yu, Zhiyuan He, Meng Fang, Rasul Tutunov, Haitham Bou-Ammar, Weilin Luo, Jun Wang

What if an AI could enter a world it does not understand, write down its best guess about the rules, test that guess, and steadily replace it with something better, without changing the language model underneath?

Abstract

Learning to act in unfamiliar environments requires agents to infer how the world works and revise that understanding as new evidence arrives. Yet limited observations can support multiple world models that explain past interactions but predict different outcomes in unseen states. We introduce MEMENTO 3, building on the Memento series to enable frozen LLM agents to continually learn explicit world models through external memory. The agent maintains a natural-language rulebook as persistent semantic memory, recording revisable hypotheses about environment dynamics while leaving unknown aspects underspecified. It compiles this rulebook into executable code for prediction and planning. Through a continual loop of observation, reflection, rule revision, compilation, and verification, the agent uses prediction errors to refine both the rulebook and its code. Updated code is accepted only when the LLM judges it faithful to the rulebook and cell-exact replay reproduces the observed transitions. We investigate this process as a model-based route to recursive self-improvement (RSI): the agent autonomously explores the environment, revises its world model, and uses verified updates to guide subsequent interaction and learning, while the underlying LLM remains fixed. A population extension maintains multiple world models in parallel, sharing interaction evidence and using their predictions to guide exploration. On ARC-AGI-3, the single-model agent clears every level of all 25 public games, achieves a mean Relative Human Action Efficiency (RHAE) of 100.0, and uses 44% of the human action count. In an Atari Pong case study, a learned feedback controller wins 21 : 0 in each of three evaluated episodes with different openings, without further LLM calls. Advance To The Next Colour In Its Cycle.

Transcript

What if an AI could enter a world it does not understand, write down its best guess about the rules, test that guess, and steadily replace it with something better—without changing the language model underneath? Learning to act in unfamiliar environments requires an agent to infer how the world works and revise that understanding as new evidence arrives.

Limited observations can still support multiple explanations of the past that predict different outcomes in unseen states. The central idea is to keep those changing beliefs in a natural-language rulebook, then turn the rulebook into code that can predict what happens and help choose actions.

The rulebook records the agent’s current hypotheses about objects, actions, changes, and goals. Because those hypotheses are written down, later observations can be used to assess them and revise them. The code gives those written rules an executable form for prediction and planning.

The rulebook states the rules the agent believes, while execution fixes how those rules are applied. Learning a world model from interaction involves forming and revising hypotheses about an unfamiliar environment’s rules as new evidence arrives.

New observations drive revisions to the rulebook and the executable version, and the revised model guides subsequent interaction. The agent constructs possible rulebooks and executable versions from its interaction record. The language model expresses candidate rules in a rulebook and compiles them into a simulator with its own internal state.

Replay tests the simulator’s predicted observations against the history and reconstructs its current internal state. An accepted rulebook and executable version then supply the beliefs used for planning. The learning loop uses prediction errors to refine both the rulebook and its code.

Updated code is accepted only when it is judged faithful to the rulebook and replay reproduces the observed transitions. The underlying language model remains fixed while the agent explores, revises its world model, and uses verified updates to guide later interaction and learning.

The system can also maintain multiple world models in parallel. These models share interaction evidence and use their predictions to guide exploration. The point is not to trust one first guess forever, but to let several guesses be tested against the same evidence.

Both arms clear all nine levels and score RHAE one hundred when replayed against the live server. Compared with the verified single-model trajectory, the population reduces the total from eight hundred ninety-nine to five hundred ninety-seven scored actions and matches or improves per-level action efficiency on eight of the nine levels.

The system reaches the one-hundred ceiling on all twenty-five games, yielding a mean RHAE of one hundred point zero. A ceiling score requires every level to be cleared. The total action count for every game is below its human baseline. The controller reaches a perfect twenty-one to zero result from three different openings, while the same language model without an internal game model ends at minus nineteen.

Crucially, it plays this well without further language-model calls. In Atari Pong, the learned model supports a feedback controller that wins twenty-one to zero in three evaluated episodes with different openings, without language-model calls during execution. By carrying verified model updates into later planning and exploration, the agent uses what it has learned to shape the evidence for further revisions.

MEMENTO 3 turns experience into a revisable rulebook and working simulator, letting an AI use verified lessons to choose better actions later. That could make systems more dependable when they face unfamiliar situations.

A derivative work by Paperi · AI-generated script, voice and captions · pages and figures unaltered

Made with Paperi.

Drop in a research PDF — get a narrated video walkthrough like this one, with highlights that follow the narration. Free to start.

Try it with your paper →

More in Computer Science

Predictive Multiplicity in Cell-Fate Assignment: Label-Free Rashomon Sets and the Limits of Per-Cell Certification 4:24

Predictive Multiplicity in Cell-Fate Assignment: Label-Free Rashomon Sets and the Limits of Per-Cell Certification

A cell can receive radically different fate confidence from models that fit the same data equally well. The unsettling part is that adding more plausible models does not necessarily make any one cell’s fate call more trustworthy.

Paint outside the box: synthesizing and selecting training data for visual grounding 4:48

Paint outside the box: synthesizing and selecting training data for visual grounding

Teaching a computer to find the exact object described in a sentence usually demands vast amounts of carefully labeled pictures. This paper shows how carefully made artificial examples can help when those pictures are scarce.

ChatGPT’s feedback credibility and students’ perceived learning: the role of agent-related social perceptions 3:39

ChatGPT’s feedback credibility and students’ perceived learning: the role of agent-related social perceptions

When ChatGPT gives writing advice, students may trust it not only because of what it says, but because the system seems capable and understanding. That trust is linked to how much learning they feel has happened.

All 21 papers in Computer Science →