Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks
Overview3:59CCAI
paperi.ai
0:00 / 0:00
Haoyu Zhao, Zhengxu Yu, Zhiyuan He, Meng Fang, Rasul Tutunov, Haitham Bou-Ammar, Weilin Luo, Jun Wang
What if an AI could enter a world it does not understand, write down its best guess about the rules, test that guess, and steadily replace it with something better, without changing the language model underneath?
Learning to act in unfamiliar environments requires agents to infer how the world works and revise that understanding as new evidence arrives. Yet limited observations can support multiple world models that explain past interactions but predict different outcomes in unseen states. We introduce MEMENTO 3, building on the Memento series to enable frozen LLM agents to continually learn explicit world models through external memory. The agent maintains a natural-language rulebook as persistent semantic memory, recording revisable hypotheses about environment dynamics while leaving unknown aspects underspecified. It compiles this rulebook into executable code for prediction and planning. Through a continual loop of observation, reflection, rule revision, compilation, and verification, the agent uses prediction errors to refine both the rulebook and its code. Updated code is accepted only when the LLM judges it faithful to the rulebook and cell-exact replay reproduces the observed transitions. We investigate this process as a model-based route to recursive self-improvement (RSI): the agent autonomously explores the environment, revises its world model, and uses verified updates to guide subsequent interaction and learning, while the underlying LLM remains fixed. A population extension maintains multiple world models in parallel, sharing interaction evidence and using their predictions to guide exploration. On ARC-AGI-3, the single-model agent clears every level of all 25 public games, achieves a mean Relative Human Action Efficiency (RHAE) of 100.0, and uses 44% of the human action count. In an Atari Pong case study, a learned feedback controller wins 21 : 0 in each of three evaluated episodes with different openings, without further LLM calls. Advance To The Next Colour In Its Cycle.
Transcript
What if an AI could enter a world it does not understand, write down its best guess about the rules, test that guess, and steadily replace it with something better—without changing the language model underneath? Learning to act in unfamiliar environments requires an agent to infer how the world works and revise that understanding as new evidence arrives.
Limited observations can still support multiple explanations of the past that predict different outcomes in unseen states. The central idea is to keep those changing beliefs in a natural-language rulebook, then turn the rulebook into code that can predict what happens and help choose actions.
The rulebook records the agent’s current hypotheses about objects, actions, changes, and goals. Because those hypotheses are written down, later observations can be used to assess them and revise them. The code gives those written rules an executable form for prediction and planning.
The rulebook states the rules the agent believes, while execution fixes how those rules are applied. Learning a world model from interaction involves forming and revising hypotheses about an unfamiliar environment’s rules as new evidence arrives.
New observations drive revisions to the rulebook and the executable version, and the revised model guides subsequent interaction. The agent constructs possible rulebooks and executable versions from its interaction record. The language model expresses candidate rules in a rulebook and compiles them into a simulator with its own internal state.
Replay tests the simulator’s predicted observations against the history and reconstructs its current internal state. An accepted rulebook and executable version then supply the beliefs used for planning. The learning loop uses prediction errors to refine both the rulebook and its code.
Updated code is accepted only when it is judged faithful to the rulebook and replay reproduces the observed transitions. The underlying language model remains fixed while the agent explores, revises its world model, and uses verified updates to guide later interaction and learning.
The system can also maintain multiple world models in parallel. These models share interaction evidence and use their predictions to guide exploration. The point is not to trust one first guess forever, but to let several guesses be tested against the same evidence.
Both arms clear all nine levels and score RHAE one hundred when replayed against the live server. Compared with the verified single-model trajectory, the population reduces the total from eight hundred ninety-nine to five hundred ninety-seven scored actions and matches or improves per-level action efficiency on eight of the nine levels.
The system reaches the one-hundred ceiling on all twenty-five games, yielding a mean RHAE of one hundred point zero. A ceiling score requires every level to be cleared. The total action count for every game is below its human baseline. The controller reaches a perfect twenty-one to zero result from three different openings, while the same language model without an internal game model ends at minus nineteen.
Crucially, it plays this well without further language-model calls. In Atari Pong, the learned model supports a feedback controller that wins twenty-one to zero in three evaluated episodes with different openings, without language-model calls during execution. By carrying verified model updates into later planning and exploration, the agent uses what it has learned to shape the evidence for further revisions.
MEMENTO 3 turns experience into a revisable rulebook and working simulator, letting an AI use verified lessons to choose better actions later. That could make systems more dependable when they face unfamiliar situations.
A derivative work by Paperi · AI-generated script, voice and captions
· pages and figures unaltered
Made with Paperi.
Drop in a research PDF — get a narrated video walkthrough like this one,
with highlights that follow the narration. Free to start.
A cell can receive radically different fate confidence from models that fit the same data equally well. The unsettling part is that adding more plausible models does not necessarily make any one cell’s fate call more trustworthy.What if several cell-fate models fit the data equally well, yet disagree about individual cells? This paper finds that model multiplicity is real, and that certifying a fate across all those models can make predictions worse.
Teaching a computer to find the exact object described in a sentence usually demands vast amounts of carefully labeled pictures. This paper shows how carefully made artificial examples can help when those pictures are scarce.What if the safest way to generate a labeled object image is not to redraw the object at all? This paper paints outside the box, then filters the synthetic examples before training.
Raúl Pérez‐López, Rubén González-Vallejo, Laura Lucia‐Palacios, Iguácel Melero‐Polo, Andreea Trifu
When ChatGPT gives writing advice, students may trust it not only because of what it says, but because the system seems capable and understanding. That trust is linked to how much learning they feel has happened.ChatGPT’s feedback may feel more credible when the system seems competent and empathetic, but that credibility does not matter equally to every student. This study traces how social impressions become learning perceptions.