Persistent capability in fixed-model agents

A conventional fixed-goal agent run begins with an objective, produces a reasoning trace and some actions, then terminates. Even if the trace contains a useful procedure, the next run usually receives it only if a human or a separate memory mechanism chooses to carry it forward. Voyager makes that carry-forward operation part of the controller. A successful episode emits an executable artifact, a semantic description of the artifact, and a completed-task record. A failed episode emits a failure record. Both alter subsequent decisions.

That is why the paper uses the phrase in-context lifelong learning. The adaptation is not parametric: there is no gradient step and no weight update. Instead, later model calls receive a different effective action vocabulary because the prompt can include programs that earlier calls discovered and verified. They also receive a different decision context because the world and exploration history have changed. The base model is fixed; the system presented to the model is not.

This relocates the learning problem. Voyager is not trying to teach GPT-4 the concept of a furnace from scratch. GPT-4 already carries substantial Minecraft and programming knowledge. The system’s job is to turn that prior knowledge into grounded procedures, decide which procedure is worth acquiring next, reject procedures that fail in execution, and expose successful procedures as dependencies for later synthesis. In software terms, it is closer to a continuously populated build environment than to an online-trained policy network.

Why was this important? It demonstrated a credible route from isolated LLM competence to accumulating operational capability without fine-tuning after every experience. The durable object is not a transcript saying what worked; it is code that can run again. That made curriculum, artifact memory, and verification look like one control architecture rather than three optional agent features.

Let s_t denote the structured Minecraft state at iteration t: inventory, equipment, position, nearby blocks and entities, biome, health, hunger, time, and related observations. Let H_t contain completed and failed tasks, and let L_t be the executable skill library. The curriculum C receives selected parts of s_t and H_t and emits one immediate goal g_t. This is the outer loop: decide what capability should be acquired next.

The inner loop then tries to close that goal. The generator G receives g_t, s_t, retrieved programs from L_t, and the accumulated execution trace τ_t. It emits a JavaScript program p_t. The environment transition T executes p_t against s_t and returns a new state plus chat and interpreter evidence. A verifier V judges whether the resulting state satisfies g_t. That verdict selects one of three transitions: commit, retry, or abandon.

The write-backs create the learning dynamics. Commit adds p_t to L_t under a generated semantic description and appends g_t to completed history. Retry extends τ_t with environment messages, exceptions, prior code, and critique before calling G again. After four unsuccessful synthesis rounds, abandon appends the task—not the full retry trace—to failed history. Both success and exhausted failure therefore alter the next call to C.

Now the design pressure is visible. If C proposes goals without regard to state, G burns its budget on missing prerequisites. If G produces behavior without a persistent artifact, the system repeatedly pays for the same reasoning. If V cannot distinguish execution from task completion, bad programs enter L_t and become dependencies. Voyager works only if scheduling, synthesis, execution, verification, and memory share explicit state transitions.

Consider the paper’s concrete state: the agent owns a furnace, four raw iron, one coal, seven oak planks, a crafting table, and a stone pickaxe. “Craft an iron pickaxe” is directionally valuable but skips a state transition because the inventory contains raw iron rather than ingots. “Smelt four raw iron” sits on the frontier: it is novel, immediately feasible, and its success appears directly in inventory. The curriculum’s job is to find that narrow band between tasks already mastered and tasks whose prerequisites are missing.

Its prompt receives four classes of input. First are directives: discover diverse things, avoid repetition, prefer tasks just beyond current capability, and return one concrete task in a constrained verb-object form. Second is current state. Third is completed and failed history, which serves as an external estimate of capability and difficulty. Fourth is additional Minecraft context: GPT-3.5 generates five to ten questions about the current situation, can retrieve relevant wiki material, answers the questions, and gives the answers to GPT-4.

Verifiability is a genuine scheduling constraint, not documentation polish. The task must produce an observable postcondition through the available text state. “Acquire one iron ingot” can be checked in inventory. A spatial arrangement may be impossible to verify without vision. The curriculum is therefore choosing over the intersection of useful tasks and judgeable tasks; anything outside that intersection cannot participate safely in the commit loop.

The state representation is warmed up. At zero completed tasks the prompt includes core inventory, equipment, nearby blocks, and position. Nearby entities appear after five tasks, full inventory after seven, recently seen blocks and biome after ten, and health, hunger, time, and additional context after fifteen. This is not learning inside C. It is an authored observation schedule that withholds dimensions until they are likely to affect decisions, reducing early prompt noise.

All model calls use temperature 0 except the curriculum, which uses 0.1. Code generation and verification are biased toward repeatability; task selection is allowed a small amount of variation so exploration does not collapse onto one locally plausible sequence. Put together, the scheduler is state-conditioned, history-conditioned, constraint-heavy, and slightly stochastic. Remove one of those properties and the frontier estimate changes.

In hierarchical control terms, a generated function behaves like an option or macro-action: it has an intended outcome, runs for a variable duration, and may invoke lower-level procedures. That compression is essential. A primitive-action sequence becomes invalid after one navigation or inventory mismatch, and preserving the token trace does not create a reliable callable behavior. A named function can check preconditions, branch on state, invoke dependencies, and be executed again.

Mineflayer supplies the abstraction boundary. The prompt exposes helpers such as mineBlock, craftItem, smeltItem, placeItem, and killMob, plus selected lower-level APIs. GPT-4 is instructed to reuse those helpers rather than bypass them, inspect inventory before consuming resources, avoid infinite loops and event listeners, call exploreUntil when targets are missing, report intermediate progress through bot.chat(), and make functions generic enough to become dependencies of later functions.

The code model is therefore not called with a task in isolation. Voyager assembles a build context: generation rules; control primitives and API documentation; five retrieved skill programs; the last generated code; chat emitted during execution; interpreter errors; critic feedback; live agent state; and the task plus generated context. The required response has three parts: explain what failed or is missing, plan the state transitions, then emit one async function taking bot as its only argument.

That resembles an incremental compiler pipeline more than unconstrained tool use. Platform rules and APIs define the target runtime. Retrieved programs act as linked dependencies. Previous code plus runtime feedback form the failed build. GPT-4 performs diagnosis and emits a new artifact. Execution is the integration test. The analogy is imperfect—there is no type checker for semantic preconditions—but it makes clear why prompt composition is part of the architecture.

The benchmark also makes synthesis unusually recoverable. On death the bot respawns near the closest ground and retains inventory. Crafting tables and furnaces are recycled after execution. The world is sandboxed, and the API removes pixel perception and low-level motor control. Code succeeds as a learning substrate partly because failure is cheap and the action interface is broad, inspectable, and forgiving.

Write path first. Once V declares a task successful, GPT-3.5 generates a short description of the main function. Voyager embeds that description with text-embedding-ada-002. The description embedding is the key; the verified JavaScript program is the value. Natural language supplies a semantic address, while source code remains the executable payload. The system never embeds a trajectory and hopes the model can reconstruct the procedure later.

On the read path, GPT-3.5 proposes a general solution to the new task. Voyager combines that solution with environment feedback to create query context, embeds the context, and ranks stored description embeddings. For “craft an iron pickaxe,” the retrieved set includes Smelt Iron Ingot, Craft Stick, Make Crafting Table, Make Furnace, and Craft Wooden Pickaxe. That mixture contains direct prerequisites, a prerequisite of a prerequisite, and an analogous recipe.

The retrieval result explains an important systems choice. Appendix B evaluates 309 samples: top-1 accuracy is 80.2%, while top-5 reaches 96.5%. Voyager therefore spends prompt budget to increase recall. The retriever does not have to identify the single correct dependency; it has to place useful candidates in context. GPT-4 then performs the final relevance filtering and composition while generating the new program.

This is stronger than a textual episodic memory because reuse is operational: the returned value can execute. It is weaker than a package system. Skills have no typed preconditions or postconditions, dependency versions, provenance policy, compatibility tests, stale-artifact detector, or retirement process. Semantic similarity also says nothing about whether two programs can safely compose against the current state.

The library avoids catastrophic forgetting only in a narrow external-memory sense: stored functions do not vanish because no weights are updated. It can still suffer semantic collision, contaminated writes, brittle dependencies, or prompt-context pressure. The durable pattern is not “add a vector database.” It is “semantic address, executable payload, high-recall retrieval, generator-side composition, verification-gated write.”

Environment messages describe intermediate effects inside an otherwise valid program. The supplied helpers and generated code use bot.chat() to report progress or missing prerequisites: “I cannot make stick because I need 2 more planks.” That message means the program reached a meaningful branch but the planned resource state was wrong. The next synthesis round should add a collection or crafting dependency.

Interpreter errors answer a different question: did the program use JavaScript, Mineflayer, and the Minecraft object model correctly? “No item named acacia_axe” is not a missing-resource problem. The requested object does not exist. Figure 5 shows GPT-4 replacing the invented acacia axe with a wooden axe. Runtime validity and task progress must remain separate because they induce different repairs.

The GPT-4 critic evaluates the goal postcondition. It receives the task and resulting structured state, then asks whether the requested outcome is now true. It is not given an independent before/after trajectory. A program may terminate without exceptions and still fail this test. Conversely, the critic can infer success from inventory evidence—for example, coal after a coal-mining task. V is therefore the promotion gate between an executed artifact and persistent memory.

Each new round receives previous code, environment messages, the interpreter trace, and critic critique. After critic-confirmed success, Voyager stores the function and marks the task completed. After at most four failed generation rounds, it abandons the task, records failure, and returns control to the curriculum. The bounded retry policy prevents a single hard objective from consuming the open-ended exploration budget.

The gate is learned and correlated. GPT-4 participates in synthesis and another GPT-4 call judges success from the same textual abstraction. The paper records verifier mistakes, including failure to treat spider string as evidence that a spider was killed. A false negative wastes work; a false positive is worse because it can insert a mislabeled program into L_t and contaminate future compositions. Production versions need deterministic assertions or independent evaluators wherever possible.

At t=0 the paper’s state contains a furnace, four raw iron, coal, seven oak planks, a crafting table, and a stone pickaxe. The aspirational objective “craft an iron pickaxe” is not yet executable because the recipe requires ingots. C selects “smelt four raw iron.” This is a better immediate goal because its inputs are present, its output is observable in inventory, and completion moves the dependency graph toward the aspirational objective.

G then receives that task, current state, the prompt rules, and retrieved skills. Useful candidates may include smelting, furnace creation, stick crafting, crafting-table creation, and an analogous pickaxe recipe. GPT-4 emits a function that checks for a furnace and fuel, places or reuses the furnace, calls the smelting helper, and emits progress through chat. T executes the program; inventory changes from raw iron plus coal to iron ingots.

V judges the scheduled task, not the eventual pickaxe goal. If four ingots now exist, the smelting postcondition holds. The system commits the function under a generated description and adds the smelting task to completed history. That single commit creates three durable changes: s_t contains the needed material, H_t records the frontier step as mastered, and L_t contains a reusable smelting procedure.

On the next outer-loop iteration, “craft an iron pickaxe” can move onto the feasible frontier: four ingots are present, seven oak planks can produce the two required sticks, and the crafting table already exists. The generator can retrieve the newly committed smelting function together with stick and pickaxe procedures. This is the compounding mechanism in concrete form: a verified program changes both the resources in the world and the code available to interpret the next task.

Failure also has a role. If smelting exhausts four attempts, the failed-task record tells C not to immediately repeat the same objective under equivalent conditions. Voyager does not learn an explicit value function from that failure, nor does it preserve the full failed execution trace in curriculum history; it supplies the unsuccessful task label to a language-model scheduler. The control benefit is real, but it depends on the model interpreting that coarse history sensibly.

The main system uses gpt-4-0314 for curriculum, code generation, and self-verification; gpt-3.5-turbo-0301 for context and skill descriptions; and text-embedding-ada-002 for retrieval. The environment is MineDojo controlled through Mineflayer. Each main method receives 160 prompting iterations and is run for three trials. A prompting iteration is a model call in code generation, not a normalized environment step, token count, dollar cost, or wall-clock duration.

ReAct, Reflexion, and AutoGPT did not operate in this setting out of the box, so the authors adapted them to structured state and executable code. ReAct receives environment feedback and agent state. Reflexion additionally receives execution errors and Voyager’s self-verification module. AutoGPT receives state, feedback, and errors while decomposing the top-level exploration goal, but lacks Voyager’s automatic curriculum, persistent skill library, and explicit task-success gate. These are system-level comparisons, not untouched reference implementations.

Within that contract, Voyager discovers 63 unique items in 160 prompts—3.3 times the evaluated baselines—and travels 2.3 times farther than the baselines. It reaches wooden, stone, and iron tools in 6 ± 2, 11 ± 2, and 21 ± 7 prompting iterations across successful trials. AutoGPT requires 92 ± 72, 94 ± 72, and 135 ± 103. Diamond is not a reliable success: Voyager reaches it once in three trials at prompt 102; every other method is 0/3.

Map coverage adds a separate behavioral signal. An item counter alone could reward local crafting chains. The larger traversal envelope shows that Voyager leaves local regions and encounters a wider distribution of biomes and resources. It does not prove better navigation in a low-level sense because Mineflayer supplies navigation primitives; it shows that the high-level controller schedules behavior that results in broader movement.

The transfer experiment is the cleanest evidence for executable memory. The authors clear inventory, create a new world, provide four unseen tasks, and allow at most 50 prompting iterations. Full Voyager solves diamond pickaxe, golden sword, lava bucket, and compass in all three trials, at 19 ± 3, 18 ± 7, 21 ± 5, and 18 ± 2 iterations. Voyager without the library is generally slower and misses one diamond-pickaxe trial. Giving Voyager’s library to AutoGPT improves AutoGPT, although it still solves tasks less reliably. Because the world and inventory reset while the programs persist, the intervention isolates transferable value in the artifacts better than the main exploration curve does.

Replacing the automatic curriculum with random item goals reduces discovered items by 93%. The random sampler draws from 101 items that Voyager discovered, so it possesses a plausible goal vocabulary. What it lacks is prerequisite ordering and state fit. A manual diamond-oriented curriculum does better than random but plateaus early because it is narrow and world-agnostic. The contrast supports C as a frontier estimator rather than merely a source of varied task text.

Removing L_t produces the most diagnostic shape: early acquisition remains competitive, then the curve flattens. Early Minecraft tasks are simple enough for GPT-4 to synthesize from prior knowledge and current prompt context. Later tasks require longer compositions and repeated prerequisites. Without persistent verified code, each call rebuilds those procedures. The plateau is the signature of missing accumulation, not simply worse first-attempt intelligence.

The feedback ablations distinguish repair from promotion. Removing environment feedback or interpreter errors harms discovery because G loses localized evidence about intermediate effects and program validity. Removing self-verification is most damaging among the feedback channels, reducing discovery by 73%, because the controller loses the goal-level decision that determines retry versus advance and protects the write path.

Replacing GPT-4 with GPT-3.5 for code generation yields 5.7 times fewer unique items, while GPT-4 remains in curriculum and verification. That localizes the gap to executable synthesis within this architecture. The result is a ceiling condition: retrieval, feedback, and scheduling amplify a strong generator, but they do not make generator quality interchangeable. A repair loop cannot compound if it rarely produces valid artifacts worth committing.

These are component ablations, not a complete causal decomposition. Prompt wording, state representation, retrieval count, model prior, verifier correlation, and the forgiving runtime are not independently crossed in a factorial design. The safe conclusion is that the shipped bundle’s curriculum, program memory, feedback, verifier, and strong code model are jointly load-bearing in this benchmark—not that each quoted percentage transfers to another domain.

Inside the measured system are structured text observations, high-level Mineflayer actions, LLM task scheduling, JavaScript synthesis, execution feedback, GPT-4 verification, and external program memory. Outside it are pixels, raw sensory inference, low-level motor control, weight updates, independent ground-truth judging, and expensive irreversible consequences. The paper advances the first stack. It deliberately does not solve the second.

Grounding failures remain visible. The curriculum can propose nonexistent copper swords; the code generator invents APIs or treats cobblestone as fuel. The same pretrained domain prior that makes GPT-4 productive can produce confident false assumptions. A proprietary scientific or enterprise domain would need authoritative retrieval, typed resource models, and stronger feasibility checks before C could distinguish a novel task from a fictional one.

The library needs package semantics. A production artifact should carry typed preconditions and postconditions, dependencies, tests, provenance, authority requirements, version compatibility, observed success rates, and retirement rules. Retrieval should filter on those contracts before semantic similarity. A verified function should run inside a sandbox with resource limits, scoped credentials, auditable effects, idempotency or rollback, and an explicit blast radius.

Verification should be decomposed further. Deterministic invariants should judge what code and state can express exactly. Independent models or humans should handle ambiguous semantic outcomes. A learned judge should not be the sole authority for promoting an artifact that later receives broader permissions. Four retries limit cost; they do not establish safety.

With those qualifications, the architecture remains durable: maintain an adaptive backlog from live state; compile work into executable artifacts; surface real execution evidence; distinguish runtime completion from goal satisfaction; and admit artifacts to persistent memory only through a promotion gate. Voyager’s lasting contribution is showing that these pieces can create system-level capability accumulation even while the foundation model remains unchanged.