Persistent capability in fixed-model agents — Voyager

Abstract

This chapter explains why Voyager mattered as an agent architecture: it moved adaptation out of model weights and into an explicit controller whose persistent state consists of task history, changed world state, and verified executable programs. A visual systems reconstruction preserves the paper's appendix-level implementation details while making every write-back and promotion decision explicit.

The treatment is aimed at advanced engineering readers. It formalizes the controller, derives a capability-frontier interpretation of the curriculum, traces skill-library retrieval and admission, reconstructs the iron-pickaxe example, and reads the experiments as evidence under a narrow software-mediated benchmark rather than as a claim of general embodied intelligence.

After studying this chapter, you should be able to:

Objective 1: Locate Voyager’s adaptation in external system state and explain why this is not model training.
Objective 2: Reconstruct the curriculum, generator, environment, verifier, history, and skill-library dataflow.
Objective 3: Explain why executable programs serve simultaneously as actions, temporal abstractions, and memory artifacts.
Objective 4: Separate environment feedback, interpreter errors, and task verification by the contract each evaluates.
Objective 5: Read the exploration, transfer, and ablation results without exceeding the benchmark’s actual scope.
Objective 6: Translate the architecture into a safer artifact-learning system outside Minecraft.

Notation used in this chapter
Symbol	Meaning	Type
$\theta$	Parameters of the pretrained language model	fixed model state
$s_t$	Structured Minecraft state at outer-loop iteration t	environment state
$H_t$	Completed-task and failed-task history	persistent control state
$L_t$	Library of verified executable programs	persistent artifact state
$g_t$	Immediate task proposed by the automatic curriculum	goal
$p_t$	JavaScript program synthesized for the current task	action program
$e_t$	Environment messages and interpreter evidence produced by execution	runtime evidence
$\tau_t$	Within-task repair context: prior code, messages, errors, and critique	ephemeral trace
$v_t$	Self-verifier judgment for the current task	Boolean + critique
$C$	Automatic-curriculum task policy	controller
$G$	GPT-4 code-generation policy	generator
$T$	Mineflayer-mediated environment transition	runtime
$V$	GPT-4 task-success verifier	critic
$R_t$	Top-five programs retrieved for the current task	dependency set
$d_i$	Generated natural-language description of stored program i	semantic address
$E$	text-embedding-ada-002 encoder used for skill retrieval	embedding model

1.1 Locate the learning

Adaptation outside the model parameters

Before unpacking any module, hold onto one distinction: GPT-4 does not learn Minecraft during the experiment. The models are fixed. The surrounding control system accumulates task history, changed world state, and verified JavaScript programs.

Derived comparison · source-grounded

Ordinary fixed-goal run

goalpromptLLMactiontrace

RUN TERMINATES

The trace may contain useful behavior, but it is not promoted into the next run's callable context.

behavioral capability at t + 1

Voyager across runs

GPT-4 modelweights θ fixed

EXTERNAL SYSTEM STATE

History · completed + failed tasks
World state · inventory + equipment + observations
Library · verified executable programs

t → t + 1Each verified run enlarges what the unchanged model can schedule, retrieve, and call.

Figure 1.1. Where adaptation is stored. Voyager accumulates capability in external, executable state: history changes the curriculum, world state changes feasibility, and verified programs change the action vocabulary available to the next fixed-model call.Study question. Where does learning happen if the model weights never change?

A conventional fixed-goal agent run begins with an objective, produces a reasoning trace and some actions, then terminates. Even if the trace contains a useful procedure, the next run usually receives it only if a human or a separate memory mechanism chooses to carry it forward. Voyager makes that carry-forward operation part of the controller. A successful episode emits an executable artifact, a semantic description of the artifact, and a completed-task record. A failed episode emits a failure record. Both alter subsequent decisions.

That is why the paper uses the phrase in-context lifelong learning. The adaptation is not parametric: there is no gradient step and no weight update. Instead, later model calls receive a different effective action vocabulary because the prompt can include programs that earlier calls discovered and verified. They also receive a different decision context because the world and exploration history have changed. The base model is fixed; the system presented to the model is not.

This relocates the learning problem. Voyager is not trying to teach GPT-4 the concept of a furnace from scratch. GPT-4 already carries substantial Minecraft and programming knowledge. The system’s job is to turn that prior knowledge into grounded procedures, decide which procedure is worth acquiring next, reject procedures that fail in execution, and expose successful procedures as dependencies for later synthesis. In software terms, it is closer to a continuously populated build environment than to an online-trained policy network.

Why was this important? It demonstrated a credible route from isolated LLM competence to accumulating operational capability without fine-tuning after every experience. The durable object is not a transcript saying what worked; it is code that can run again. That made curriculum, artifact memory, and verification look like one control architecture rather than three optional agent features.

Definition 1.1 (externalized adaptation).

A fixed model exhibits externalized adaptation when its parameters remain unchanged while persistent environment, history, or artifact state changes the behavior available to later invocations.

Proposition 1.1. Capability can grow without a parameter update.

Voyager's effective policy changes across iterations because the conditioning state changes, even though the language-model parameters do not.

\pi_t(a\mid g)=\pi_\theta(a\mid g,s_t,H_t,L_t),\qquad \theta_{t+1}=\theta_t

(1.1)

This is a systems proposition derived from the architecture, not a theorem stated by the paper: a larger verified library can enlarge the actions represented in context without changing

\\theta

Section summary

The shortest accurate thesis is: Voyager keeps the model fixed and makes verified programs plus exploration history persistent, so the system can acquire new callable capabilities across episodes.

Source noteIntroduction and Figure 2 · pp. 2–3

1.2 Define the controller

Controller architecture and persistent state

A fixed-goal agent asks how to reach a target. Voyager must first decide which target is both useful and reachable. The paper’s contribution only becomes legible when those two control problems are separated.

Derived control notation · article reconstruction

s_t

typed world state

curriculum CC(s_t, H_t) → g_t

feasible novel task

code generator GG(g_t, s_t, top5(L_t), τ_t) → p_t

task · state · skills · trace

environment TT(s_t, p_t) → (s', τ')

execute program

verifier VV(g_t, s') → verdict

goal postcondition

COMMIT

L' = L ∪ (description, p_t)H.completed += g_t

RETRY ≤ 4

τ_t += env + error + critique

write feedback back into G

ABANDON

H.failed += g_t

persist task label; retry trace expires

State write-back: s' becomes the next observed world state whether or not the program is promoted.

Control write-back: success expands L and completed history; exhausted failure expands failed history and changes the next curriculum call.

Figure 1.2. One Voyager transaction. The transition system executes a synthesized program against state, asks the critic to verify the resulting structured state, then writes either a callable program plus completed history or a failed-task label back into the next iteration's context.Study question. What exactly flows through one Voyager iteration, and what persists afterward?

Let s_t denote the structured Minecraft state at iteration t: inventory, equipment, position, nearby blocks and entities, biome, health, hunger, time, and related observations. Let H_t contain completed and failed tasks, and let L_t be the executable skill library. The curriculum C receives selected parts of s_t and H_t and emits one immediate goal g_t. This is the outer loop: decide what capability should be acquired next.

The inner loop then tries to close that goal. The generator G receives g_t, s_t, retrieved programs from L_t, and the accumulated execution trace τ_t. It emits a JavaScript program p_t. The environment transition T executes p_t against s_t and returns a new state plus chat and interpreter evidence. A verifier V judges whether the resulting state satisfies g_t. That verdict selects one of three transitions: commit, retry, or abandon.

The write-backs create the learning dynamics. Commit adds p_t to L_t under a generated semantic description and appends g_t to completed history. Retry extends τ_t with environment messages, exceptions, prior code, and critique before calling G again. After four unsuccessful synthesis rounds, abandon appends the task—not the full retry trace—to failed history. Both success and exhausted failure therefore alter the next call to C.

Now the design pressure is visible. If C proposes goals without regard to state, G burns its budget on missing prerequisites. If G produces behavior without a persistent artifact, the system repeatedly pays for the same reasoning. If V cannot distinguish execution from task completion, bad programs enter L_t and become dependencies. Voyager works only if scheduling, synthesis, execution, verification, and memory share explicit state transitions.

Derived state-transition model

$g_t=C(s_t,H_t)$
(2.1)
The outer controller selects one immediate, feasible task from observable state and task history.
$R_t=\operatorname{Top5}(g_t,e_t,L_t)$
(2.2)
The retriever exposes five semantically relevant program artifacts from the persistent library.
$p_t=G(g_t,s_t,R_t,\tau_t)$
(2.3)
The code model compiles the task, state, retrieved programs, and current repair context into one program.
$(s_{t+1},e_{t+1})=T(s_t,p_t)$
(2.4)
Execution changes the world and emits environment messages or interpreter errors.
$v_t=V(g_t,s_{t+1})$
(2.5)
The critic judges the scheduled task from the resulting structured state.

Therefore

v_t=1\Rightarrow (p_t,g_t)\text{ persist};\qquad v_t=0\Rightarrow \text{repair or record }g_t\text{ as failed}

Section summary

Treat Voyager as a transactional controller: C proposes a goal, G compiles a program, T produces evidence, V controls promotion, and the resulting writes update H and L for the next transaction.

Source noteMethod overview, Figure 2, and Algorithm 1 · pp. 2–3, 19

1.3 Estimate the frontier

Automatic curriculum as frontier estimation

“Ask GPT-4 what to do next” is too vague. Voyager constructs a constrained scheduling decision from live state, exploration history, generated domain context, and a definition of what can be verified.

Derived mental model · source-grounded

OBSERVABLE s_t

inventory · equipment · nearby blocks/entities · biome · health · position

HISTORY H_t

completed tasks suppress repetition · failed tasks estimate difficulty

CANDIDATE TASK FILTER: feasible now ∧ novel ∧ verifiable ∧ useful progression

LOW INFORMATION

Repeated / redundant

Already completed or produces no capability increment.

BEHIND FRONTIER

Mastered

Prerequisites and verified programs already exist.

SELECT HERE

Feasible + novel frontier

One reachable state transition that creates a new reusable skill.

BEYOND FRONTIER

Impossible / too hard

Missing materials, tools, knowledge, or tractable subgoals.

less learning signalcapability frontier →lower feasibility

Figure 1.3a. The capability frontier. The scheduler uses visible state to reject unreachable goals and completed/failed history to reject repetitions or known dead ends, aiming at the narrow band that is both novel and executable now.Study question. What makes a curriculum task the next task rather than merely a plausible task?

Consider the paper’s concrete state: the agent owns a furnace, four raw iron, one coal, seven oak planks, a crafting table, and a stone pickaxe. “Craft an iron pickaxe” is directionally valuable but skips a state transition because the inventory contains raw iron rather than ingots. “Smelt four raw iron” sits on the frontier: it is novel, immediately feasible, and its success appears directly in inventory. The curriculum’s job is to find that narrow band between tasks already mastered and tasks whose prerequisites are missing.

Its prompt receives four classes of input. First are directives: discover diverse things, avoid repetition, prefer tasks just beyond current capability, and return one concrete task in a constrained verb-object form. Second is current state. Third is completed and failed history, which serves as an external estimate of capability and difficulty. Fourth is additional Minecraft context: GPT-3.5 generates five to ten questions about the current situation, can retrieve relevant wiki material, answers the questions, and gives the answers to GPT-4.

Verifiability is a genuine scheduling constraint, not documentation polish. The task must produce an observable postcondition through the available text state. “Acquire one iron ingot” can be checked in inventory. A spatial arrangement may be impossible to verify without vision. The curriculum is therefore choosing over the intersection of useful tasks and judgeable tasks; anything outside that intersection cannot participate safely in the commit loop.

The state representation is warmed up. At zero completed tasks the prompt includes core inventory, equipment, nearby blocks, and position. Nearby entities appear after five tasks, full inventory after seven, recently seen blocks and biome after ten, and health, hunger, time, and additional context after fifteen. This is not learning inside C. It is an authored observation schedule that withholds dimensions until they are likely to affect decisions, reducing early prompt noise.

All model calls use temperature 0 except the curriculum, which uses 0.1. Code generation and verification are biased toward repeatability; task selection is allowed a small amount of variation so exploration does not collapse onto one locally plausible sequence. Put together, the scheduler is state-conditioned, history-conditioned, constraint-heavy, and slightly stochastic. Remove one of those properties and the frontier estimate changes.

Definition 1.2 (capability frontier).

The set of tasks that are novel relative to

H_t

, feasible in

s_t

, and verifiable from the observations available after execution.

Derived formalization. The paper implements the scheduler through prompting rather than an explicit optimizer. Its selection pressure can nevertheless be written as a constrained objective:

g_t^*\in\underset{g\in\mathcal{G}}{\arg\max}\;\operatorname{Novel}(g\mid H_t)+\lambda\operatorname{Useful}(g)\quad\text{s.t.}\quad \operatorname{Feasible}(g\mid s_t)=1,\;\operatorname{Verifiable}(g)=1

(3.1)

A REAL FRONTIER DECISION

Visible state: furnace ×1 · stone_pickaxe ×1
oak_planks ×7 · crafting_table ×1
raw_iron ×4 · coal ×1
Too early: Craft an iron pickaxe
Frontier task: Smelt 4 raw iron

OBSERVATION WARM-UP

0 tasksinventory summary · equipment · nearby blocks · position
5+ nearby entities
7+ full inventory
10+ seen blocks · biome
15+ health · hunger · time · generated context

Figure 1.3b. A curriculum decision and its observation schedule. This original reconstruction combines the paper's reported task decision with the observation rules documented in Appendix A.3: task-format constraints, verifiability, generated context, and staged state exposure.

Section summary

C does not optimize an explicit learned reward. Its policy is encoded in prompt directives, visible state, task history, a staged observation schedule, and a requirement that success be externally judgeable.

Source noteSection 2.1 and Appendix A.3 · pp. 3–4, 20–24

1.4 Compile behavior

Programs as actions and temporal abstractions

Voyager does not emit keyboard actions or Mineflayer calls one at a time. GPT-4 writes an async function whose execution can span navigation, collection, crafting, combat, and many environment steps.

Schematic comparison · example code is illustrative

PRIMITIVE-ACTION POLICY

looksteplookstepturnstepminewaitmineopenslotcraftclose… hundreds more

A single navigation or inventory mismatch invalidates the remaining suffix. The useful plan survives only as a long token trace.

NAMED PROGRAM

async function craftIronPickaxe(bot) {
  await smeltIron(bot, 3);       // verified dependency
  await ensureSticks(bot, 2);    // verified dependency
  await craftAtTable(bot, "iron_pickaxe", 1);
}

CALLone model-level actionSPANSmany environment steps and feedback eventsPERSISTSnamed source code re-enters later prompts

Action: invoking the function causes a temporally extended state transition.

Memory: storing its implementation preserves a composable procedure, not just a narrative of what happened.

Figure 1.4. Primitive actions and temporal abstractions. A named program compresses a fragile, long-horizon action sequence into one callable abstraction, can reuse verified dependencies, and remains executable memory after the original episode ends.Study question. Why does Voyager generate code instead of emitting Minecraft actions directly?

In hierarchical control terms, a generated function behaves like an option or macro-action: it has an intended outcome, runs for a variable duration, and may invoke lower-level procedures. That compression is essential. A primitive-action sequence becomes invalid after one navigation or inventory mismatch, and preserving the token trace does not create a reliable callable behavior. A named function can check preconditions, branch on state, invoke dependencies, and be executed again.

Mineflayer supplies the abstraction boundary. The prompt exposes helpers such as mineBlock, craftItem, smeltItem, placeItem, and killMob, plus selected lower-level APIs. GPT-4 is instructed to reuse those helpers rather than bypass them, inspect inventory before consuming resources, avoid infinite loops and event listeners, call exploreUntil when targets are missing, report intermediate progress through bot.chat(), and make functions generic enough to become dependencies of later functions.

The code model is therefore not called with a task in isolation. Voyager assembles a build context: generation rules; control primitives and API documentation; five retrieved skill programs; the last generated code; chat emitted during execution; interpreter errors; critic feedback; live agent state; and the task plus generated context. The required response has three parts: explain what failed or is missing, plan the state transitions, then emit one async function taking bot as its only argument.

That resembles an incremental compiler pipeline more than unconstrained tool use. Platform rules and APIs define the target runtime. Retrieved programs act as linked dependencies. Previous code plus runtime feedback form the failed build. GPT-4 performs diagnosis and emits a new artifact. Execution is the integration test. The analogy is imperfect—there is no type checker for semantic preconditions—but it makes clear why prompt composition is part of the architecture.

The benchmark also makes synthesis unusually recoverable. On death the bot respawns near the closest ground and retains inventory. Crafting tables and furnaces are recycled after execution. The world is sandboxed, and the API removes pixel perception and low-level motor control. Code succeeds as a learning substrate partly because failure is cheap and the action interface is broad, inspectable, and forgiving.

Definition 1.3 (temporal abstraction).

A callable action whose execution spans multiple environment transitions while presenting one named operation to the higher-level controller.

Article notation. The paper describes code as temporally extended action; Equation 4.1 makes the hidden execution horizon explicit.

p_t:s_t\longmapsto \bigl[(s_t,e_t),(s_{t+1},e_{t+1}),\ldots,(s_{t+h},e_{t+h})\bigr]

(4.1)

The horizon $h$ is hidden behind the function call. Storing the function preserves the procedure that generated the trajectory, not merely a textual summary of the trajectory.

Mechanism reconstruction · Appendix A.4

Prompt build context

ordered inputs

00 platformrules + Mineflayer control API
10 observationcurrent state
20 targettask + generated context
30 linktop-5 retrieved programs
40 incrementalprior code
50 runtime.stdoutenvironment feedback
51 runtime.stderrinterpreter error
60 testcritic feedback

assembled without weight updates

Compiled response

one deployable artifact

01 explain
Interpret the last outcome and constraints.
02 plan
Name preconditions, dependencies, and intended state transitions.

03 emit

async function task(bot) {
  // composed executable behavior
}

Runtime evidence is carried into the next build, so repair is an incremental recompile rather than a clean prompt.

Figure 1.5. The code-generation build context. Voyager assembles an ordered build context from platform rules, live state, retrieved code, and all prior runtime evidence; GPT-4 must turn that context into an explanation, a plan, and one asynchronous JavaScript function.Study question. What does the code model actually receive, and what contract must it emit?

Section summary

The code action space converts pretrained model knowledge into debuggable, temporally extended procedures—but only because Mineflayer and the experimental safety assumptions provide a tractable runtime.

Source noteSection 2.2, full code prompt, and experimental details · pp. 4–5, 24–33, 38

1.5 Separate address from payload

Executable skill memory

Now follow the two paths separately. The write path determines what becomes memory. The read path determines what the next synthesis call is allowed to reuse.

Mechanism reconstruction · Figure 4 and Appendix B.4.4

Write path · after verification

W1
Verified JavaScript program
async function smeltIron(…)
W2 · GPT-3.5
Generate semantic description
“Smelt raw iron into iron ingots using a furnace and fuel.”
W3 · ada-002
Store embedding as key + source as value
keydescription embeddingvalueJavaScript program

Read path · before generation

R1 · query text
Task + generated general solution + environment feedback
R2 · ada-002
Embed query, rank description keys
80.2% top-196.5% top-5309 samples
R3 · dereference
Return top-5 descriptions and their programs
R4 · GPT-4
Compose useful dependencies into new code

Description is the retrieval interface.

Program is the executable payload.

Figure 1.6. Skill-library write and read paths. Verification admits the source code, GPT-3.5 gives it a semantic address, and ada-002 indexes that address; later, a context-rich query retrieves five description/program pairs for GPT-4 to compose.Study question. How does a successful behavior become retrievable executable memory?

Write path first. Once V declares a task successful, GPT-3.5 generates a short description of the main function. Voyager embeds that description with text-embedding-ada-002. The description embedding is the key; the verified JavaScript program is the value. Natural language supplies a semantic address, while source code remains the executable payload. The system never embeds a trajectory and hopes the model can reconstruct the procedure later.

On the read path, GPT-3.5 proposes a general solution to the new task. Voyager combines that solution with environment feedback to create query context, embeds the context, and ranks stored description embeddings. For “craft an iron pickaxe,” the retrieved set includes Smelt Iron Ingot, Craft Stick, Make Crafting Table, Make Furnace, and Craft Wooden Pickaxe. That mixture contains direct prerequisites, a prerequisite of a prerequisite, and an analogous recipe.

The retrieval result explains an important systems choice. Appendix B evaluates 309 samples: top-1 accuracy is 80.2%, while top-5 reaches 96.5%. Voyager therefore spends prompt budget to increase recall. The retriever does not have to identify the single correct dependency; it has to place useful candidates in context. GPT-4 then performs the final relevance filtering and composition while generating the new program.

This is stronger than a textual episodic memory because reuse is operational: the returned value can execute. It is weaker than a package system. Skills have no typed preconditions or postconditions, dependency versions, provenance policy, compatibility tests, stale-artifact detector, or retirement process. Semantic similarity also says nothing about whether two programs can safely compose against the current state.

The library avoids catastrophic forgetting only in a narrow external-memory sense: stored functions do not vanish because no weights are updated. It can still suffer semantic collision, contaminated writes, brittle dependencies, or prompt-context pressure. The durable pattern is not “add a vector database.” It is “semantic address, executable payload, high-recall retrieval, generator-side composition, verification-gated write.”

Definition 1.4 (executable skill).

A verifier-approved JavaScript program stored as the payload of a semantically addressed library entry.

Mechanism reconstruction. Equations 5.1–5.3 formalize the read and write paths described by the paper; the authors do not present them in this notation.

Semantic address, executable payload

$k_i=E(d_i),\qquad L_t=\{k_i\mapsto p_i\}_{i=1}^{n_t}$
(5.1)
A generated description dᵢ is embedded as the key; the verified source program pᵢ remains the value.
$q_t=E(\operatorname{plan}(g_t)+e_t)$
(5.2)
A generated task plan and environment feedback become the retrieval query.
$R_t=\underset{i\in\{1,\ldots,n_t\}}{\operatorname{arg\,top\text{-}5}}\;\cos(q_t,k_i)$
(5.3)
The system optimizes recall by injecting five candidate descriptions and programs into the generator context.

Section summary

L_t behaves like an untyped package cache indexed by intent: descriptions make capabilities discoverable, programs make them callable, and verification decides which artifacts are eligible to persist.

Source noteFigure 4, Section 2.2, and Appendix B.4.4 · pp. 4–5, 42

1.6 Control promotion

Iterative repair and verification

Calling all of this “reflection” hides the contracts. Voyager exposes three evidence channels, and each blocks a different kind of invalid artifact from being promoted.

Mechanism reconstruction · Figures 5–6 and Appendix A.5

ENVIRONMENT MESSAGES

Intermediate effects

Did inventory, position, or nearby-world state change as expected?

INTERPRETER ERRORS

Program validity

Did JavaScript execute against the available API without syntax/runtime failure?

GPT-4 CRITIC

Goal postcondition

Does the resulting structured state satisfy the natural-language task?

PROMOTION GATE

critic-confirmed success?

Commit

write skill

Retry

max four

Abandon

record failure

Correlated-judge risk: GPT-4 synthesizes behavior and another GPT-4 call judges task success; this is not independent ground truth.

Figure 1.7. Evidence channels and the promotion gate. Environment messages expose intermediate effects, interpreter errors reject invalid executions, and the critic evaluates the goal postcondition; only the final success branch commits, while other evidence drives bounded repair or abandonment.Study question. What evidence can block a program from entering long-term memory?

Environment messages describe intermediate effects inside an otherwise valid program. The supplied helpers and generated code use bot.chat() to report progress or missing prerequisites: “I cannot make stick because I need 2 more planks.” That message means the program reached a meaningful branch but the planned resource state was wrong. The next synthesis round should add a collection or crafting dependency.

Interpreter errors answer a different question: did the program use JavaScript, Mineflayer, and the Minecraft object model correctly? “No item named acacia_axe” is not a missing-resource problem. The requested object does not exist. Figure 5 shows GPT-4 replacing the invented acacia axe with a wooden axe. Runtime validity and task progress must remain separate because they induce different repairs.

The GPT-4 critic evaluates the goal postcondition. It receives the task and resulting structured state, then asks whether the requested outcome is now true. It is not given an independent before/after trajectory. A program may terminate without exceptions and still fail this test. Conversely, the critic can infer success from inventory evidence—for example, coal after a coal-mining task. V is therefore the promotion gate between an executed artifact and persistent memory.

Each new round receives previous code, environment messages, the interpreter trace, and critic critique. After critic-confirmed success, Voyager stores the function and marks the task completed. After at most four failed generation rounds, it abandons the task, records failure, and returns control to the curriculum. The bounded retry policy prevents a single hard objective from consuming the open-ended exploration budget.

The gate is learned and correlated. GPT-4 participates in synthesis and another GPT-4 call judges success from the same textual abstraction. The paper records verifier mistakes, including failure to treat spider string as evidence that a spider was killed. A false negative wastes work; a false positive is worse because it can insert a mislabeled program into L_t and contaminate future compositions. Production versions need deterministic assertions or independent evaluators wherever possible.

Definition 1.5 (promotion gate).

The decision rule separating code that merely executed from code authorized to become persistent reusable memory.

Article-derived update rule. Write task history as $H_t=(H_t^{\mathrm{done}},H_t^{\mathrm{fail}})$ . Equation 6.1 then restates Algorithm 1 as a piecewise commit transition.

(L_{t+1},H_{t+1}^{\mathrm{done}},H_{t+1}^{\mathrm{fail}})=\begin{cases}(L_t\cup\{d_t\mapsto p_t\},\;H_t^{\mathrm{done}}\cup\{g_t\},\;H_t^{\mathrm{fail}})&v_t=1\\(L_t,\;H_t^{\mathrm{done}},\;H_t^{\mathrm{fail}})&v_t=0\text{ and retries remain}\\(L_t,\;H_t^{\mathrm{done}},\;H_t^{\mathrm{fail}}\cup\{g_t\})&v_t=0\text{ after four rounds}\end{cases}

(6.1)

Only the task label survives an exhausted failure. The detailed repair trace $\\tau_t$ exists within the four-round inner loop and then expires.

Section summary

Environment effects, program validity, and goal satisfaction are three separate contracts. Only the last controls persistence, while all three contribute evidence for repair.

Source noteFigures 5–6, Section 2.3, and Appendix A.5 · pp. 5–6, 34–37

1.7 Execute one transaction

Worked controller trace: the iron-pickaxe dependency

Walk the iron-pickaxe example end to end. The point is not that GPT-4 knows the recipe. The point is how the controller turns that knowledge into an ordered, verified capability increment.

Constructed trace · paper-derived state and controller behavior

tcontrol phasestate / Δstatedecision or artifactwhy this transition is admissible

0Aspirational taskfurnace ×1 raw_iron ×4 coal ×1 oak_planks ×7 crafting_table ×1Craft an iron pickaxeThe paper’s state has the table and wood for sticks, but raw iron must first become ingots.
1Curriculum CsameAdmit: Smelt raw ironChooses the nearest verifiable prerequisite, not the final goal.
2Retrieve top-5samesmelting · stick · table · furnace · wooden-pickaxeProvides a material procedure, recipe dependencies, and analogous tool code.
3Synthesize Gsameasync function smeltIron(bot, 4)Calls verified furnace/fuel behavior and emits progress messages.
4Execute Traw_iron −4 coal −1 iron_ingot +4Environment: “Smelted 4 iron ingots”The observed delta establishes the required material transition.
5Verify Viron_ingot ×4critic: success = truePostcondition for the scheduled smelting task is satisfied.
6Commitlibrary + smeltIron history + completedStore description + executable programThe procedure and completion record persist beyond this trace.
7Next frontieriron_ingot ×4 oak_planks ×7 crafting_table ×1Admit: Craft an iron pickaxeThree ingots are present; the existing planks can supply two sticks at the existing table.

ordering invariantsmelt(raw iron) must precede craft(iron pickaxe): the latter's recipe postcondition is unreachable until the former changes inventory from raw material to ingots.

Figure 1.8. Constructed iron-pickaxe trace. The curriculum first schedules smelting because the initial state has raw iron but no ingots; execution creates four ingots, verification commits the smelting program, and the next scheduler call can admit the iron-pickaxe task with its material precondition satisfied.Study question. How does one verified subtask move “Craft an iron pickaxe” onto the feasible frontier?

At t=0 the paper’s state contains a furnace, four raw iron, coal, seven oak planks, a crafting table, and a stone pickaxe. The aspirational objective “craft an iron pickaxe” is not yet executable because the recipe requires ingots. C selects “smelt four raw iron.” This is a better immediate goal because its inputs are present, its output is observable in inventory, and completion moves the dependency graph toward the aspirational objective.

G then receives that task, current state, the prompt rules, and retrieved skills. Useful candidates may include smelting, furnace creation, stick crafting, crafting-table creation, and an analogous pickaxe recipe. GPT-4 emits a function that checks for a furnace and fuel, places or reuses the furnace, calls the smelting helper, and emits progress through chat. T executes the program; inventory changes from raw iron plus coal to iron ingots.

V judges the scheduled task, not the eventual pickaxe goal. If four ingots now exist, the smelting postcondition holds. The system commits the function under a generated description and adds the smelting task to completed history. That single commit creates three durable changes: s_t contains the needed material, H_t records the frontier step as mastered, and L_t contains a reusable smelting procedure.

On the next outer-loop iteration, “craft an iron pickaxe” can move onto the feasible frontier: four ingots are present, seven oak planks can produce the two required sticks, and the crafting table already exists. The generator can retrieve the newly committed smelting function together with stick and pickaxe procedures. This is the compounding mechanism in concrete form: a verified program changes both the resources in the world and the code available to interpret the next task.

Failure also has a role. If smelting exhausts four attempts, the failed-task record tells C not to immediately repeat the same objective under equivalent conditions. Voyager does not learn an explicit value function from that failure, nor does it preserve the full failed execution trace in curriculum history; it supplies the unsuccessful task label to a language-model scheduler. The control benefit is real, but it depends on the model interpreting that coarse history sensibly.

Example 1.1

Move the iron pickaxe onto the feasible frontier

Problem

The paper's state contains a furnace, four raw iron, coal, seven oak planks, a crafting table, and a stone pickaxe. Determine the immediate task whose verified completion makes an iron pickaxe feasible.

Solution

The missing recipe precondition is iron ingots, not wood or a crafting surface. The curriculum therefore selects Smelt 4 raw iron. If execution succeeds, the inventory transition is

\{4\,\mathrm{raw\ iron},1\,\mathrm{coal}\}\xrightarrow{\;p_{\mathrm{smelt}}\;}\{4\,\mathrm{iron\ ingots}\}

(7.1)

Equation 7.1 is an author-constructed trace using the paper's reported inventory and task. The critic verifies the smelting task from the resulting inventory. Commit then changes three distinct stores: world state gains ingots, completed history gains the task, and the library gains a callable smelting program. Because the existing planks can provide two sticks and the crafting table is already present, Craft an iron pickaxe is now admissible.

Section summary

Capability compounds only at the commit boundary: execution changes the world, verification authorizes persistence, and the resulting state plus artifact shifts the next curriculum frontier.

Source noteAlgorithm 1, Figures 2–6, and full prompts · pp. 2–6, 19–37

1.8 Define the measurement

Evaluation protocol and empirical results

Read the evaluation contract before the multipliers. Otherwise item counts, traversal, and tech-tree speed sound broader than the interface justifies.

The main system uses gpt-4-0314 for curriculum, code generation, and self-verification; gpt-3.5-turbo-0301 for context and skill descriptions; and text-embedding-ada-002 for retrieval. The environment is MineDojo controlled through Mineflayer. Each main method receives 160 prompting iterations and is run for three trials. A prompting iteration is a model call in code generation, not a normalized environment step, token count, dollar cost, or wall-clock duration.

ReAct, Reflexion, and AutoGPT did not operate in this setting out of the box, so the authors adapted them to structured state and executable code. ReAct receives environment feedback and agent state. Reflexion additionally receives execution errors and Voyager’s self-verification module. AutoGPT receives state, feedback, and errors while decomposing the top-level exploration goal, but lacks Voyager’s automatic curriculum, persistent skill library, and explicit task-success gate. These are system-level comparisons, not untouched reference implementations.

Within that contract, Voyager discovers 63 unique items in 160 prompts—3.3 times the evaluated baselines—and travels 2.3 times farther than the baselines. It reaches wooden, stone, and iron tools in 6 ± 2, 11 ± 2, and 21 ± 7 prompting iterations across successful trials. AutoGPT requires 92 ± 72, 94 ± 72, and 135 ± 103. Diamond is not a reliable success: Voyager reaches it once in three trials at prompt 102; every other method is 0/3.

Map coverage adds a separate behavioral signal. An item counter alone could reward local crafting chains. The larger traversal envelope shows that Voyager leaves local regions and encounters a wider distribution of biomes and resources. It does not prove better navigation in a low-level sense because Mineflayer supplies navigation primitives; it shows that the high-level controller schedules behavior that results in broader movement.

The transfer experiment is the cleanest evidence for executable memory. The authors clear inventory, create a new world, provide four unseen tasks, and allow at most 50 prompting iterations. Full Voyager solves diamond pickaxe, golden sword, lava bucket, and compass in all three trials, at 19 ± 3, 18 ± 7, 21 ± 5, and 18 ± 2 iterations. Voyager without the library is generally slower and misses one diamond-pickaxe trial. Giving Voyager’s library to AutoGPT improves AutoGPT, although it still solves tasks less reliably. Because the world and inventory reset while the programs persist, the intervention isolates transferable value in the artifacts better than the main exploration curve does.

Definition 1.6 (prompting iteration).

One code-generation prompting round in the reported evaluation. It is not an environment step, token-normalized compute measure, wall-clock duration, or total count of every model invocation in the system.

Metric notation introduced here. The paper reports milestone iteration counts and success fractions; Equation 8.1 names the stopping-time quantity those tables summarize.

I_m^{(r)}=\min\{j\leq B:\text{milestone }m\text{ is achieved in trial }r\}

(8.1)

Tables must be read with both $I_m$ and the success fraction. A low average over one successful trial is not comparable to three reliable successes without the denominator.

Table 1.1. Tech-tree mastery: prompting iterations, lower is better
Method	Wood	Stone	Iron	Diamond
ReAct	N/A (0/3)	N/A (0/3)	N/A (0/3)	N/A (0/3)
Reflexion	N/A (0/3)	N/A (0/3)	N/A (0/3)	N/A (0/3)
AutoGPT	92 ± 72 (3/3)	94 ± 72 (3/3)	135 ± 103 (3/3)	N/A (0/3)
Voyager, no library	7 ± 2 (3/3)	9 ± 4 (3/3)	29 ± 11 (3/3)	N/A (0/3)
Voyager	6 ± 2 (3/3)	11 ± 2 (3/3)	21 ± 7 (3/3)	102 (1/3)

Table 1.2. New-world transfer: prompting iterations and successful trials
Method	Diamond pickaxe	Golden sword	Lava bucket	Compass
ReAct	N/A (0/3)	N/A (0/3)	N/A (0/3)	N/A (0/3)
Reflexion	N/A (0/3)	N/A (0/3)	N/A (0/3)	N/A (0/3)
AutoGPT	N/A (0/3)	N/A (0/3)	N/A (0/3)	N/A (0/3)
AutoGPT + Voyager library	39 (1/3)	30 (1/3)	N/A (0/3)	30 (2/3)
Voyager, no library	36 (2/3)	30 ± 9 (3/3)	27 ± 9 (3/3)	26 ± 3 (3/3)
Voyager	19 ± 3 (3/3)	18 ± 7 (3/3)	21 ± 5 (3/3)	18 ± 2 (3/3)

Section summary

The evidence supports efficient accumulation and transfer of high-level Minecraft procedures per LLM prompting iteration under a structured software interface; it does not measure perception-to-motor embodied intelligence.

Source noteSections 3.1–3.3, Figures 7–8, and Tables 1–2 · pp. 6–8

1.9 Read interventions, not slogans

Ablations and causal attribution

An endpoint multiplier tells you who won. The curve shape tells you which mechanism stopped producing new capabilities.

Replacing the automatic curriculum with random item goals reduces discovered items by 93%. The random sampler draws from 101 items that Voyager discovered, so it possesses a plausible goal vocabulary. What it lacks is prerequisite ordering and state fit. A manual diamond-oriented curriculum does better than random but plateaus early because it is narrow and world-agnostic. The contrast supports C as a frontier estimator rather than merely a source of varied task text.

Removing L_t produces the most diagnostic shape: early acquisition remains competitive, then the curve flattens. Early Minecraft tasks are simple enough for GPT-4 to synthesize from prior knowledge and current prompt context. Later tasks require longer compositions and repeated prerequisites. Without persistent verified code, each call rebuilds those procedures. The plateau is the signature of missing accumulation, not simply worse first-attempt intelligence.

The feedback ablations distinguish repair from promotion. Removing environment feedback or interpreter errors harms discovery because G loses localized evidence about intermediate effects and program validity. Removing self-verification is most damaging among the feedback channels, reducing discovery by 73%, because the controller loses the goal-level decision that determines retry versus advance and protects the write path.

Replacing GPT-4 with GPT-3.5 for code generation yields 5.7 times fewer unique items, while GPT-4 remains in curriculum and verification. That localizes the gap to executable synthesis within this architecture. The result is a ceiling condition: retrieval, feedback, and scheduling amplify a strong generator, but they do not make generator quality interchangeable. A repair loop cannot compound if it rarely produces valid artifacts worth committing.

These are component ablations, not a complete causal decomposition. Prompt wording, state representation, retrieval count, model prior, verifier correlation, and the forgiving runtime are not independently crossed in a factorial design. The safe conclusion is that the shipped bundle’s curriculum, program memory, feedback, verifier, and strong code model are jointly load-bearing in this benchmark—not that each quoted percentage transfers to another domain.

Random curriculum−93%Task vocabulary without ordering is not a curriculum.

No self-verification−73%The controller loses its task-level commit decision.

GPT-3.5 code model5.7× fewerScaffolding cannot compensate for weak synthesis.

Table 1.3. Selected ablation effects. All three values are reported interventions, not a combined or additive model. The skill-library ablation is best read from the curve shape: similar early progress, then a later plateau.

Section summary

Read the ablations as failure locations in a compounding pipeline: bad ordering prevents starts, missing memory causes later plateaus, weak evidence corrupts repair and promotion, and a weak generator starves the library.

Source noteSection 3.4, Figure 9, and Appendix B.3 · pp. 8–9, 39

1.10 Translate the architecture

Scope, limitations, and architectural transfer

Voyager becomes most useful when its abstractions are separated from the conditions that made them experimentally cheap.

Scope synthesis · paper methods and limitations

Inside measured system

What the benchmark exercises

Observation: structured text state
Embodiment: high-level Mineflayer API
Control: LLM task scheduling
Action: JavaScript program synthesis
Admission: execution + LLM verification
Learning substrate: external program memory

Outside measured system

What the interface abstracts away

Perception: pixels and raw sensory inference
Actuation: low-level motor control
Adaptation: model-weight updates
Stakes: costly irreversible failure
Judging: independent ground-truth verification

The comparison is about open-ended capability acquisition inside this software-mediated interface, not general embodied intelligence.

Figure 1.9. Boundary of the measured system. Under structured text observations and a high-level Minecraft API, a fixed LLM can acquire more diverse and transferable capabilities over time by combining automated curriculum, program synthesis, execution feedback, verification, and external executable memory.Study question. What is the strongest claim the benchmark actually supports?

Inside the measured system are structured text observations, high-level Mineflayer actions, LLM task scheduling, JavaScript synthesis, execution feedback, GPT-4 verification, and external program memory. Outside it are pixels, raw sensory inference, low-level motor control, weight updates, independent ground-truth judging, and expensive irreversible consequences. The paper advances the first stack. It deliberately does not solve the second.

Grounding failures remain visible. The curriculum can propose nonexistent copper swords; the code generator invents APIs or treats cobblestone as fuel. The same pretrained domain prior that makes GPT-4 productive can produce confident false assumptions. A proprietary scientific or enterprise domain would need authoritative retrieval, typed resource models, and stronger feasibility checks before C could distinguish a novel task from a fictional one.

The library needs package semantics. A production artifact should carry typed preconditions and postconditions, dependencies, tests, provenance, authority requirements, version compatibility, observed success rates, and retirement rules. Retrieval should filter on those contracts before semantic similarity. A verified function should run inside a sandbox with resource limits, scoped credentials, auditable effects, idempotency or rollback, and an explicit blast radius.

Verification should be decomposed further. Deterministic invariants should judge what code and state can express exactly. Independent models or humans should handle ambiguous semantic outcomes. A learned judge should not be the sole authority for promoting an artifact that later receives broader permissions. Four retries limit cost; they do not establish safety.

With those qualifications, the architecture remains durable: maintain an adaptive backlog from live state; compile work into executable artifacts; surface real execution evidence; distinguish runtime completion from goal satisfaction; and admit artifacts to persistent memory only through a promotion gate. Voyager’s lasting contribution is showing that these pieces can create system-level capability accumulation even while the foundation model remains unchanged.

Definition 1.7 (claim boundary).

The strongest statement supported after conditioning on observation interface, action abstraction, evaluator, runtime assumptions, trial count, and measured outcome. Voyager supports accumulation of high-level Minecraft procedures under a structured software interface; it does not establish general perception-to-motor embodied intelligence.

Section summary

The defensible claim is narrow and consequential: under a structured, forgiving software interface, a fixed LLM can accumulate transferable executable capability when scheduling, synthesis, evidence, verification, and memory are coupled by explicit commit rules.

Source noteLimitations, broader impacts, and experimental appendix · pp. 9–11, 38

Chapter summary

Summary point 1:
Voyager keeps $\\theta$ fixed while changing $(s_t,H_t,L_t)$ , so adaptation lives in the controller's external state.
Summary point 2:
The automatic curriculum chooses the next capability to acquire; the code generator chooses how to acquire it.
Summary point 3:
Programs compress long action sequences into reusable temporal abstractions and remain executable after the episode ends.
Summary point 4:
The skill library separates semantic addressing from executable payloads and uses top-5 retrieval to trade prompt budget for recall.
Summary point 5:
Environment messages, interpreter errors, and critic judgments test different contracts; only the critic controls promotion.
Summary point 6:
Transfer and ablations support artifact accumulation under the Mineflayer interface, not general low-level embodied intelligence.

Review questions

Mechanism reconstruction

Answer these without looking back. Each question tests whether you can regenerate the architecture rather than recognize its vocabulary.

R1.1Draw the boundary between fixed model parameters and the external state that changes after a Voyager episode. Which writes create the apparent lifelong learning?
R1.2Reconstruct C, G, T, V, H, and L. What does each consume, emit, and write back?
R1.3Given a furnace, raw iron, and coal but no ingots, why should the curriculum schedule smelting before an iron pickaxe? What information makes that decision possible?
R1.4If the automatic curriculum lost failed-task history, what repeated behavior would you expect and why?
R1.5Why does a JavaScript function act as both a temporal abstraction and a memory artifact in the Mineflayer interface?
R1.6Reconstruct the code-generation build context. Which inputs support first-pass synthesis, and which exist only to repair a failed build?
R1.7On the library write path, what are the key and value? On the read path, how is the query produced and why are five results injected?
R1.8For each evidence channel—environment message, interpreter error, and critic verdict—name a failure the other two cannot reliably detect.
R1.9Walk the iron-pickaxe trace from state to scheduled subtask to committed artifact. Which three persistent states change at commit?
R1.10Why is the reset-world transfer study stronger evidence for executable memory than the main unique-item curve?
R1.11Read the ablation curves counterfactually: what behavior should appear without curriculum, without memory, without verification, and with a weaker code generator?
R1.12Which parts of the result would fail to transfer first to a browser agent, laboratory system, or robot with costly irreversible actions?

End-of-chapter problems

Architectural transfer beyond Minecraft

P1.1Core

Specify the transaction boundary

Choose a coding or research agent. Define its task state, executable artifact, three feedback channels, commit predicate, retry budget, and rollback behavior. Then identify which predicate is still delegated to a correlated LLM judge.

Deliverable: a state machine plus the invariants enforced at every transition.

P1.2Design

Design the skill package format

Replace Voyager’s untyped vector-store values with a package contract: semantic description, code, preconditions, postconditions, dependencies, provenance, tests, version, and retirement policy. Show how retrieval and composition change.

Deliverable: a typed schema and one worked example with a dependency failure.

P1.3Research

Rebuild the ablation

Design a factorial evaluation that separates curriculum instructions, state conditioning, task history, model prior, retrieval count, and verifier independence while keeping model-call budget constant.

Deliverable: an experiment matrix, primary metrics, and the causal claim each comparison would support.

Notes and primary source

Each section supplies a page-level source note. The complete paper remains the authority for exact prompts, item lists, related work, and supplementary figures. Custom equations in this chapter are labeled as derived formalizations when they are not notation used by the authors.

Open VOYAGER: An Open-Ended Embodied Agent with Large Language Models

Nick Bryant

The Lower Worlds · architecture study

Give Agents Lives, Not Just Jobs

A Small Model With a World Around It

Human–Agent Interaction Is More Than Chat

Digital Twins Need More Than Your Chat History

Marketing Is a World Agents Can Work In

Treasure Hunter

AI Robot Workshop

Fractal Authoring

Building procedural environments in Valenfeld

The Transformer Is the Transistor

The Lower Worlds: Awakening

ARCOS / VN.AI

Procurement Sciences

Investing systems

Interesting Inventions

Prompt build context

Compiled response