Robotics 2030 — A Guide for Hackers
Build the complete path from an instruction to a bounded physical effect, then make its failures and updates inspectable.
Evidence cutoff: 2026-08-31
Exploratory structural draft — not an Early Access release
Preface
This book is for the experienced software engineer, hardware hacker, roboticist from one specialty, technical founder, or advanced student who wants one coherent model of a modern robot. The reader should be able to program in Python, read C or C++, use Linux and Git, and work through algebra, basic calculus, probability, and introductory mechanics. The book supplies the robotics-specific connections: how geometry, current, timestamps, controllers, learned policies, networks, and release evidence constrain one another.
Hacker describes a method here, not recklessness. Inspect the interface. Build the smallest mechanism that preserves the real problem. Instrument it. Break one assumption deliberately. Keep only the result that survives measurement. Simulation and guarded bench work are legitimate implementation tiers. Nothing in this book grants permission to bypass a protective device, improvise around moving machinery, or treat a software demonstration as a safety case.
The evidence language matters because different sources answer different questions. An empirical study reports observations under a stated apparatus, task set, model, and denominator; it does not automatically transfer to another robot. An official specification establishes what a standard or maintained interface says it does, not that a particular integration meets its deadlines. A production report is first-party evidence about a named private or deployed system and retains that source's incentives and disclosure limits. A frontier preprint can establish a new mechanism or result before peer review, but carries exceptional replication and freshness risk. Book synthesis is this book's explicit organization or forecast built from such premises; it is not an observed fact. Citations sit beside the factual sentence they support, and setup, version, denominator, first-party status, and limitations stay attached to the claim.
Two recurring model acronyms appear in the admitted contents: a large language model (LLM) works over language and related tokenized context, while a vision-language model (VLM) connects visual input with language. A vision-language-action model (VLA) additionally produces a robot action representation. Later chapters will examine the architectures and evidence behind those compact labels.
One fictional system carries the engineering state through all seventeen chapters. Relay is a mobile manipulator in a small workshop tool crib. It is not a product, company, benchmark, accident report, or deployment. Relay begins as an inspectable message path and accumulates a body, actuators, power, sensors, compute, control, learned policies, data, cloud services, and release evidence. Its capstone mission remains fixed: retrieve the blue torque wrench from shelf B, place it in drawer 3, close the drawer, and report evidence of completion. Any later change to Relay's hardware, coordinate conventions, action interface, timing, or success predicate must be explicit and versioned.
Contents
Part I — Start with the physical consequence
- One Robot, Six Clocks
- Choose the Work Before the Body
Part II — Build the machine that learning must inhabit
- Structure, Joints, and Transmission
- Actuation, Contact, and Thermal Reality
- Power, Wiring, Sensors, and Embedded Safety
- Compute, Buses, and Time
Part III — Turn measurements into controlled motion
- Frames, State, and Perception
- Feedback, Planning, and Manipulation
- Robot Operating Systems and Middleware
Part IV — Add models that learn and transfer
- Build the Data Engine
- From Demonstrations to Action Distributions
- LLMs, VLMs, and Vision-Language-Action Policies
- World Models, Simulation, and Transfer
Part V — Operate, update, and judge the whole robot
- Edge, Cloud, and Fleet Learning
- Safety, Security, Evaluation, and Release
- Who Is Building What
- Capstone: Release Relay
Living reference appendices
- Appendix A. Mathematics and notation — Units, vectors, frames, rotations, transforms, twists and wrenches, Jacobians, probability, optimization, and control notation.
- Appendix B. Mechanical field reference — Load cases, joints, bearings, fasteners, fits, tolerances, materials, transmissions, backlash and compliance, lubrication, cable routing, manufacturability, and inspection.
- Appendix C. Electrical field reference — Power tree, battery and battery-management system, converters, fusing, motor drives, grounding and shielding, connectors, wire sizing, encoders, inertial measurement units, force and tactile sensing, cameras and lidar, emergency-stop and safe-torque-off boundaries, and instrumentation.
- Appendix D. Computer and network field reference — Microcontrollers, processors, central processing units (CPUs), graphics processing units (GPUs), neural processing units (NPUs), memory movement, Linux, real-time Linux, real-time operating systems, containers, Controller Area Network (CAN) and CAN Flexible Data Rate (CAN FD), EtherCAT, serial, Ethernet time-sensitive networking, Data Distribution Service (DDS), Zenoh, time synchronization, storage, logging, and over-the-air updates.
- Appendix E. Model and data field reference — Observation and action schemas, dataset manifests, vision-language-action model and action-head taxonomy, fine-tuning, quantization, serving, evaluation trial accounting, model cards, and data lineage.
- Appendix F. Builder atlas and source guide — A dated company and project map, deployment evidence level, official source, stack layer, status, acquisition or pivot history, and freshness trigger.
- Appendix G. Build and review checklists — Requirements, design review, bring-up, calibration, fault injection, data collection, model promotion, site acceptance, maintenance, incident, and rollback templates.
A correct answer at the wrong time
The following failure is illustrative, not a report of a real accident. Relay is moving its arm toward the blue torque wrench. A semantic planner has already interpreted the operator's instruction, and a local action policy is executing chunk C42. That chunk is allowed to influence motion only until t = 0.30 s. The planner requests a successor under goal G17, but the response stalls. At t = 1.20 s, a semantically plausible result finally returns: continue toward the wrench and prepare the grasp.
The answer fits the words. It no longer fits the robot. During the missing 900 milliseconds, Relay's state publisher kept sampling, its servo kept running, and C42 expired. The wrist may now be in a different pose, the object may have moved, or a protective transition may have withdrawn motion authority. If the late result enters an ordinary first-in, first-out (FIFO) queue and is executed because it eventually arrived, a correct interpretation becomes a wrong physical command. Reliable delivery has preserved the bytes while destroying their relevance.
Nothing supernatural happened. One producer finished after the state and authority that justified its work had ended. The repair is therefore not “use a smarter model” or “never use a network.” The repair is to make time, source state, authority, expiry, cancellation, local limits, and failure behavior part of the interface. This chapter builds that interface from the physical task upward.
Meet Relay before the architecture
Relay works in a small workshop tool crib. Its fixed mission is: retrieve the blue torque wrench from shelf B, place it in drawer 3, close the drawer, and report evidence of completion. This is a compact task with enough structure to expose the full stack. Relay must connect language to a particular object, navigate or reach to a shelf, grasp, move, open or access a drawer, place, close, and verify the final state. A natural-language sentence names the work; it is not a motor command.
Chapter 1 does not yet choose Relay's final morphology, motor, bus, operating system, or model. It creates Relay v0 as a chain of actors and records. A state sample is an immutable observation of the robot at a production time, such as joint position and velocity under system epoch 4. A subgoal converts semantic intent into a bounded physical objective, such as “move the wrist into the admitted pre-grasp region for the blue wrench,” with limits and an expiry. An action chunk is a finite sequence of future targets derived from a named state sample. A trajectory point is one scheduled target inside that chunk. A servo command is the fast local command, such as bounded desired joint velocity, that can reach the actuator interface. A protective state says whether the local executor may admit motion at all: ACTIVE, DEGRADED_HOLD, PROTECTIVE_STOP, or another machine-specific state.
Now trace the complete path once, without compressing it into “the robot decided.”
- The operator supplies the instruction: “Retrieve the blue torque wrench from shelf B, place it in drawer 3, close the drawer, and report evidence.”
- A semantic component interprets the referents and sequence. It writes a result for goal
G17, not joint motion. - A task layer turns that interpretation into a bounded subgoal with a target region, preconditions, limits, identifier, production time, and validity horizon.
- A local skill or action policy reads the subgoal plus fresh state sample
S880and produces action chunkC42. The chunk carries its source state, epoch, production time, execution times, and expiry. - A local executor admits or rejects each point. It checks epoch, source-state age, current time, cancellation, protective state, and Relay's local position and velocity envelope.
- A servo compares the admitted target with measured state and emits a bounded velocity, torque, or position command at its own cadence.
- A drive turns that command into phase switching and current regulation. Motor torque acts through the transmission and joint.
- The physical arm changes position, velocity, contact force, temperature, and energy state.
- Sensors observe the effect. The state estimator produces later samples, and task logic evaluates outcome evidence: wrench identity, grasp state, drawer state, object placement, intervention, and unknowns.
- Selected evidence may later enter evaluation, data selection, training, and a proposed software or model change. That change receives a version, bounded rollout, and rollback path before it gains wider authority.
The chain is deliberately longer than a model application programming interface (API). Each crossing changes representation and authority. A phrase becomes a subgoal; a subgoal becomes a finite proposal; a proposal becomes a locally admitted target; a target becomes current; current becomes force; force changes the world; observations become evidence; evidence may justify a later release. Bugs hide when adjacent links are collapsed.
Three time words will recur. Production time is when a record was created at its source. Data age is current monotonic time minus that production time, after any clock-domain conversion has been made explicit. A deadline is the latest time by which a computation or effect must occur to satisfy its consumer. A validity horizon is the interval during which a particular record may still exercise its declared authority. A message can arrive before a network timeout yet be too old for its physical use. Conversely, an old observation may remain acceptable for a slow inventory report while being unacceptable for collision response.
Two identity words finish the minimal vocabulary. An epoch changes when a restart, reconfiguration, or authority reset invalidates records from the previous system instance. A wall-clock timestamp alone cannot reliably distinguish an old process that reconnects after a restart. Cancellation is the explicit withdrawal of future authority from a goal, chunk, or point. It is not the same as dropping one packet; consumers must know which future work has become impermissible.
Six clocks, not six computers
Relay's actors do not all owe the same response time. The six-clock model below is book synthesis: an original placement and failure-analysis tool, not an external standard or a measurement of every robot. The illustrative bands describe orders of magnitude that make the boundaries visible. They are not universal recommendations, and a small system may run several clocks inside one process or processor.
| Clock | Illustrative design band | What it owns | What happens when a slower layer stalls |
|---|---|---|---|
| Electrical | tens of microseconds to sub-millisecond | commutation, current limiting, inverter protection | Continues locally; electrical protection cannot wait for a model. |
| Servo | sub-millisecond to several milliseconds | torque, velocity, or position feedback; joint and contact response | Continues or enters locally defined degraded behavior. |
| State | milliseconds to tens of milliseconds | timestamping, sensor fusion, local geometry, collision state | Continues with freshness and uncertainty rules. |
| Skill | tens to hundreds of milliseconds | action chunks, local manipulation or navigation policy, replanning triggers | Usually continues within bounded authority and remains cancellable. |
| Semantic | hundreds of milliseconds to seconds | language grounding, task decomposition, tool use, operator dialogue | May wait or fail; has no direct actuator authority. |
| Fleet | minutes to release cycles | data selection, evaluation, rollout, rollback, maintenance, model refresh | Never joins the immediate control path. |
The electrical clock is closest to energy conversion. A motor drive may regulate current and protect its inverter much faster than an application computer can schedule a Python callback. Later chapters will derive those mechanisms; for now the placement rule is enough. If violating a constraint can damage the drive within its switching or current-control horizon, a remote semantic component cannot own that protection.
The servo clock repeatedly turns measured error into an actuator command. Its concern is not whether “blue torque wrench” was the right phrase. It must track or reject a target while respecting local bounds. A servo may run on a drive, microcontroller, real-time operating system, or carefully engineered host process. Location alone does not make it deterministic or safe. The responsible question is whether its complete sensor-to-command path meets the named deadline under the named load and faults.
The state clock creates the robot's usable “now.” Camera frames, encoders, inertial measurements, force readings, odometry, calibration, transforms, and local collision observations do not become one coherent fact merely because they share a message bus. Their production times, coordinate frames, uncertainty, and freshness rules determine which state a consumer actually read. When Relay's semantic response arrives at t = 1.20 s, the state clock is why the system can prove that source sample S880 is obsolete rather than merely feeling suspicious about it.
The skill clock produces bounded behavior close enough to the body to react to changing task state. It might run a fixed grasp routine, a motion planner, a receding-horizon controller, or a learned action policy. “Skill” names the cadence and responsibility, not the implementation fashion. Its output is more specific than a semantic intention and less authoritative than current or torque. It must be cancellable because a higher layer can change the goal while a lower layer discovers an obstacle or loss of contact.
The semantic clock interprets language and long-horizon context. It can spend hundreds of milliseconds or seconds choosing a tool, decomposing a mission, asking an operator a question, or recovering from an ambiguous instruction. Those delays may affect throughput. They need not corrupt physical execution if semantic output remains advisory until a faster local boundary binds it to current state, a validity horizon, and available skills.
The fleet clock operates after immediate action: select logs, label outcomes, replay failures, evaluate candidates, sign artifacts, deploy a candidate to a small initial cohort in a canary release, monitor regressions, and roll back. A fleet service may coordinate use of a shared passage or issue a lease. That permission does not replace local collision protection. The fleet clock's power is broad and delayed: it can change what many robots do tomorrow, which is why it needs a stronger change and evidence trail even though it never commands this millisecond's motor current.
These clocks answer different questions. A faster semantic model can reduce Relay's idle time without improving current regulation. A 500 hertz servo can track the wrong target perfectly. A highly accurate tool classifier can return too late for the source scene. A fleet model promotion can improve average task success while reintroducing a rare stale-command fault. Rate is one property; authority and evidence determine the consequence.
One maintained Robot Operating System 2 (ROS 2) implementation illustrates the distinction without proving a deadline. The August 2026 Jazzy documentation for ros2_control describes a Controller Manager read-update-write loop and configuration for update rates, fixed-priority first-in-first-out scheduling (SCHED_FIFO), memory locking, CPU affinity, overrun handling, asynchronous components, and fallbacks. Those are documented mechanisms on a named software line, not measurements of Relay and not proof that arbitrary kernels, drivers, controllers, or hardware meet hard real-time or safety requirements. ros2_control Controller Manager documentation
Five contracts across the clocks
A cadence table places work, but placement alone does not make a command reviewable. Relay needs five contracts at every consequential crossing. This exact five-contract grouping is book synthesis, not a certification scheme or a requirement to deploy five services. A tiny system may implement all five in one component. The questions remain distinct because different faults answer them.
The physical contract
The physical contract states what the mechanism can do and survive: geometry, mass, inertia, force, torque, stiffness, friction, backlash, compliance, power, energy, temperature, payload, and duty cycle. For Relay v0, the lab uses only a joint workspace of [-1.0, 1.0] radians and a velocity bound. That simplicity is explicit. It does not claim that position and velocity capture collision, gravity, contact, braking distance, temperature, or stored energy in a physical arm.
The local executor consumes the physical contract before admitting an action point. If a candidate target lies outside the workspace, it is rejected even when its goal, timestamp, and model are otherwise valid. The physical contract is final in a blunt sense: software can change a representation but cannot bargain with an empty battery, a saturated drive, a stripped gear, or an unstable contact.
The state contract
The state contract says what each observation means: producer, sequence, epoch, production time, coordinate frame, units, calibration version, uncertainty or confidence, and what remains unobserved. The lab's StateSample is deliberately narrow: epoch, sequence, monotonic production time, position, and velocity. Relay's later perception state will add camera and object frames, calibration, wrench identity, drawer pose, collision margin, and missing-data semantics.
A freshness check belongs to the consumer because acceptable age depends on use. The action policy may accept a 20 millisecond joint sample while rejecting one that is 150 milliseconds old. An inventory service may accept the same record for a historical report. Transport order cannot answer that question. Production time and the consumer's declared maximum age can.
The action and authority contract
The action and authority contract states who may influence which interface, using which units and frame, from which source state, during which epoch and horizon, under which preconditions and limits, with what cancellation and degraded behavior. Authority is the bounded right of a named producer, over a named interface and epoch, to influence physical execution until a locally checked validity condition ends.
That definition explains the opening failure. C42 had authority until t = 0.30 s; the planner's late result did not inherit it. Candidate C43 must cite a fresh source sample, match the current epoch, remain unexpired, satisfy the physical envelope, and arrive while the protective state permits admission. Authentication can show who sent a message. It does not show that the message is fresh or physically admissible.
The evidence contract
The evidence contract defines what observation can establish the intended effect. Encoder tracking might show that a joint followed a path. It cannot show that Relay grasped the correct wrench. A wrist image may support object identity while failing to prove stable contact. Drawer closure, object placement, intervention count, recovery, cycle time, and unknown state each need explicit predicates. A successful model call is evidence that computation returned, not that the task succeeded.
Evidence flows upward in the same specificity order that commands flowed downward. Current and encoder data support actuator behavior. Fused state supports motion outcome. Object and drawer observations support the task predicate. Joined identifiers support causal reconstruction. Repeated trials, failure accounting, and intervention logs support evaluation. No single witness inherits the scope of all the others.
The change contract
The change contract binds behavior to hardware, calibration, firmware, operating system, package, container, model, prompt, dataset, policy, and configuration versions. It defines how a candidate receives limited authority, how evidence is compared, and how rollback works. A model trained on last month's logs is not “the same robot” when camera calibration, action normalization, firmware, or tool geometry changed underneath it.
The fleet clock owns much of this work, but local components enforce compatibility and activation. A cryptographically authorized artifact can still be behaviorally wrong. Conversely, a high-performing checkpoint with unknown lineage cannot support a controlled rollout. Change authority follows evidence: one simulator, then a guarded bench, then perhaps one robot or one site, before a fleet-wide release.
Together, the contracts form a causal audit. The physical contract bounds possibility. The state contract identifies the world the producer saw. The action contract bounds permissible influence. The evidence contract tests the effect. The change contract records which version may act next. Omit one and a fault becomes ambiguous: Was the target impossible, stale, unauthorized, unverified, or introduced by a changed artifact?
An action chunk is not a servo loop
In an illustrative example, suppose Relay's action policy emits five target positions spaced 20 milliseconds apart. The generated target cadence is 50 hertz because the chunk contains one target for each 20 millisecond interval. That statement does not reveal how often the model observes the robot, how often it generates a replacement chunk, or how often the servo closes feedback around the mechanism.
Those are three separate cadences:
- Model observation and replan cadence: how often the action policy receives a new observation and computes a new proposal. In the illustrative lab, the nominal request cadence is 10 hertz, but a request can take longer than 100 milliseconds and overlap, stall, or be cancelled.
- Generated target cadence: the temporal spacing of targets inside the returned action chunk. The illustrative five-point chunk uses 20 millisecond spacing, or 50 hertz.
- Servo cadence: how often a local controller reads measured state, checks the currently admitted target, and writes a command. The illustrative lab uses 500 hertz.
A policy could therefore observe at 10 hertz, generate one second of 50 hertz targets, and feed a 500 hertz servo. Calling the result a “50 hertz model” loses the distinction that matters. If inference stalls for 300 milliseconds, the servo still runs 150 illustrative periods. It may interpolate and track the admitted chunk, stop at the chunk's validity boundary, or follow a local fallback. It does not acquire 50 hertz closed-loop model feedback merely because future targets were sampled at that spacing.
Action Chunking with Transformers (ACT) provides a bounded empirical mechanism, not a production claim. In six low-cost ALOHA bimanual tasks, ACT predicted multi-step joint-action chunks and temporally combined overlapping predictions; the paper reported 80–90 percent success from about 50 demonstrations, roughly ten minutes of demonstrations, per task. The setup used one hardware family, fixed workcells, operator demonstrations, small trial counts, and task-specific training, so the result does not establish continuous operation, arbitrary embodiment, hard deadlines, or safety. ACT paper, Robotics: Science and Systems 2023
The useful idea for Relay is narrower than the reported score. Chunking lets a slower producer describe a short future trajectory and can smooth or temporally combine overlapping proposals. It also creates a commitment question. A long chunk hides inference latency because execution can continue while the next result is computed. The same long chunk increases open-loop exposure because more future motion was chosen from an older observation. Relay's executor must decide how much of an old chunk remains admissible, and that decision belongs in the action contract rather than in a model marketing rate.
SmolVLA offers a newer mechanism example with a different evidence role. The 2025 frontier preprint describes an approximately 450-million-parameter flow-matching vision-language-action (VLA) model whose LeRobot inference path decouples chunk generation from execution, and reports training on one GPU plus deployment on consumer hardware. That supports the feasibility of a compact asynchronous model/executor split on the reported surfaces. It does not establish high-rate dexterity, a hard deadline, production reliability, semantic parity with larger private systems, or a universal consumer-hardware latency; the comparisons aggregate different protocols and the implementation is fast-moving. SmolVLA preprint
Asynchrony prevents the servo from blocking on the model, but it cannot repair stale information. Consider two chunks generated concurrently. C42 is executing from source state S880. The policy begins C43 from the same state, then returns late. During inference, later samples S881 through S910 record motion and perhaps a moved wrench. Even if every point in C43 has an execution time in the future, its source state may violate the maximum observation age. The executor rejects it. A future timestamp is not a freshness certificate.
Epochs close a second hole. Relay's application computer restarts and increments its system epoch from 4 to 5. An old policy process reconnects and delivers a chunk whose monotonic validity interval appears current. Its epoch=4 makes the result invalid before any geometric check. This prevents a previous owner of an interface from regaining authority merely because its packets survived longer than its process instance.
Cancellation closes a third. The operator redirects Relay from “grasp the wrench” to “hold position,” or the state estimator reports that the object identity is uncertain. Cancelling goal G17 invalidates future points derived only from that goal. A bounded queue should remove or skip them. Publishing a new goal without cancelling the old one leaves two plausible intents racing toward one actuator boundary.
The stalled-planner trace
The table below is an original illustrative text artifact. It is not copied from a source figure, it records no real incident, and every event time is chosen only to expose the authority gap. There is no designed visual render of this trace in this draft.
| Illustrative time | Semantic clock | Skill/action clock | State clock | Servo and protective state | Physical joint |
|---|---|---|---|---|---|
0.00 s | Request G17 leaves for semantic interpretation. | Successor work is requested. | S880 records current position under epoch 4. | ACTIVE; local servo continues periodic ticks. | Inside local envelope. |
0.10 s | No response yet. | C42, derived from S880, begins its admitted interval. | New samples continue. | Points from C42 pass time, epoch, freshness, and envelope checks. | Joint moves toward its bounded target. |
0.20 s | No response yet. | Next chunk computation is stalled. | Samples S881... show the changing joint. | Servo tracks the current admitted point; it does not await the planner. | Motion continues only under C42. |
0.30 s | No response yet. | Validity boundary: C42 expires. | A fresh sample records the expiry state. | ACTIVE -> DEGRADED_HOLD, cause ACTION_TTL; illustrative zero-velocity command begins. | Position holds inside the simplified local envelope. |
0.31–1.19 s | Request remains outstanding. | No chunk has valid authority. | Samples continue through approximately S1000. | Servo and state activity continue in degraded mode. | No semantic result extends motion. |
1.20 s | Late-result boundary: response for G17 arrives. | Candidate C43 is tied to stale source state and is rejected. | Current state is far newer than S880. | No actuator effect occurs until fresh state and local admission create new authority. | Joint remains in the illustrative hold. |
Caption-equivalent explanation. Relay's current action remains executable only through its declared validity boundary. State and servo clocks continue while the next semantic result is delayed. When valid action expires, the local protective state enters its configured degraded behavior. The response at 1.20 s has no physical effect until a fresh source state, matching epoch, unexpired action, uncancelled goal, and local envelope check admit new work. The trace proves only that the timing and authority relationship can be represented; it does not specify a physically safe response or recommended timing for a real robot.
The gap from 0.30 s to 1.20 s is not dead time in the robot. It is time in which the slower producer has no valid physical authority while faster layers continue enforcing their contracts. This is the key diagnostic. Logs that record only model request and response would show a 1.2-second latency. Logs that record only servo ticks would show uninterrupted execution. Joining goal, source state, chunk, point, command, protective transition, and plant state reveals why uninterrupted computation did not imply uninterrupted permission to move.
Placement is decided per operation
“Local or cloud?” is too coarse because one robot contains operations with different inputs, deadlines, compute costs, privacy exposure, and failure consequences. A cloud service can be the right place to evaluate a candidate model across millions of stored examples while being the wrong place to limit phase current. A robot can use remote semantic reasoning without granting the wide-area network direct actuator authority.
Five credible patterns cover much of Relay's design space. None is universally best.
| Pattern | What owns immediate action | When it wins | Cost or risk moved elsewhere |
|---|---|---|---|
| Fixed classical skill graph | Named local skills and controllers selected by explicit state transitions | Task variation is low, success predicates are crisp, compute is constrained, and validation or maintainability dominates semantic flexibility. | Engineers author transitions and recovery coverage; novel instructions outside the library fail closed or require operator handling. |
| Entirely local VLA | An on-robot VLA proposes chunks, with local executors, servos, and protection still enforcing bounds | Connectivity is unacceptable, privacy is strict, the hardware can carry the pinned model, and the required task distribution fits local inference and thermal budgets. | Robot cost, power, heat, model packaging, and per-device update complexity increase; “on-device” still does not imply deterministic execution. |
| Site-edge inference | A nearby server serves one or more robots; each robot retains local admission and fallback | Several robots can share a larger accelerator, site networking can be engineered and observed, and low typical latency materially improves throughput. | Site network and server availability become operational dependencies; tail latency, queueing, isolation, and partition behavior still need contracts. |
| Split cloud semantic plus local skill/action | Remote service handles expensive interpretation or planning; robot edge grounds, admits, executes, and degrades locally | Semantic work benefits from larger or frequently updated models, its delay is tolerable, and Relay can finish or cancel bounded local work during remote loss. | Interfaces must preserve source state, goal identity, validity, privacy, and cancellation; local skill coverage remains necessary. |
| Asynchronous cloud data, evaluation, and training | No immediate physical authority; cloud proposes future artifacts and rollout decisions | Large storage, cross-robot evidence, expensive training, replay, comparison, and coordinated release dominate. | Data selection, lineage, privacy, evaluation leakage, signing, canarying, and rollback become first-class engineering work. |
The fixed graph deserves a strong case. If Relay handles twenty known tools in a controlled crib with a stable drawer and a guarded cell, an explicit sequence of inspect, approach, grasp, verify, transfer, place, and close may be cheaper to validate and easier to diagnose than a general policy. Language can select among those skills without producing motion. The graph loses when task variation and exception handling make authored transitions the dominant cost, but “classical” is not a synonym for obsolete.
An entirely local VLA wins a different workload. A field robot with intermittent connectivity may need all task-relevant inference on the machine. A compact model can reduce network dependence, preserve local data, and simplify some latency paths. It still sits above current control and protection unless the complete learned path has separately met those obligations. Local memory pressure, accelerator contention, thermal throttling, warmup, and software restart are forms of variable latency too.
Site-edge inference is often the neglected middle. A workshop can place an accelerator one switch away and amortize it across several robots. Typical response may be far better than a distant region, while model maintenance stays centralized. That is a real architectural advantage, not an exception to the chapter. The robot still requires expiry and degraded behavior because congestion, server restarts, switch faults, and competing requests can extend the tail. Median latency helps throughput; the authority contract governs what happens outside the median.
The split semantic/local design matches Relay's opening case. A remote component can interpret an unfamiliar instruction, retrieve documentation, or revise a long plan. The local stack decides whether the proposed subgoal is available, whether current state still supports it, and how to execute it. One first-party/private frontier report from July 2026 describes separate Gemini Robotics 2 action, embodied-reasoning (ER) 2 planning/progress/recovery, and On-Device 2 local variants across disclosed embodiments, with embodiment adaptation rather than zero-shot arbitrary-body control. That report is evidence that one private model family used differentiated roles, not proof of a universal architecture, independent reliability, or superiority. Google DeepMind Gemini Robotics 2 production report
Asynchronous cloud work has the widest safe timing margin because it changes future releases rather than present current. Relay can upload selected logs after reconnect, run evaluation, train, and receive a signed candidate. First-party platform documentation offers bounded examples: Intrinsic's specification effective July 8, 2026 says its probabilistic artificial-intelligence (AI) services do not override robot-controller safety systems, while Viam documents local capture, offline buffering, and resumed synchronization. These sources define their own platform boundaries; they are not comparative fleet-reliability studies and provide no bound on wide-area latency or cloud availability. Intrinsic technical specifications and Viam synchronization documentation
The placement rule is to use shared compute and information where they earn their cost while containing the consequence of delay or loss. Semantic interpretation, fleet coordination, evaluation, training, and artifact storage can gain enormously from remote resources. The fastest protective behavior stays with the component that can observe the relevant physical state and act within the required failure horizon. If an engineered remote path can meet a named deadline and availability obligation, it may own more. The contracts make that expansion reviewable.
The message boundary in code
The software lab makes the action contract executable with immutable Python records. This compact excerpt omits transport and controller implementation while preserving the fields required for the failure tests:
from dataclasses import dataclass
@dataclass(frozen=True)
class StateSample:
epoch: int
seq: int
produced_ns: int
position_rad: float
velocity_rad_s: float
@dataclass(frozen=True)
class Subgoal:
epoch: int
goal_id: str
produced_ns: int
valid_until_ns: int
target_position_rad: float
max_velocity_rad_s: float
@dataclass(frozen=True)
class ActionPoint:
epoch: int
chunk_id: str
source_state_seq: int
execute_at_ns: int
valid_until_ns: int
target_position_rad: float
@dataclass(frozen=True)
class ServoCommand:
epoch: int
command_seq: int
produced_ns: int
velocity_rad_s: float
reason: str
Immutability prevents a queued record from silently changing after validation. It does not validate the record. The local executor must compare epoch with the active system epoch, confirm that now_ns <= valid_until_ns, check that execute_at_ns is neither obsolete nor unreasonably far ahead, look up the age of source_state_seq, reject a target outside the physical envelope, honor cancellation, and require an ACTIVE protective state. Model name and version, semantic request ID, calibration and schema versions, and richer frames belong in the production message; the excerpt stays small enough to expose the time and authority mechanism.
Bounded queues are part of the contract. State is usually replaceable: a newer valid sample can supersede an older sample. Commands require more care. A reliable unbounded first-in, first-out queue can deliver every old action after a consumer reconnects. That behavior may satisfy delivery semantics and fail the robot. The consumer must revalidate each item, and the queue needs an explicit maximum plus a policy for replacement, rejection, or backpressure.
Lab: make a slow model fail without freezing the robot
This lab is software simulation only, using Python 3.11 or later and the standard library. It models one position-controlled joint rather than rigid-body dynamics. The illustrative plant integrates velocity at 1 kilohertz and enforces [-1.0, 1.0] radians; the servo runs at 500 hertz; state publishes at 100 hertz; the action policy is requested nominally at 10 hertz and returns five targets spaced 20 milliseconds apart. These numbers make failures visible. They are not recommendations for any physical robot.
The semantic planner translates move to +0.6 rad into a bounded subgoal and has no plant or servo handle. The servo never awaits the planner or policy task. Events are written as JSON Lines with monotonic time, component, epoch, type, and joined goal, state, chunk, point, command, and transition identifiers. The output includes tail observations, not only averages: servo-period percentiles and maximum, deadline misses, state age at policy use, policy latency, action age at execution, rejection counters, queue drops, degraded transitions by cause, final and maximum position, and time from loss of valid action to a zero-velocity command.
Run the following core fault scenarios:
| Scenario | Injection | Required observation |
|---|---|---|
| Baseline | Fixed illustrative semantic latency of 40 ms and policy latency of 20 ms. | Target is reached within tolerance; the servo never waits on inference; all executed points are valid. |
| Remote semantic stall | After motion begins, delay a planner response by 1.2 s without stopping state or servo. | Existing bounded work may finish; expiry causes degraded behavior; the late semantic result remains advisory until local validation. |
| Policy stall during a chunk | Delay one policy response by 300 ms; compare whole-valid-chunk execution with at most one point beyond the last replan boundary. | Both stay inside the local envelope; the trace exposes latency hiding versus open-loop exposure. |
| Stale observation | Pause state publication for 150 ms with maximum permitted state age of 60 ms. | Output derived from stale state is rejected; new action waits for fresh state. |
| Old process after restart | Increment the epoch, then deliver one old-epoch chunk. | Every old point is rejected even if its time interval appears current. |
| Unbounded reliable FIFO counterexample | Replace bounded latest-valid behavior with an unbounded reliable queue, stall the consumer, then reconnect. | Old commands execute late in the deliberately failing version; restoring expiry and epoch validation makes the scenario pass. |
The lab passes only when all conditions hold in three deterministic runs per scenario:
- No plant command executes outside the workspace or velocity envelope.
- The servo task never awaits a semantic or policy call.
- No expired, stale-source, or wrong-epoch action reaches the plant.
- Loss of valid action produces zero velocity within the configured local timeout plus one servo period.
- Queue length never exceeds its declared bound.
- Every protective transition records a cause and the last accepted state and action identifiers.
- The output contains a complete causal trace, not only aggregate metrics.
- The reliable-FIFO counterexample fails before validation is added, then passes after expiry and epoch checks return.
Reaching +0.6 rad while violating any condition is a lab failure. The target is only one outcome predicate; the contracts define whether the path was admissible and inspectable.
For this simulation, DEGRADED_HOLD emits zero velocity. A physical mechanism may instead require braking, gravity compensation, controlled retreat, removal of motor power, or another hazard-derived response. Zero velocity is not a universally safe state. Hardware extension belongs only on a low-energy guarded bench with an independent physical emergency stop, conservative current, speed, and workspace limits, and a separate risk review. Do not reproduce network-loss faults by removing or bypassing a protective controller.
Forecast dated 2026-08-31
The following statements are book synthesis, dated 2026-08-31, with a horizon through 2030. They are forecasts built from current mechanisms and constraints, not empirical facts about 2030.
| 2030 book-synthesis forecast | Premises visible at the cutoff | Falsifier or material narrowing condition |
|---|---|---|
| Robots will remain hierarchies of clocks even if several clocks share one model or processor. | Published chunked and asynchronous action mechanisms, maintained periodic control surfaces, operating-system scheduling, and slower fleet evaluation all expose distinct state and deadline obligations. | Material robot decisions repeatedly cannot be placed in the six clocks without hiding another cadence that changes authority, state, or release behavior; the taxonomy must then be revised. |
| Cross-layer commands will need the five contracts in proportion to physical consequence, though implementations may collapse them. | Current interfaces separately expose action horizons, lifecycle and transport behavior, controller loops, signed update metadata, and correlated telemetry. | Representative fault, restart, stale-data, and upgrade tests omit one or more contracts without creating ambiguity about permissible effect, source state, outcome, or rollback. |
| Cloud value through 2030 will appear first and most durably in expensive reasoning, data, evaluation, training, coordination, and bounded rollout; the fastest protective path will retain local fallback. | Compact on-device inference exists alongside private families with separate reasoning/action/local roles; first-party platforms document local controller boundaries and offline data buffering. | Before 2030, independently audited remote paths repeatedly meet named worst-case deadline, availability, cybersecurity, and functional-safety obligations for fastest protective actions across network and service loss without local protective fallback. |
The third forecast is deliberately narrower than “robots will use the cloud.” Many already use remote compute and fleet services in some form; that observation does not determine where physical authority should live. The prediction concerns the consequence of losing a path. If a remote action generator supplies chunks that a local boundary revalidates and can safely outlive or cancel, the design is compatible with the forecast. If deterministic networking and audited remote control mature enough to carry the complete obligation, the authority boundary should move.
The forecast also avoids treating local inference as inherently safe. A local model can stall, overheat its compute module, contend for memory, read stale state, restart into a new epoch, or produce an out-of-envelope target. Removing the wide-area network removes one failure source and one privacy boundary. It does not remove the need for admission, protection, observability, evaluation, and rollback.
Reconstructing Relay v0
Relay v0 now has no unearned mechanical sophistication, but it has a complete causal skeleton. An instruction becomes semantic interpretation. Interpretation becomes a bounded subgoal. A skill or action policy reads fresh state and proposes a finite chunk. The local executor checks source age, time, epoch, cancellation, protective state, and physical envelope. The servo closes faster feedback and the drive produces current. The mechanism changes. Sensors produce outcome evidence. Selected evidence can later justify a versioned change, but only after evaluation and bounded rollout.
The six clocks tell a builder where a delay or change can matter. The five contracts tell a reviewer what must cross each boundary. Action chunking moves some future work into a slower producer without turning generated target cadence into feedback cadence. Local, site-edge, and cloud placement move compute and operational burden; none cancels the physical obligation. The stalled-planner trace makes the core causal claim visible: after C42 expires, late semantic plausibility cannot silently recreate actuator authority.
Transfer the model to a mobile base and the fields remain recognizable while the physical envelope changes to speed, acceleration, stopping distance, localization confidence, and collision margin. Transfer it to a balancing humanoid and local validity horizons shrink; a high-rate whole-body or balance layer and independent protection cannot wait behind semantic or wide-area latency. Add two Relay robots and a cloud-issued shared-passage lease, and the fleet clock may allocate permission while each robot still owns local sensing, stopping, and behavior when the lease or network expires.
What this does not prove
This chapter does not prove that cloud inference is unsafe, local inference is safe or deterministic, the illustrative timing bands are appropriate for a particular robot, a zero-velocity hold is a safe physical state, ROS 2 or a learned policy is hard real-time, a layered architecture is certified, or any cited model is superior.
Chapter 2 starts from the authority path just built and moves one level outward. Before choosing an arm, mobile base, or humanoid form, Relay must define the work envelope, contacts, uncertainty, throughput, economics, and evidence that make a body worth building.
