A perspective from the data layer of humanoid robot training
I recently spent time working on egocentric video data — first-person footage of humans doing everyday things with their hands — and how it could be used to train humanoid robots. It’s a narrow problem, but it forced me to answer a bigger question: what do these models actually need to learn, and why is it so hard to give it to them?
That work pulled me into the two architectures competing to become the brain of the next generation of robots: Vision-Language-Action models and world models. I’m not a researcher in either. But sitting at the data layer gives you a particular vantage point — you see what these models consume, where they’re data-hungry, and where more data stops helping. This post is my read on where the field is heading, from that vantage point.
What Comes After LLMs?
LLMs learned the statistical structure of text. The next wave of AI has to learn the statistical structure of the physical world — how objects move, how forces act, what happens when you push, pour, or grasp. The vehicle for that transition is the humanoid robot.
For sixty years, robotics meant programming a machine to do one thing precisely, in a cage, because it couldn’t adapt to anything outside its parameters. What changed is the same thing that changed everything else: transformers that generalize across modalities, vision-language models that reason jointly over what they see and what they’re told, and compute cheap enough to train at scale. Teaching a robot a task used to be a years-long research project. Companies are now doing it in months.
From LLM to VLM to VLA
The path to today’s robot brains is a straight line through three architectures:
LLM → VLM → VLA
An LLM takes text in and produces text out. A VLM (vision-language model) adds a vision encoder — now the model takes images and text in, and produces text out. It can describe what it sees, answer questions about a scene, reason about spatial relationships.
A VLA takes the final step: images and language instructions in, robot actions out. The key insight — first demonstrated at scale by Google’s RT-2 in 2023 — is that robot actions can be treated as just another token type. Discretize continuous motor movements into tokens, and the same transformer that predicts words can predict actions. Everything the model learned from web-scale image data transfers into control: a robot that never saw a banana in its training demonstrations can still pick it up, because it knows what bananas look like from the internet.
The lineage since then: RT-1 proved robotics transformers work at scale (2022), RT-2 brought web-scale generalization (2023), OpenVLA open-sourced the recipe (2024), and Physical Intelligence’s π0 pushed it furthest with diffusion-based action generation (2024).
And the results are genuinely impressive. Physical Intelligence’s demos are the ones that made me stop and pay attention: a robot folding laundry — deformable objects, one of the classically hard problems in manipulation — and clearing a table into a dishwasher. Most striking, doing this in homes it had never seen. That last part matters more than the tasks themselves: the promise of a VLA is precisely that it generalizes to environments outside its training data, the way an LLM answers questions it was never explicitly trained on.
Where VLAs Hit a Wall
But working at the data layer, you start to see the shape of the ceiling. Three failure modes show up consistently:
Long-horizon tasks. Compound errors are brutal: 95% per-step reliability gives you 60% success on a ten-step chain. Folding a shirt is one skill; cooking a meal is a sequence of dozens, and today’s VLAs degrade sharply as the chain grows.
Instruction drift. As a task progresses, irrelevant observations dilute the model’s attention to the original instruction. The robot does things that are locally reasonable but globally wrong — it forgets what it was trying to do.
Generalization limits. The environment generalization is real but shallow. Change the physics of the task — not just the room — and success rates collapse. More demonstrations help less and less.
The common thread: a VLA is fundamentally reactive. It maps what it sees to what it should do next. It has no mechanism for asking “what happens if I do this?” before doing it. And that question, I’ve come to believe, is the whole game.
World Models: Learning to Predict, Not Just React
A world model learns to simulate: given the current state of the world and an action, predict what the world looks like next. Where a VLA asks “what should I do given what I see?”, a world model asks “what will happen if I do this?” — and can plan through imagined futures before committing to real ones.
Yann LeCun has been arguing for this direction longer than almost anyone, and his JEPA architecture is worth understanding because it’s a genuinely different bet from the transformer lineage above.
The obvious way to build a predictive model is to predict the future in pixel space — generate the next video frame. JEPA (Joint Embedding Predictive Architecture) rejects this. Its core idea: most of the pixel-level future is unpredictable and irrelevant. The exact ripple pattern of spilling water, the flutter of a curtain — no model can predict these, and no robot needs to. JEPA instead predicts in representation space: it learns abstract embeddings of world state and predicts how those embeddings evolve. It learns that the glass will fall without simulating every photon of the falling. That’s much closer to how humans model the world — we predict consequences, not video frames. LeCun’s blunt version of the argument: you cannot get to physical intelligence by scaling next-token prediction, because the world is not made of tokens. JEPA is now the foundation of his post-Meta startup, AMI Labs.
Fei-Fei Li’s World Labs comes at it from a different angle: spatial intelligence. Her June 2026 taxonomy splits world models into three functions — renderer (generate visual representations), simulator (model how objects respond to forces), and planner (reason over action sequences). Their product Marble generates persistent, navigable 3D environments: a robot training in one can approach the same shelf from different angles and find the same objects in the same places, because the geometry is consistent rather than regenerated per frame.
NVIDIA’s Cosmos attacks the problem I lived inside: data. Nobody has enough real robot demonstrations. Cosmos uses video world models to generate physically plausible synthetic training data. The model problem and the data problem become the same problem — better world models generate better data, which trains better robots.
The Philosophical Fork
Underneath the company logos, the field has split on one question: where does world understanding live?
The structure camp — LeCun, World Labs, Cosmos — says you must build it explicitly. A separate, inspectable model of how the world works, with policies plugged into it.
The scale camp — Physical Intelligence and the VLA lineage — believes world understanding emerges inside the weights if you train on enough diverse data, the same way LLMs developed reasoning nobody explicitly built. Some researchers claim to find world-model-like internal representations when they probe VLAs. I’m skeptical of how far that goes. A VLA translates observations into motions; nothing in its architecture simulates the world or checks a plan against predicted consequences. Calling that an implicit world model is, for now, the scale camp’s hope more than a demonstrated fact.
Google DeepMind’s Gemini Robotics is the most interesting data point, because it splits the difference architecturally. It pairs two models: Gemini Robotics-ER, an embodied reasoning model that plans multi-step tasks, calls tools, and can pull live context (their demo: sorting trash into bins using recycling rules fetched from the web), and the Gemini Robotics VLA, which turns each planned step into motor commands. To be precise, ER isn’t a world model — it doesn’t simulate physics; in Fei-Fei Li’s taxonomy it’s a planner, not a simulator. But the design concedes the structural point: reactive execution alone isn’t enough, and the reasoning layer has to be explicit and separate.
The open ecosystem is converging the same way. Hugging Face’s LeRobot has become the de facto open toolkit, NVIDIA’s GR00T models are openly available for post-training, and Cosmos integration is bringing world-model-generated synthetic data to teams that can’t collect real data at scale.
My Take: VLAs Are the Present, World Models Are the Future
VLAs work today and will keep improving. For well-defined tasks over short horizons, they’re a real solution now, and the laundry-folding demos aren’t tricks.
But the ceiling is structural, not incremental. Long-horizon planning, genuine generalization, acting safely in novel situations — these require predicting consequences before acting, and a reactive observation-to-action mapping has no place to put that capability. You can’t fine-tune your way to a simulator.
World models have their own hard problems — sim-to-real gaps, compute cost, learning accurate physics from video. But these look to me like engineering problems on the right path, whereas the VLA ceiling looks like the wrong abstraction for the long game.
My prediction: world models won’t replace VLAs — they’ll become the layer underneath them. World models handle prediction, planning, and synthetic data generation; VLAs handle dexterous real-time execution. Gemini Robotics’ reasoner-plus-executor split is an early, partial version of this stack, and I expect the rest of the field to converge on it whether or not anyone names it.
Watching what these models need from their training data is what convinced me. VLAs are data-hungry in a way that suggests they’re memorizing behavior rather than understanding the world — and the physical world is too varied to memorize. At some point, a robot has to stop pattern-matching and start predicting.
That’s what world models are built to do. That’s why I think they win.
Sreenivas Makam's Blog 