Teaching a robot everyday tasks requires varied practical experience.
The fastest way to explain what a robot does is to show a video. A lengthy explanation of how a control algorithm is optimized may not get the idea across. Show a four-legged robot running and jumping over obstacles, and people understand immediately. “It runs well.” That is enough.
But the more impressive the movements become, the stranger the question that follows: if robots can move that well, why is it still difficult to bring one home and leave the laundry to it?
Laundry-folding demos exist. The question is whether the robot can keep working in a different home, with unfamiliar clothes, or after a slightly awkward grasp. People's expectations are higher than they might seem. They want a robot that understands the situation and responds on its own, beyond performing a prescribed movement successfully once.
The approaches attracting attention in robotics keep changing as the field tries to meet those expectations. Model-based control, reinforcement learning, imitation learning, VLAs, and world models all compete. Looking at these technical developments and the industry around them, I keep returning to two things.
First, the amount of data a robot can use for learning and decision-making matters. In particular, throughput—the number of samples it can obtain and use per unit of time—is an important way to understand the current competition. Faster simulation, easier collection of human demonstrations, and learning from existing video all address this problem.
Second, people bet on people. Who is building the technology, what research they have done, and who they are connected to influence trust and investment alongside the technology itself. From where I stand, a substantial part of the money follows the people behind the technology, sometimes more than the technology itself.
Faster Simulation Changes How We Teach Robots
Playing Go well seems much harder for a person than folding laundry. Why, then, did AlphaGo achieve such a powerful result in Go first?
Go has explicit rules. Once a stone is placed, the next state can be calculated, and many games can be played without moving pieces on a physical board. With laundry, the shape of the fabric changes depending on where it is grasped. Friction, wrinkles, and hidden surfaces all affect the outcome. Generating experience in the same way is much harder.
The ability to predict the world matters in robot control, too. Model predictive control, or MPC, uses a model to predict future motion and calculate actions. It executes part of the plan, then solves the problem again from the new state. An accurate model makes this powerful, but the calculation must finish while the robot is still moving. Spend too long deciding where to place a foot, and the robot has already fallen.
Reinforcement learning, or RL, can shift much of that burden into trial and error performed in advance. Desired behaviors—moving forward, staying upright, and avoiding unnecessarily large forces—receive rewards. The result of repeated learning in simulation is encoded in a policy. A policy is essentially a function that takes what the robot observes and produces its next action.
Locomotion was a particularly good fit. Goals can be expressed relatively clearly, and many simulated robots can move at the same time. Varying ground friction, body mass, and external forces lets a robot accumulate experience that would take a long time to gather with a single physical machine.
Parallel simulations across varied environments let robots gain experience that would be difficult to collect physically.
This should not be read as a story in which MPC ended and RL won. MPC can also use GPUs to evaluate many candidate actions in parallel. A learned policy can be combined with real-time planning. As computing conditions change, methods that were previously difficult can become useful again.
When I say the amount of data matters, I mean how many possible situations a robot can examine and experience. An accurate model belongs in that picture, too: it lets us predict the results of many situations without physically experiencing each one. If the model is inaccurate, simply running it faster is not enough. The accuracy of the experience matters alongside the amount of data and its throughput.
When a Goal Is Hard to Score, a Person Can Demonstrate It
Applying a method that works well for walking to household work introduces another bottleneck. Cloth, cables, and food deform easily and are difficult to manipulate. Simulating them both realistically and quickly is already a substantial challenge.
Deciding what counts as success becomes more complicated, too. Distance traveled can be turned into a score relatively easily. But what does it mean to fold laundry well? Flattening a garment, arranging its sleeves, and making the final folds all need to be captured in the desired behavior. Writing each part into a reward becomes a large task.
One option is to have a person show the robot what to do. This is imitation learning, or IL. We collect pairs of observations and actions, then learn the relationship between them. Demonstrations can be created by remotely operating a robot or physically guiding its movements. A goal that is difficult to describe mathematically may be easy to demonstrate by hand.
Collecting demonstrations does not make everything work automatically. Imagine a dataset containing examples of people going around an obstacle on its left and on its right. Both are correct. Average the two, however, and the result may lead straight into the obstacle. There can be several valid answers to the same action problem.
Averaging two valid paths, one to the left and one to the right, can lead to a collision.
Diffusion Policy is one way to address this. It learns a distribution of possible actions and generates actions from that distribution. Predicting a short action sequence also helps it handle continuous movement. Progress in imitation learning has depended on how demonstrations are represented as actions, as well as on collecting the data itself. Diffusion Policy research
Small mistakes can still accumulate when a robot encounters situations outside its demonstrations. Reproducing a demonstration well and recovering independently after a failure remain different capabilities.
A vision-language-action model, or VLA, connects this to vision and language. It takes a camera view and an instruction such as “put the cup next to the bowl,” then produces robot actions. The hope is that learning from many tasks and environments will make it possible to adapt to new tasks with relatively little additional experience.
RL and IL have been studied alongside each other for a long time and can be combined. Each has shown strengths in different settings: some tasks lend themselves to rewards and simulation, while others lend themselves to human demonstration. Attention shifts for the same reason. Which method is advantageous depends on where the desired experience can be obtained more easily.
When Demonstration Collection Becomes the Bottleneck, the Interface Changes
Scaling imitation learning quickly runs into the limits of human time. An hour of task experience requires someone to spend an hour demonstrating in front of a robot. Tasks, homes, and objects all differ. How many situations can we cover if a robot and an operator must always be brought to the same location?
This is why I pay attention to interfaces such as UMI. UMI is a device and learning framework that lets people demonstrate tasks using a handheld gripper and record the experience in a form a robot can learn from. A gripper is the robot's mechanical clamp for holding objects.
A person uses the gripper to move cups or fold clothes. A camera attached to it observes the scene, while the system records the gripper's motion and opening and closing actions. Matching the viewpoints and gripper shapes used by the demonstrator and the robot reduces the gap that arises when converting footage of a person's bare hands into robot actions. UMI research
Matching the human and robot grippers and viewpoints reduces the gap when transferring demonstration data.
The crucial change is that a robot arm does not need to be present while the data is being collected. A person can carry the device into another home or workspace. Experience is no longer confined to the laboratory in the same way.
Seen this way, interface design is a technology that affects learning performance. What gets recorded, and how executable robot motion is captured, determine the amount of usable experience. A system is also needed to collect demonstrations consistently across environments and connect them to learning. UMI matters because it makes this process easier, increasing both the amount of data available to robots and the speed of collection.
The popularity of two-finger grippers can be understood through the same lens. A surprising number of tasks can be performed without reproducing all five human fingers. A competition to teach those tasks quickly and improve performance became viable first. More complex hands will also need ways to teach them to use their capabilities fully.
Recording gripper motion does not solve every manipulation problem, though. Pressing hard on an object may require changing the arm's posture and transferring body weight into it. Even when the fingertips move to the same position, the way the whole body transmits force can differ. As more fingers and whole-body actions enter the picture, both the information that must be recorded and the differences between human and robot bodies grow.
I think this question belongs in any discussion of hardware performance: can we actually teach this body to use the capabilities it has?
Motion Learned from Video Still Has to Work in a Physical Body
As the cost of collecting human demonstrations rises, another question naturally follows. Could robots learn from video that already exists?
The internet contains videos of people moving objects, cooking, and playing sports. They offer the possibility of drawing on much broader experience than demonstrations collected from scratch specifically for robots. That possibility is attracting interest in world models that learn how the world changes through video.
A world model predicts what will happen next from the current state. Research on world action models, or WAMs, connects this to action by considering future scenes and the actions that produce those changes together. The aim is to learn patterns of physical change from video and use them in robot behavior.
It is difficult to say that we have reached a stage where humans no longer need to specify anything. Knowledge learned from video still has to be connected to robot control, and adaptation data is needed for new bodies. DreamZero, for example, uses a video model while also learning actions from robot data and explicitly addressing adaptation to new embodiments. World Action Models are Zero-shot Policies
Watching a lot of basketball does not mean everyone can dunk. The same applies to robots. Their arm lengths and masses, and the forces and speeds their joints can produce, differ from those of humans. A movement that looks plausible in a video still has to be checked against what this particular robot can execute.
Matching poses is especially insufficient for fast throwing or jumping. The timing, direction, and magnitude of forces matter. Dynamics describes these relationships between force and motion. More video does not remove the physical limits of hardware.
That is why I do not assume every problem will eventually be solved by adding more data. Accumulating similar scenes differs from accumulating experience that helps explain new situations. Training also costs compute and time. Methods that adapt well with little experience remain necessary.
Even so, data is the question running through today's technical competition. Can experience be generated at scale in simulation? Can people leave demonstrations more easily? Can existing video be used for learning? The methods differ, but the push to use more experience more quickly continues. I do not know which method will ultimately be the answer, but I see this competition over throughput as an important current in robotics today.
People Bet on People
People's abilities, experience, and established trust shape expectations of technology and investment.
Alongside how robots are trained, I pay attention to who starts companies and where the money goes. The connections among people stand out. Follow promising companies, and people linked through the same laboratories and collaborative research keep appearing.
Sunday's founders, Tony Zhao and Cheng Chi, worked on ALOHA and on Diffusion Policy and UMI, respectively. Experience building systems for teaching robots through demonstrations and methods for learning from those demonstrations came together in one company. Sunday's team
These connections are part of what I have come to consider important while observing Silicon Valley. People who studied in the same lab or worked on problems together move to different companies, start new ones, and follow one another's technology. The company name may be new, while the experience behind it has been accumulating for years.
These relationships mean more than knowing a lot of names. People who have read the same papers and worked on similar problems can understand one another's technology more easily. Existing relationships also make it easier to learn early about what others are preparing, work together, or begin a new effort. Networks accumulate trust and information alongside technical understanding.
I think a substantial part of capital also moves according to these people and relationships. The harder it is for an outsider to judge the technical details, the more important it becomes to know who has worked on the problem and through whom that person can be trusted. A background in a well-known lab, assessments from former colleagues, and established relationships influence the expectations a company starts with.
It is difficult to separate belief in a technology's potential from belief in a person. Still, when considerable expectations and capital gather before a company has fully demonstrated itself through products and revenue, I find it hard to conclude that only the technology is being evaluated. “People bet on people” is an observation about that reality.
This does not mean that only established groups can succeed. But it is worth understanding the starting positions of the competition. Add data collection systems and training infrastructure to years of accumulated experience, and implementing an algorithm from a paper may no longer be enough to catch up. Real money and elapsed time can create substantial gaps.
A company's interests also matter when evaluating what it says about its technology. It has reasons to describe its chosen approach as the most promising one. Neither investment size nor impressive demos reveal the eventual winner. Even if companies merge or methods acquire new names, accumulated experience and people who can work together are likely to carry into the next effort.
To understand technical progress, we need to examine what experience reaches a robot, how much of it there is, and how quickly it can grow. To understand why capital moves, we need to examine who builds the technology and whose trust they have earned. Follow only algorithm names and investment figures, and it is easy to miss those two things.