My Visit to the Startup Building General-Purpose Robots That Learn Like Humans

From Will Knight’s AI Lab newsletter

Last week, I took just a 15-minute trip from my home to witness robots pull off feats so impressive they left me genuinely stunned. I visited the Cambridge, Massachusetts headquarters of a young robotics startup called Generalist AI, where I got a front-row look at the company’s dexterous robot arms tackling everyday small tasks: stacking cups, dropping blocks into bowls, and similar routine chores.

What shocked me most was how quickly the robots solved new problems—they moved with the same quick, intuitive problem-solving sense that a flesh-and-blood person would. After watching just one short instructional video of a task, the arms mastered a whole range of new challenges, and most remarkably, they never needed any specialized task-specific training to pull it off.

One of the most striking demos drove this home. A robot was told to sweep a small block into a bowl using a dustpan and brush. When researchers removed the brush from the workspace entirely, the robot improvised on the spot: it repurposed the dustpan as a brush, and flicked the block straight into the bowl. In another test, a dual-armed robot watched a short clip of a person unzipping a purse and pulling out banknotes. I watched open-mouthed as the robot successfully unzipped an entirely different style of purse, then carefully removed the notes on its own. Even more incredible, when its right gripper couldn’t get a solid hold on the cash, it automatically switched to its left gripper to get a better angle. “Ha,” one nearby engineer remarked after the demo. “It’s never done that before.”

“This is exactly the kind of capability that got everyone so excited about GPT-3,” Pete Florence, Generalist AI co-founder and CEO, told me, referencing OpenAI’s groundbreaking 2020 large language model. “With that model, you could just prompt it for a new task and it had a real shot at pulling it off. That’s what we’re building here for robots.”

Generalist AI centers its work on teaching robots the basic physics of the physical world, an approach inspired by the intuitive understanding of how objects work that humans develop from early childhood. This framework seems to be the key to the model’s ability to transfer skills learned in one scenario to entirely new, untested situations. After watching the company’s demos, I couldn’t help but compare the robots’ improvisation to how young children experiment and problem-solve when shown a new task. Researchers themselves are often surprised by the creative solutions the robots come up with—for example, one robot used a banana left in its workspace to sweep up small objects, a trick it was never taught.

While this might sound trivial, physical common sense is still a major gap in modern machine intelligence, and studying how infants learn about their world so efficiently has given AI researchers critical new insights for building smarter robots.

I met Florence and co-founder/CTO Andrew Barry in a conference room overlooking teams of researchers training robot systems with special motion-capture grippers. The third co-founder, chief scientist Andy Zeng, and the whole founding team come with impressive credentials: all three previously worked at Google DeepMind and Boston Dynamics on some of the world’s most advanced robotic hardware and models.

Traditional AI robot training requires feeding a model thousands of pre-labeled examples of a specific task to get it to work. This method is notoriously unreliable, too: even a small change, like a shift in room lighting, can leave a robot completely unable to complete a task it was trained on. Generalist AI and a small group of other cutting-edge robotics startups are investing heavily in a new approach: general-purpose robotic models trained via human demonstration.

To collect training data, the company builds special glove-like devices that mimic robot grippers, with built-in cameras to capture human movement as people complete a wide range of chores. I saw a shipping crate stacked with hundreds of these gripper gloves, ready to be sent to data collectors in Mexico and other locations around the world. Florence and his team are cagey about the exact details of their training methodology, but confirm they have already amassed an enormous dataset of high-quality physical interaction data. Unlike many other teams working on smarter robots, Generalist AI also built its entire AI model stack from scratch, rather than relying on existing open-source language models.

Danfei Xu, a Georgia Tech roboticist familiar with Generalist AI’s work, says the startup stands out among groups pursuing general robot intelligence. “They’ve pushed this concept to the extreme, and they’ve executed incredibly well,” Xu says. Beyond their massive high-quality dataset, he adds, “they are excellent roboticists, and they’ve done really rigorous, groundbreaking science.” Xu also notes that the demos Generalist has released so far signal the company is already focused on real commercial deployment. “They are the closest to something that’s actually deployable,” he says.

Karen Liu, a Stanford University roboticist who also follows the company’s progress, echoed that assessment. “Generalist’s data approach collects physical interaction data at large scale without tying it too closely to one particular robot hardware,” Liu says. “Their strongest results suggest that this bet may be working.”

That said, Generalist AI is transparent that its models’ learning skills are not yet reliable enough for widespread use. On average, robots successfully complete a newly demonstrated task only around 59% of the time; for commercial use, the ideal success rate would be 99% or higher. It also remains unclear how well these adaptive skills will generalize to every possible task and real-world environment.

Even so, the potential for this kind of quickly adaptive robot in sectors like manufacturing is enormous. That potential was highlighted by an unexpected spontaneous moment one recent evening, captured on camera: an engineer was stacking small cups on a table in front of a dual-armed robot, just testing to see what the machine would do. Out of nowhere, the robot joined in, grabbing extra cups with its two grippers and adding them to the stack, finishing off a neat, orderly pile on its own. As the robot finished, the engineer broke out into delighted, surprised shouts, excited by the robot’s unscripted problem-solving.

---

This is an installment of Will Knight’s AI Lab newsletter. Read previous newsletters here.