Ep#109: Enigma - Connecting 100 Robots To The Internet
Shownotes
What happens if you connect a hundred AI-powered robots to the internet and just let people give them any instructions they want? Enigma founder Jonathan Jacobi joins us to talk about his experience deploying a fleet of robots on the internet, keeping them operational, and watching people truly test the generality and robustness of frontier robotics models.
Watch Episode 109 of RoboPapers with Chris Paxton and Ruijie He today!
Learn More
Robots.online website (no longer active)
Transcript
Transcript is automatically generated by Riverside and edited by ChatGPT.
Chris Paxton (00:05)
Hey everyone, and welcome to another episode of RoboPapers. Today we’re super excited to have Jonathan Jacobi from Enigma here to tell us about their work in foundation models and about putting a hundred robots online.
So, could you introduce yourself?
Jonathan Jacobi (00:19)
Yeah. Thank you, Chris. Thank you, RJ. It’s a pleasure to be on RoboPapers. Big fans.
A bit about Enigma: we’re a company working on foundation models for robotics, as Chris said, as well as building innovative new interfaces for robotics. How should we interact with robots? How should the interface feel to users?
We’re also interested in the back-and-forth between interface work and research on the models.
About a month and a half ago, we put a hundred real robots online. You could actually interact with them over the internet. They were running our models through a variety of interfaces, and you could simply ask them to do different things.
So we’re here to talk about the models, the interfaces, and the experience we’ve built.
Chris Paxton (01:21)
Awesome. So what did people do with all these robots that you put online?
Jonathan Jacobi (01:25)
The basic concept was that we wanted people to feel robotics today.
We all see a lot of demos. We see a lot of cherry-picked demos—some of them are cherry-picked, some of them aren’t—and we wanted people to feel that something is here right now.
One of the problems is that not enough people have robots in their houses. So how do we make enough people feel that robotics is happening now, while we still have this challenge of not having enough robots deployed in the world for the average person to use?
We thought quite hard about how to make that possible. One of the best ideas we came up with was: maybe we can set up a huge warehouse with a lot of robots, put our models on those robots, and let anyone online interact with them.
We also got a very cool domain: robots.online.
We tried to create enough different setups that one robot could be in one scenario, another robot could be in another scenario, and people would have enough flexibility to do random things that they could really feel the AI work we’ve done.
They could ask the robot to do things themselves, without the logistical challenge of putting a hundred humanoids all over the world.
That’s why we used tabletop robots—arms that we customized and designed ourselves, building on public open-source designs—and let people interact with them freely.
Some robots were artists. People could paint. People could do pantomime. People did very unexpected things.
Someone asked one of the robots holding a paintbrush to pretend that the brush was a broom.
Jonathan Jacobi (03:50)
So it actually swept the floor with the paintbrush.
We have some very cool examples of that that we can walk through later.
The concept was that, at Enigma, we focus on generalization—text, video, audio—and we have very different interfaces and input methods for our models. We wanted people to be able to experience that generalization directly and see that it’s working today.
Some of the robots were artists. Some were doing chemistry experiments. We had one that could defuse a bomb. We had one that could do lightsaber-esque fighting.
Not actual lightsabers. Don’t sue me.
But basically, we let people do different things with different robots online.
RJ He (05:01)
Okay, cool. Can you describe a little more of the interface that was set up?
It sounds like each one is a single manipulator fixed to a tabletop, and then the interface is a regular chat-prompt type of thing, as if you were speaking to an AI agent. Is that right?
Jonathan Jacobi (05:21)
Yeah. That’s a good question.
We had a static manipulator. One of them was based on the SO-101 design. We took the SO-101 and said, “We want this to look really polished.”
So we brought in incredibly talented industrial designers and worked with them for about a month or two. They were working full-time on designing this new version.
The robot you’re seeing on the screen is an SO-101 that we customized. The colors, the enclosure, everything was custom-designed. You can see that even the cables are hidden within the robot rather than hanging around it.
These were static manipulators, and each table had a different type of interface.
The artist robot had a chat interface similar to an AI agent. But we wanted to make it less boring than simply asking the robot to paint or perform pantomime, so we turned it into a game.
You could give it an idea and it would paint something associated with that idea, and then you had to guess what it painted.
For pantomime, you could tell it to pretend that an object was something else, and the robot would mime with it.
So that was one chat-based interface.
Then we had other interfaces. Let me walk through some of the things we built.
This is our company page. You can see more than a hundred robots here. This is part of the actual warehouse. Those are spare robots and leftover components that we built.
Here’s one of the chat interfaces where people were doing pantomime. You could do drawing, pantomime, and a few other things.
And here’s another interesting interface.
You could tap objects directly on the screen and then ask the robot to do something with them. You could tap the green bottle and say, “Pick this up,” then tap the orange flask and say, “Pour this into this.”
Jonathan Jacobi (07:45)
You could say, “Throw this here,” or whatever you wanted.
We wanted to introduce a new kind of interactive interface—not just text, not just teleoperation, not just audio, but something new.
Our vision is that if you extrapolate this ten or a hundred steps forward, the interface almost feels like a game. It feels like you’re playing the physical world.
Think about something like StarCraft. Imagine having an interface where you can control a hundred robots. You can tap, type, drag and drop, and build agents on top of that to automate entire processes.
That’s where we imagine the world going. Controlling robots becomes incredibly intuitive: you can type, tap, drag, drop, maybe talk to them.
Another interface was the bomb-defusing robot. You were on a mission to defuse a bomb, and while you’re trying to do it, an operator is shouting at you: “You have to defuse it faster!” So you’re under pressure while talking to and controlling the robot.
People did very unexpected things with this.
Someone said, “Knock on the bomb twice at the bottom.”
Someone else said, “Pour the orange flask over the pink button.”
The pink button wasn’t standing upright—it was lying on the table.
Chris Paxton (09:45)
Can I ask—what simulation is that? That actually looks pretty good. The water physics there.
Jonathan Jacobi (09:50)
This is the physical world. This is not simulation.
Chris Paxton (09:52)
That’s not simulation? Okay. Well, that makes sense.
RJ He (09:54)
No way.
Jonathan Jacobi (09:56)
Yes. Every one of these is the physical world. None of this is simulation.
Chris Paxton (10:01)
Right. I knew the others were real. I don’t know why that one looked like a simulation to me. Sorry for doubting you there.
RJ He (10:05)
Yeah, agreed.
Actually, maybe that’s a good question.
First, what was the main motivation for doing this whole thing? Was it about the human-robot interface, or was it about seeing what kinds of prompts and questions people would ask?
And then the second question is about this simulation-versus-real-world issue. Why did you need to do i…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com