Elon Musk is kinda worried about AI. (“AI is a fundamental existential risk for human civilization and I don’t think people fully appreciate that,” as he put it in 2017.) So he helped found a research nonprofit, OpenAI, to help cut a path to “safe” artificial general intelligence, as opposed to machines that pop our civilization like a pimple. Yes, Musk’s very public fears may distract from other more real problems in AI. But OpenAI just took a big step toward robots that better integrate into our world by not, well, breaking everything they pick up.
OpenAI researchers have built a system in which a simulated robotic hand learns to manipulate a block through trial and error, then seamlessly transfers that knowledge to a robotic hand in the real world. Incredibly, the system ends up “inventing” characteristic grasps that humans already commonly use to handle objects. Not in a quest to pop us like pimples—to be clear.
The researchers’ trick is a technique called reinforcement learning. In a simulation, a hand, powered by a neural network, is free to experiment with different ways to grasp and fiddle with a block. “It's just doing random things and failing miserably all the time,” says OpenAI engineer Matthias Plappert. “Then what we do is we give it a reward whenever it does something that slightly moves it toward the goal it actually wants to achieve, which is rotating the block.” The idea is to spin the block to show certain sides, each marked with an uppercase letter, without dropping it.
If the system does something random that brings the block slightly closer to the right position, a reward tells the hand to keep doing that sort of thing. Conversely, if it does something dumb, it’s punished, and learns to not do that sort of thing. (Think of it like a score: -20 for something very bad like dropping the object.) “Over time with a lot of experience it gradually becomes more and more versatile at rotating the block in hand,” says Plappert.
The trick with this new system is that the researchers have essentially built many different worlds within the digital world. “So for each simulation we randomize certain aspects,” says Plappert. Maybe the mass of the block is a bit different, for example, or gravity is slightly different. “Maybe it can't move its fingers as quickly as it normally could.” As if it’s living in a simulated multiverse, the robot finds itself practicing in lots of different “realities” that are slightly different from one another.
This prepares it for the leap into the real world. “Because it sees so many of these simulated worlds during its training, what we were able to show here is that the actual physical world is just yet one more randomization from the perspective of the learning system,” says Plappert. If it only trains in a single simulated world, once it transfers to the real world, random variables will confuse the hell out of it.