Researchers at Brown and the Technical University of Berlin have produced a program that can identify simple sketches of objects almost as well as humans. The computer application enables "semantic understanding" of abstract sketches as they are being drawn in real time. The research was presented at computer graphics conference SIGGRAPH and the paper is now available online.
Computers can already match sketches to objects provided they are accurate representations—e.g., matching police sketches to actual faces in mug shots. However, more abstract sketches—the more cartoonish drawings that most people can easily produce—present a different challenge.
For example, in order to draw a rabbit, a person might draw a cartoonish creature with big ears, buckteeth and a fluffy cotton tail that another person would recognise despite the fact that it bears little resemblance to an actual rabbit.
James Hays, assistant professor of computer science at Brown explains: "It might be that we only recognise it as a rabbit because we all grew up that way. Whoever got the ball rolling on caricaturing rabbits like that, that's just how we all draw them now."
Getting a computer to understand this is more challenging. In order to create the program, Hays and his colleagues Mathias Eitz and Marc Alexa from the Technical University in Berlin had to create a large database of sketches to teach a computer how humans sketch objects.
They started by coming up with a list of everyday objects from an existing computer vision (photographic) dataset called LabelMe. They ended up with a set of 250 object categories. They then used Amazon's Mechanical Turk to hire people to sketch objects from each category—20,000 sketches in total. This data was fed into existing machine learning algorithms to teach the program which sketches belong to which categories.
From there, they team created an interface through which they could input new sketches and the computer could try to identify them in real time, as they are being drawn.
The program is capable of identifying sketches correctly with around 56 percent accuracy, as long as the object falls under one of the 250 categories. This is not bad given that humans were able to identify the sketches with 73 percent accuracy.
"The gap between human and computational performance is not so big, not as big certainly as it is in other computer vision problems," Hays said.
Hays and team plan to expand the database to include more categories. One way of doing this is by creating a Pictionary-style game that can collect the data that players input—essentially Draw Something. They have created a similar game which is available on iTunes already. Ultimately the program could be used to develop a better sketch-based interface and search applications.