Image recognition is something humans do without a second thought—a quick glance is typically enough for us to identify most items in an image. But computers have a much harder time of this task and frequently get the basics wrong. Researchers have long recognized that providing computers with the ability to recognize context would help them avoid typical errors, such as deciding that a person was riding a dog, rather than a horse. A research paper (PDF) presented yesterday at the IEEE International Conference on Computer Vision reports great success with providing semantic context, and notes a convenient source of it: the Google Sets beta, which generates lists of associated terms derived from Google's webcrawling.
Does it make semantic sense to backhand a lemon?
The image at left shows the process at work. The first stage of the image recognition pipeline is to segment the image into its constituent parts, in this case the player, racket, and ball. A best fit for segmentation is accepted when minor, random perturbations to the image don't change the segmentation scores. Next up, sets of potential identifications for the individual features were generated using a standard method called "Bag of Features." Each of the members of these identity sets are also given a confidence score.
This is where semantic context becomes critical. In this image, the player and tennis racket were correctly identified with high confidence scores, but so was the tennis ball—the computer was confident it was a lemon. It's rare for a lemon to be connected with a tennis racket and a human form, however, and semantic context could help the computer look further down its set of possible identities to find the item that fit well with the other items in the image, recognizing that a ball had a better semantic fit. The paper describes how context enabled similar corrections to be made, such as a cow floating in a harbor being corrected to a ship and a person sitting on a vehicle having the vehicle changed from a car to a motorbike.
The researchers also compared two sources of semantic context. In one case, they generated an association matrix by training the computer on a subset of the image collection used for the tests. In the second, they tried several online sources of word associations before settling on Google Sets. In both cases, the training outperformed Google Sets, but at a cost of requiring human intervention. The data from Google Sets also wasn't much of a help on an image set where the computer could already correctly identify over 60 percent of the image components. In the case of a second image set where the computer started with a low success rate (45 percent), however, Google Sets made a large difference, bringing the success rate up to nearly 60 percent.
Given this success, the authors are planning on continuing their search for other online sources of object associations; they specifically mention looking into Amazon.com's groupings. They also hope to leverage the location information generated during their analysis, operating on the assumption to physical proximity should reinforce semantic context. This sort of system won't be good for handling all images—there are photos of swimming cows out there—but it could be a large improvement over what's currently available.