An algorithm developed at the University
of Toronto can search large repositories of images to quantify
relationships between individuals based on the locations of tags
found in those images.
The technology -- called relational social image search -- can
help to improve image search and better understand relationships
between people in images without resorting to using computationally
intensive facial- or object-recognition technology.
The patented system has been developed by Parham Aarabi, a professor in electrical and computer
engineering at the University of Toronto, and his former
Master's student Ron Appel. It works by taking the closeness of
people within pictures as a proxy for the closeness of their social
relationships. For example, if you are tagged in Facebook or Flickr
pictures with your mother and are close together, and you are also
tagged in separate pictures with your father, the algorithm can
determine that there is a relationship between those two and assess
how strong that relationship may be. Imagine another set of photos
where you feature with both your parents, but only your mother is
tagged. If you search for your father in the batch of photos, these
untagged images will also be returned because of the high
likelihood that he features in the pictures.
"If you want to search a trillion photos, normally that takes at
least a trillion operations. It's based on the number of photos you
have. Facebook has almost half a trillion photos, but a billion
users -- it's almost a 500 order of magnitude difference. Our
algorithm is simply based on the number of tags, not on the number
of photos, which makes it more efficient to search than standard
approaches," says Aarabi.
Aarabi has been working on this project since 2005. Previous
work has focussed on searching and "understanding" images based on
content recognition -- which is used to find out who or what is in
a photo. This works in some cases, but can be very computationally
"Instead, we decided to focus on a very basic tag information
that is often available but almost always ignored, which is the
location of tags in images. By running the location of tags in
images through our mathematical model, we obtain a relativity graph
which helps us both understand social relationships and also to
search images better," Aarabi told Wired.co.uk.
Aarabi believes that relational social image search could be
used to allow people to search for images on social networks based
on context. "So, for example, searching for myself, our system
would return not only the images where I am tagged in, but also
images where I wasn't tagged, or even images that I am not in but
are related to me (for example, by having several family, or
several friends in them)," says Aarabi.
Another potential application is building relationship maps
based on photos taken of a scene. So if you were to take several
photos inside an office building and were to tag some of the
objects in each image, the system could build a map of the
Aarabi says that the technology has a few limitations. Firstly
it requires images where at least two or three objects are tagged.
Secondly, because the model can find images where users were not
tagged by looking at images where they were tagged, there are
privacy implications. "Would you actually want photos that you
weren't tagged in to be found?" Aarabi asks.
The work will be presented at the IEEE International
Symposium on Multimedia on 10 December 2013.