A Machine Vision Algorithm Learns to Attribute Paintings to Specific Artists

In May 2015 Babak Saleh and Ahmed Elgammal of the Department of Compuer Science, Rutgers University, described an algorithm that could recognize the Style, Genre, and Artist of a painting.

"Saleh and Elgammal begin with a database of images of more than 80,000 paintings by more than a 1,000 artists spanning 15 centuries. These paintings cover 27 different styles, each with more than 1,500 examples. The researchers also classify the works by genre, such as interior, cityscape, landscape, and so on.

"They then take a subset of the images and use them to train various kinds of state-of-the-art machine-learning algorithms to pick out certain features. These include general, low-level features such as the overall color, as well as more advanced features that describe the objects in the image, such as a horse and a cross. The end result is a vector-like description of each painting that contains 400 different dimensions.

"The researchers then test the algorithm on a set of paintings it has not yet seen. And the results are impressive. Their new approach can accurately identify the artist in over 60 percent of the paintings it sees and identify the style in 45 percent of them.

"But crucially, the machine-learning approach provides an insight into the nature of fine art that is otherwise hard even for humans to develop. This comes from analyzing the paintings that the algorithm finds difficult to classify.

"For example, Saleh and Elgammal say their new approach finds it hard to distinguish between works painted by Camille Pissarro and Claude Monet. But a little research on these artists quickly reveals both were active in France in the late 19th and early 20th centuries and that both attended the Académie Suisse in Paris. An expert might also know that Pissarro and Monet were good friends and shared many experiences that informed their art. So the fact that their work is similar is no surprise.

"As another example, the new approach confuses works by Claude Monet and the American impressionist Childe Hassam, who, it turns out, was strongly influenced by the French impressionists and Monet in particular.  These are links that might take a human some time to discover" (MIT Technology Review May 11, 2015).

Saleh, Babak, and Elgammal, Ahmed," Large-scale Classification of Fine-Art Paintings; Learning the Right Metric on the Right Feature" (http://arxiv.org/pdf/1505.00855v1.pdf, 5 May 2015.