Cody Fitzgerald, China Mauck, Rebecca Hardenbrook, graduate students in the Department of Math, College of Science, and Kiki Karahalios, graduate student in the Department of Art & Art History in the College of Fine Arts
The Fellowship in Collections Engagement in part of “Landscape, Land Art and the American West,” a joint initiative between the Utah Museum of Fine Arts and the J. Willard Marriott Library sponsored by the Andrew W. Mellon Foundation. This award fosters innovative, collections-based scholarship and creativity while highlighting the strengths of the UMFA’s and Marriott’s collections.
Our project, Learning Art: What Makes Art Western?, focuses on using machine learning techniques for automatic art classification. We are specifically interested in exploring the art and photography collections of the Utah Museum of Fine Arts and J. Willard Marriott Library to see if we can decipher what it is that makes an image evoke the feeling of the American West. Naively, one might think it’s the content of a work, such as cacti or horses that gives the viewer the sense of the American West. While this is true, select abstract paintings and patterns can also evoke the same feeling without such explicit imagery. Although detecting unifying themes of American westernness is the ultimate goal of this project, we began by first using machine learning techniques for simpler, more tangible tasks, including improving the accuracy of an existing algorithm for art classification.
Machine learning is the process of getting a machine to “learn” something, without explicitly writing a program that tells the machine what to look for. Consider the art genre classification problem. If you start with a list of artistic genres and a bank of thousands of images contained within each genre, it is possible to “teach” the machine about important features of images contained within each genre. If the machine learns well enough, you can expect it to be somewhat successful in classifying new images or paintings that it hasn’t seen before, assuming they fit into one of your chosen genres.
The system that actually does this learning is called a neural network (inspired by biological neural networks, which help us learn). A neural network is made up of several layers. At the first layer, the pixels of the original image are fed into the network. Each subsequent layer can be thought of as a specialized function that operates on the output of the previous layer. As more layers are added, the network is called a deep neural network (DNN). During the learning or training process, the parameters for these functions are iteratively updated at each step to improve the classification accuracy. A simple diagram of an example neural network and an example DNN are shown below. (Hasan, Hasan, Hoassain, Al Mamun, & Assaduzzaman, 2019).
Machine learning techniques have been applied to image classification problems for at least twenty years, but applications to art genre classification are more recent (LeCun, Cortes & Bruges, 2012). During our initial research into a direction to take this project, we discovered RASTA (Recognizing Art Style Automatically) in painting with deep learning, an algorithm for art genre classification developed in 2017 and trained on the Wikipaintings dataset[1] (a large collection of paintings from WikiArt) (Lecoutre, Negrevergne, & Yger, 2017). With our faculty advisor, Braxton Osting, and his team of undergraduates involved in the SRI (Science Research Initiative) program, we have worked through machine learning tutorials and have begun modifying code to improve upon the accuracy achieved by RASTA.
Art classification can also tell us how an individual artist’s style changes over the course of their career. Using an autoencoder, a machine learning technique for dimensionality reduction related to Principal Component Analysis[2], we can isolate defining features of major art genres. Using an additional data set of paintings that spans an artist’s career, we can classify the style of each painting and ask questions about the results. How does the artist’s style diverge over time? Does the artist ever return to a genre? Do life events, travel, influence, and/or interaction with other artists meaningfully change the artist’s original style? If so, we may be able to detect it.
These types of analyses do not completely answer the question we originally set out to answer, but they do get us closer.
Select images from the collections:
REFERENCES
- Hasan, M.Z., Hasan, K.M.Z., Hoassain, S., Al Mamun, A., and Assaduzzaman, M. (2019) Handwritten Changma Numerals Recognition Using Capsule Networks, 2019 5th International Conference on Advances in Electrical Engineering (ICAEE), 386-391. Retrieved from https://ieeexplore.ieee.org/document/8975468
- Lecoutre, A., Negrevergne, B., and Yger, F. (2017) Recognizing Art Style Automatically in Painting with Deep Learning, Proceedings of the Ninth Asian Conference on Machine Learning, in PMLR 77, 327-342. Retrieved from http://proceedings.mlr.press/v77/lecoutre17a/lecoutre17a.pdf
- LeCun, Y., Cortes, C., and Burges, C.J.C. (2021) The MNIST Database of handwritten digits. Retrieved from http://yann.lecun.com/exdb/mnist/
[1] www.wikiart.org
[2] Principal Component Analysis (PCA) works by reducing the dimension of, or rather the number of variables within, a dataset to increase one’s ability to interpret the data while simultaneously minimizing how much information you lose from the data. Find more information on PCA here.