Art

A digital renaissance is helping global audiences connect with art

The Metropolitan Museum of Art houses over 1.5 million works of art spanning 5,000 years. Now, The Met is exploring artificial intelligence to make its collection accessible to the 3.9 billion internet-connected people worldwide.

The Metropolitan Museum of Art launched their Open Access platform in 2017, making all images and data relating to public-domain artworks in its vast collection available to everyone online. The Met’s Open Access Program seeks to make the museum’s collection one of the most accessible, discoverable, and useful on the internet. Now, The Met is collaborating with Microsoft to take this initiative to the next level, utilizing artificial intelligence to help audiences better discover these artworks and develop meaningful connections with them.

As one of the world’s largest art museums, The Met houses over 1.5 million works of art with over 7 million physical visitors each year. In order to expand this reach to the internet-connected audience of 3.9 billion people worldwide, the museum must take on the herculean task of digitizing, classifying and tagging each work of art in a scalable way.

The classifying and tagging process is complex and labor intensive. What makes this process so complex? Art is inherently highly unstructured. Unlike with a document, there’s no easy way to lift information. Instead, the visual elements need to be perceived, from the media and technique to the choice of color palette and subject. There’s also a vivid story behind each piece: the artist’s history, influences, and style. In the past, gathering all this information had to be done manually for each piece—a painstaking process, particularly when it involves millions of works of art.

“For our Open Access collection, we hired a team of people to examine each image and tag what they saw,” says Jennie Choi, General Manager of Collection Information at The Met. “However, this is very labor-intensive and requires careful inspection of each piece.”

a man gestures to a group in front of a large Renaissance artwork

That’s why The Met is now exploring artificial intelligence to simplify and scale this process. In particular, the museum has worked on a proof of concept called, Art Explorer, which uses Microsoft’s Cognitive Search to examine each artwork and automatically generate all the information needed to tag and classify the piece in a fraction of the time. (The Art Explorer project was able to process hundreds of thousands of artworks this way in a single night.)

But there’s much more to the process than speed. When a digital image is loaded into Art Explorer, Cognitive Search also surfaces what objects are depicted in the piece, what other artworks in the collection are visually similar, and what relevant information (geography, artist’s history, etc.) it should pull in from the web.

This data is then organized into a searchable index that unlocks insights, uncovers relationships between pieces in the collection, and grows the knowledge base around each piece online. As people navigate the Open Access collection, they can search for oil paintings or line drawings or all works of art depicting certain animals. In other words, they can find what interests or inspires them in ways that were never possible before.

We’re in the midst of a renaissance right now, with AI bringing art and science together in a way that will enable online visitors to develop a deeper and more personally relevant connection to art.

Maria Kessler, Senior Program Manager for Digital Partnerships, The Met

Art Explorer is just one of the many ways The Met is experimenting with artificial intelligence. Last December, The Met, Microsoft, and Massachusetts Institute of Technology (MIT) collaborated in a two-day hackathon session to explore new ways for global audiences to discover, learn, and create with one of the world’s foremost art collections through artificial intelligence.

Hosted in Microsoft’s Garage space at the New England Research & Development (NERD) center, teams included participants across all three organizations – curatorial and digital staff from The Met, faculty and students from MIT Open Learning and the Knowledge Futures Group, as well as researchers and engineers from Microsoft.  Over the course of 48-hours these participants broke out into small groups to design concepts and initial prototypes which highlight the potential of artificial intelligence and open data to empower people globally through art.

Learn more about the hackathon and exciting prototypes created to help new audiences discover The Met’s Open collection.

a side by side comparison of a 'grammed sushi dinner and a painting of sushi from The Met's collection
adults converse at a table with a large renaissance painting behind them
The author presenting Tag, That's It! at the reveal event in the Great Hall of the Met Fifth Avenue on February 4, 2019.
a screenshot of a Met museum web app, Art of the Day
My Life, My Met

My Life, My Met turns an Instagram feed into a work of art. Still in development, the concept is to have AI analyze Instagram posts and substitute the closest matching artworks from The Met's Open Access collection.

Gen Studio uses AI to navigate the 5,000+ years of human creation represented in The Met collection.

Gen Studio uses AI to navigate the 5,000+ years of human creation represented in The Met collection. Viewers are placed in map-like model that organizes the artwork in exciting new ways, pointing them towards artwork based on style, materials, subject, or other characteristics.

Tag, That’s It!

This crowdsourcing game enlists the global Wiki community, partnering humans with machines to fine-tune the keywords tags for a work of art. It requires identifying which pieces contain various objects: children, chess, calligraphy, and of course, cats.

Storyteller

Storyteller uses AI to find pieces of art in The Met's collection to accompany any story. It uses voice recognition to follow a discussion and share artworks related to the stories being told. Afterwards, users can share the Storyteller thread of artworks on social media.

Artwork of the Day

This app helps individuals find a specific piece of art in the collection that will resonate with them on any given day. Each day, analyzing data like location or news, the app offers an entry point into the collection based on world events and the user's current circumstances.

“Through AI, we can see things that we couldn’t see before with the naked eye,” says Maria Kessler, Senior Program Manager for Digital Partnerships at The Met. “There are patterns—there’s information that’s discoverable in the art, behind the art, and connecting the art from one piece to another.”

And this has big implications for museums overall. Instead of observing static works of art one-by-one, viewers will be able to dynamically curate their experience based on their unique interests. Understanding the innovative ways to harness AI, The Met is poised to create meaningful new ways people connect to art.