Museums and universities from around the world have contacted Yale since LUX: Yale Collections Discovery debuted on June 1. They all want to know how the university built such a groundbreaking platform, one that can search collections as large and diverse as Yale’s from a single place and uncover sometimes-surprising connections between the items in the collections. They also ask how they can get such a system themselves.
Yale’s answer is: It’s yours for the taking.
“A core principle behind the LUX project was that, as much as possible, the technology had to be shareable with other institutions for use with their own collections,” said Jeffrey Campbell, associate director of digital asset technologies, who leads the IT team that supports Yale’s libraries and museums and that helped to create LUX. “Here at Yale, we are fortunate to have the resources available to do big, meaningful projects like LUX, and it is important to be able to contribute back to the larger cultural heritage community.”
The software Yale developed for LUX is intended to be open-sourced, meaning other organizations will be able to freely adopt and adapt it. While the backend database is proprietary, everything else is built with standards and reusability in mind. By making Yale’s approach, data format, and code available, it increases the likelihood of other organizations replicating it and creating datasets that are interoperable.
This is exciting because, “eventually, the data from these other organizations can be incorporated,” said Campbell, “which will surface more relationships and allow users to find connections to items beyond Yale’s boundaries. In turn this will more widely expose Yale’s and other organizations’ collections to researchers around the world.”
Connecting the Threads
Yale’s museum and library collections include over 17 million items dating from prehistoric times to the modern day and spanning the range of human studies. “Yale has one of the broadest, most diverse, and most extensive collections of any institution of higher education, from our natural history holdings to our art and archives, sculptures and paintings, and library collections,” said Campbell, “and there are threads that connect them. Our aim in building LUX was to expose the connective tissue between everything.”
Each of the university’s collections is unique, with its own system for organizing and cataloging its holdings — so creating a consistent format for the LUX search engine was a challenge, said Sarah Prown, project manager for the Cultural Heritage IT Collaboration, which includes the LUX project. There are significant differences in how collections are described. “You can’t just say, ‘Well, a book is described like this, and a painting is described like this, and they naturally overlap,’ because that’s not the case,” Prown said.
The work for this endeavor began years before LUX. The data pipeline was developed by Robert Sanderson, senior director for digital cultural heritage, and Kelly Davis, cultural heritage data engineer, and it transforms records from Yale’s museums, libraries, and archives as well as a wide variety of external sources into a common, standard format called Linked Art. The standard was developed for art museums specifically. However, it is built upon a solid conceptual base, and the Yale team determined that it was both feasible and useful to map library and archival data into it as well. This helped to solve the cross-domain challenge of aligning books, with fossils, with archival collections, and with paintings and other artwork.
Collaboration Leads to Innovation
LUX is the culmination of a five-year collaboration. The project was in its second year when the COVID-19 pandemic hit, forcing the group to meet via Zoom, which had an unexpected benefit: Meetings became “exponentially more productive and more inclusive,” said Prown. “Before, staff came from all over campus, and it was often hard to get to in-person meetings. Shifting to an all-Zoom format was really galvanizing in terms of participation and availability.”
“Also, [during Zoom meetings], everyone appears in the same little box; no one person has more presence in a room,” she said. “It was very equalizing. It gave everyone an opportunity to be heard. We also had a lot of side conversations in the Chat. There was a lot more going on in a one-hour Zoom meeting than we ever had in an in-person meeting.”
During the pandemic, the LUX project reduced its budget, but also extended the duration by a year. “We didn’t know it then, but that gave us the time to be a little bit more thoughtful about what was going to be done, and a bit more intentional about some of the technologies that were being examined,” said Campbell.
Also critical to LUX’s success was the cross-collection collaboration among Yale’s libraries and museums during the project. “There was just so much palpable excitement from across units, with people wanting to work with their colleagues, getting to know each other,” said Prown. “And this is not going to stop. We were able to build a sustainable collaboration. People are continuing to talk to each other and working across collections.”
In fact, Prown said, when institutions call to ask how Yale created LUX, “as much as the technology, they want to hear about the organizational structure at Yale that enabled this to happen — because apparently what we have here is pretty special in terms of openness and collaborative spirit.”
Through the Rabbit Holes
Although LUX has now launched, work on the search system continues. Yale’s collections represent hundreds of years of cataloging, and there are inevitably inconsistencies and errors. “There’s a lot of clean-up,” said Prown, “and there will forever be a lot of clean-up” as new knowledge about the collections surface.
A view of LUX: Yale Collections Discover at work.
The LUX partners are eager to get the word out about LUX, and they especially want to ensure faculty “get the tool in front of their students, so they know that it’s there and it’s useful,” said Campbell. Similarly, library and museum staff are being trained in how to use the search tool.
Scholars, researchers, and the simply curious can go to LUX: Yale Collections Discovery and type in any word or phrase (say, “trout”). LUX generates lists of results for Objects, Works, People & Groups, Places, Concepts, and Events that include the search terms. Each tab offers different ways to facet or refine your results. By choosing options in the left-hand column, users can refine their search results by a dozen different criteria, including the type of object (Yale’s trout-related holdings include books, specimens, music, prints, fossils, official documents, and more) and the Yale collections they are in (items about trout are in nearly 30 collections across campus, from Yale Divinity School to Bass Library).
Users can also search for only those items that can be accessed online or those with a digital image. In addition, there is an advanced search function and multiple ways to sort the list of links. The data on Yale’s holdings is enriched by relevant information drawn from non-Yale institutions and databases, such as the Library of Congress, with links to those sources.
“We’ve tried to build LUX so that there are no dead ends,” said Campbell. “One rabbit hole just leads to more rabbit holes, which lead to more rabbit holes.”
17 million reasons to love #YaleLUX