The Cultural Heritage Project – Improving access to Yale’s collections

October 14, 2021

The Cultural Heritage Project, CUL200 – Improving access to Yale’s collections, is an innovative project supporting the Cultural Heritage pillar. The primary objective of the project is to bring together the collections of Yale’s main collecting organizations (Yale Center for British Art, Yale University Art Gallery, Yale Peabody Museum of Natural History, and the Yale University Library) through a single web interface to allow for search and discovery across 15+ million object records. The service has been named “LUX: Yale Collections Discovery,” and its mission statement is: 

Revealing the cultural heritage collections of Yale University to the world, LUX: Yale Collections Discovery provides a unified gateway to the holdings of Yale’s museums, archives, and libraries. This transformative service enables users to identify, access, and engage with items of interest within Yale’s physical and digital collections. It also uncovers relationships among items, inviting users to explore additional materials across collections.

This is far more difficult than it may sound because differences in metadata and cataloging practices between domains (art, natural history) as well as between museums and libraries make it impossible to put all the data into a typical relational database. Furthermore, a prior proof of concept (POC) which leveraged blacklight Solr demonstrated that while it was possible to harmonize and combine the data into a single index, this type of document/text search could only satisfy the most basic requirements of the new system. For the project to be successful, we needed to evaluate technologies that are largely new to Yale staff, and which have not been used here at enterprise scale.

Following an engagement with a consulting firm, we confirmed that:

  • Our project is unique and possibly the first of its kind within the Cultural Heritage space.
  • There is no de-facto technology stack that is known to be reliable and performant.
  • To select a back-end technology, we were going to need to establish a proof of concept and test many options.

We knew that a successful solution would rely upon document (text) stores and linked data/graph relationships. We also knew that this could be accomplished using one of two architectures:

  • A single system architecture, where there was one multi-model database capable of successfully managing both the document stores and the graph relationships
  • A multiple system architecture, where multiple technologies are utilized for the document store and the graph database and the system we build would be responsible for managing the relationships between the two.

To do this, we treated this effort like its own project. This article will provide a brief outline of the framework and process we created to successfully evaluate multiple technologies, allow the larger project to move forward.

Create the team

We formed a Tech Stack Evaluation Team which was tasked with answering the following questions:

  • Is there a single-solution architecture that is performant and meets the requirements and budget for LUX?
  • Is there a multi-solution architecture that is scalable, performant and feasible to build and support?

The evaluation team was made up of technologist and metadata specialists from ITS and from our Cultural Heritage partners. The project confirmed with each of the team members managers that they supported their staff putting approximately 10 hours a week into this effort for 4-5 months.

The approach

Because of time constraints, we decided that the team would sub-divide into two groups: Single solution architecture and Multi-solution architecture – and that these groups would then further sub-divide and tackle a specific technology. This would allow for the testing to run concurrently. It is also important to note that key stakeholders required the evaluation to include both open-source and vended solutions for each of these categories.

We also needed a dataset for evaluation. We created some initial test records derived from the earlier POC work. This was followed by transformation of the flat data structure into linked data (json) into a minimal dataset that contained fields and entity types.  Finally, a wholesale transformation of the initial dataset in linked data enabled us to do performance evaluation at scale.  
The people working with vended products quickly began to engage with the vendors to explain the project, what we were doing, gain access to the software, and some technical resources. The people working with open-source solutions also began to engage with peer knowledge as well as user groups for support. This helped us to evaluate the ability of a vendor or community to help support the different technologies.

Evaluation timelines

Next, we determined and agreed upon the phases and the reporting milestones. For the phases we kept things high level:

  • Stand up environments
  • Ingest full data set (delete full data set, repeat)
  • Performance testing and analysis

For the reporting milestones, we worked backwards from when we needed a decision which helped us assign a schedule for the dates.

We also set a weekly meeting schedule so each team could report on their progress and discuss obstacles. We correctly assumed that we would have challenges both with the technology as well as with the data. While one team couldn’t necessarily help another with their system, there was ample opportunity for knowledge and work to be shared with regards to the data.

Create benchmarks

The team reviewed the requirements for LUX and established benchmarks for performing the evaluation. The benchmarks included some core requirements as well as performance metrics around data ingest, completing basic and complex queries. If any technology didn’t meet the core requirements, it was dropped from consideration.

It is important to note that the platform evaluation team constrained the testing benchmarks to determine whether certain functionalities were possible. Time and knowledge limitations meant that we only optimized performance to establish functionality, and there was no comparison of specific search/query results between platforms. Also, the evaluation did not include building a user interface, but it did include reviewing the application programming interfaces (APIs) of the different solutions and using the APIs for the purpose of performing load testing. In this way, we were able to get a sense for the ability of the APIs to meet our development needs since LUX was going to require custom user interface development.

Begin the evaluation

Once the team was assembled and the approach was determined, the real work of the evaluation began. The phases, timelines, and benchmarks were established and refined iteratively as the team began the work to stand up environments. Additionally, vendor management was engaged early on to ensure nothing was being done that would adversely affect Yale’s ability to successfully negotiate.

Scoring and consensus decision-making

We created weighted score cards to create a standard and consistent way to evaluate the results of the test which were weighted, based on the relative importance of that test. We worked with our contributing stakeholders from the museums and libraries to establish the priorities.

As this work is driven by the Cultural Heritage IT Pillar, the process required extensive communications and presentations to various stakeholder groups (Technical, Steering, and the full collaboration community). The evaluation process succeeded, in part, because of the diverse viewpoints from across the Cultural Heritage units and the existence of a strong foundational partnership within the evaluation team from the outset.

Conclusion

This approach worked well for the project’s needs. Ultimately, we were able to evaluate six different systems in a fairly short period of time. We dropped two systems shortly after starting based on their inability to meet core requirements and four technologies completed all the testing. The weighted score card then helped us to determine the two finalists. The evaluation also helped us to draft an efficient request for quote (RFQ) that vendor management was able to distribute, and the project was able to select the best product to meet our needs.

One IT at Yale