De Collectione Datorum – the challenge of developing data models and databases in humanities projects

The following blog post was initially written for Mosa Historia, the blog run by the Department of History at the University of Maastricht:

A challenge that humanities scholars from different disciplines face in their research, and that is worthy of a baroque Latin title, is structured data collection. Ideally, the data should be machine-readable as well as human-readable, they should be easy to update, and they should make different levels of reliability transparent. Lacking custom-made research infrastructures, most humanities scholars start collecting data in the form of spreadsheets but would prefer a more advanced database with convenient input and query options. Among researchers who are not tech-savvy, databases are sometimes even viewed as the Holy Grail – systems with miraculous powers that will help them solve all their data collection and analysis problems at once. Building and maintaining databases, however, is complicated and often cost-intensive, and providing a user-friendly interface is not always possible. So how can (chronically under-funded) humanities project teams manage their research data and conduct meaningful data analysis?

Photo by Jan Antonin Kolar on Unsplash

In the DigiKAR geohumanities project I have co-coordinated since the summer of 2021, we are trying to develop exemplary workflows for working with historical spatial data, including a de-centralised and low-maintenance collection of data by different team members. The DigiKAR project (“Digitale Kartenwerkstatt Altes Reich”) is led by the Institute of European History in Mainz, Germany, and questions traditional cartographic visualisations of the Holy Roman Empire. Countering perceptions of the Empire as a “patchwork quilt” (“Flickenteppich”) of multiple small territories, we highlight overlapping structures and trans-regional mobilities. For this purpose, we have chosen two regions of the early modern period as case studies: the Electorate of Mainz (covered by historians at JGU Mainz and EHESS Paris) and the Electorate of Saxony (covered by Falk Bretschneider, EHESS Paris).

The historians in the DigiKAR team are early modernists who often work with incomplete, uncertain or fragmented data that come in various formats and from multiple sources. Apart from original and barely digitised archival material, we gather semi-structured data assembled by researchers of the earlier twentieth century and XML-encoded data available via API. Most historians in DigiKAR are regular users of digital tools with graphical user interfaces (GUI) but have limited coding and database experience.

Regarding content, the historians in DigiKAR are interested in the construction of space through legal and political interactions and in biographic mobility. Regarding data queries, the historians’ requirements range from Boolean queries to complex graph-related queries.

Both requirements make a high level of data cleaning and normalisation or careful data mapping necessary, which often collides with the historians’ wishes to preserve the complexity and ambiguity of their sources. Inter alia, DigiKAR aims to balance the historiographical and technical requirements by providing iterative and comparative data analysis. The (geo)visualisations created in the project contribute to this critical analysis and the communication of project results.

As a starting point, we have developed several spreadsheet templates for our two thematic work packages, respecting their different foci. In the Mainz work package, which traces clerical and secular officials’ biographies, we collect event-oriented data using the factoid approach developed at King’s College London. This means that we gather agency-related events (e.g. “grand tour”) and general life events (e.g. “birth” and “death”) as stated in different sources. Uncertainty or vagueness of the information is captured in a comments column, where we also add important source quotations.

We use Python scripts to compare and align the spreadsheet data (cf. our GitHub deployment under construction) to abstract from the redundant, sometimes contradictory factoid entries and construct more solid chronologies of events. Our ultimate goal is to write a higher-level knowledge representation to RDF format for data queries, eventually leading to more reliable visualisations of people’s mobility and Mainz’s administrative structures.

In the Saxony work package, we have decided to approach spaces through their legal capacities, including religious rights, peace-keeping, taxation, or political representation. In this approach, a “mill” is a “mill” because certain legally defined functions within a local community are linked with the place and the people who live and work there. Like the work package Mainz, these data will be collected in spreadsheets. Still, as the Saxony case study focuses on place attributes rather than people’s networks, we intend to import those data into a relational GIS database for exploration and data enrichment early on. Our two work packages thus experiment with slightly different workflows, but both are marked by a flexible combination of (versioned) spreadsheets, script-supported data cleaning, and more solid data infrastructures.

One example of a Python script I have written for data review in the Mainz work package is the entity counter. This script iterates through a shared directory with different EXCEL files in our factoid format and counts unique entities it finds in the data columns. The results are then written to new worksheets in a new EXCEL workbook to help us find and resolve inconsistencies in using person names, place names, institution names, functions and titles. In some cases, variations are mere typos — in other cases, they may be Latin variants or other alternative descriptions. Keeping track of and mapping variants is vital for creating consistent RDF data and writing reliable data queries.

A more interpretative script we use to consolidate genealogical information in the Mainz data is my Relationship Tracer which reads family relations from the rel_pers column in our factoid lists and makes them explicit. It also reconstructs sibling relations based on known parent-child connections. The reconstruction of implicit grandparent-grandchild relations is more complicated, however, and still needs review in collaboration with historian Florian Stabel, who is responsible for the Mainz data collection.

While importing large amounts of data into the graph database and the relational GIS database is a medium-term goal that requires a lot of preparation, we are already experimenting with smaller sub-sets of our data, especially in collaboration with students. In the summer term of 2022, Bettina Braun (JGU Mainz) taught an MA seminar in history which focused on fourteen early modern cathedral provosts from Mainz. Their professional careers and mobility patterns already gave us important insights into the religious and political interconnected of Mainz in the 17th and 18th centuries. Results from the seminar (static maps and a zoomable map of individual biographies) will successively be published on our DigiKAR Projektseminar website.

Researchers interested in similar questions are welcome to contact us. We aim to organise community and networking events to exchange ideas and discuss potential collaborations.