Building featured HathiTrust collections with web automation

This post details the automated addition of hundreds of new items to a (public) HathiTrust collection. For use in my own research projects, I have set up the public collection “INSULAE (Europe)” of predominantly Western European texts on islands published between 1600 and 1800. As I wanted to add more than 3000 items to the collection, web automation with Python was the most efficient solution. Manually adding selected results takes a lot of time.

I therefore wrote a Python script using the Selenium package in order to remote-control my Google Chrome browser.

The full script is available in my Digital History repository on Github.

The major advantage of using Chrome is that I have a customised profile in which my Google password and certain browser-extensions are stored. Accessing a website via this profile permits me immediate access to content and services for which I needed to register in the past. Accordingly, web automation in Google Chrome also allows me to easily log in to my HathiTrust guest account and expand my collections.

In my Python script, this action is coded as follows:

# Opening CHROME with custom profile

options=webdriver.ChromeOptions()
options.add_argument("--user-data-dir=C:\\Users\\mbarg\\AppData\\Local\\Google\\Chrome\\User Data\\")
options.add_argument('--profile-directory=Profile 1')
driver = webdriver.Chrome(executable_path='C:\\chromedriver_win32\\chromedriver.exe', options=options)
driver.implicitly_wait(2)
driver.maximize_window()

print("Browser opened.")

# Navigate to the application home page and log in

driver.get("https://babel.hathitrust.org/Shibboleth.sso/Login?entityID=https://google.cirrusidentity.com/gateway&target=https://babel.hathitrust.org") #URL of guest log-in via GOOGLE
print("You have successfully logged in!")

The next step is to iterate through hundreds of result pages, select all results per page, and add them to the collection of my choice. To move from one result page to the next, I have used the neatly structured HathiTrust result page URLs that contain all relevant information. For instance, each results page URL ends with pn=n, with “n” being the current page number.

If “pn=1” denotes the first result page, “pn=2” indicates that the second result page has been selected. The number of results per page is also coded in the URLs and best set to 100 (sz=100). Accordingly, new result pages are opened by letting the script iterate through the full range of possible page numbers.

The total range of result pages I want to call is 105 in my sample script. And for each digit in this range I am performing the same sequence of actions:

# Iterate through result pages

QueryURL="https://babel.hathitrust.org/cgi/ls?a=srchls;q1=insula%20insulae%20insulis%20insul%20insuln%20island%20islands%20isle%20isles%20%C3%AEle%20%C3%AEles%20isola%20isole;field1=ocronly;anyall1=any;q2=insula%20insulae%20insulis%20insul%20insuln%20island%20islands%20isle%20isles%20%C3%AEle%20%C3%AEles%20isola%20isole%20geographie%20geographia%20geography;field2=title;anyall2=any;op2=AND;q3=insula%20insulae%20insulis%20insul%20insuln%20island%20islands%20isle%20isles%20%C3%AEle%20%C3%AEles%20isola%20isole;field3=ocronly;anyall3=any;q4=insula%20insulae%20insulis%20insul%20insuln%20island%20islands%20isle%20isles%20%C3%AEle%20%C3%AEles%20isola%20isole%20geographie%20geographia%20geography;field4=subject;anyall4=any;op4=AND;op3=OR;yop=between;lmt=ft;pdate_start=1600;pdate_end=1800;sz=100;pn="
pages=range(1, 105)

for p in pages:
    driver.refresh()
    target=[QueryURL, str(p)]
    s=""
    targetURL="".join(target)
    print(targetURL)  
    driver.get(targetURL)# navigate to target URL
    driver.find_element_by_xpath("//button[@id='action-select-all']").click() # select all items on page
    dropdown=Select(driver.find_element_by_id("collection-chooser")) # navigate to dropdown
    dropdown.select_by_visible_text("INSULAE (Europe)") # choose collection from list of options
    driver.find_element_by_id("addits").click() # add selected results to collection
    print("Page no.", p, "has been added!")

The comments (starting with #) detail all the individual steps and also explain the function of the HTML elements I have called. Apart from selecting all results per page, I also needed to navigate to the “collections” drop-down-menu, select the right collection from the list and press the “add” button.

In order to identify the necessary HTML elements, I checked the source code with the Chrome developer tools.

As source codes are usually long and complex, developer tools in web browsers help us link the visuals of the website with the relevant sections in the underlying HTML. You may directly copy the HTML elements highlighted into your script.

When you are working with Selenium “Select” for the first time, it can be challenging to reference HTML elements correctly as only a limited number of “Select” attributes are allowed. This is why I highly recommend reading the official Selenium “Working with select elements” documentation which lists the proper syntax for different programming languages.

Last but not least, web automation isn’t magic and requires some patience. Each website takes time to respond and script performance also depends on your own equipment and internet connection. To give you some idea at what speed my script was run on my private desktop PC, here is a real-time screen capture:

If you are interested in more web automation and web scraping scripts, you may want to visit my “Digital History” repository on GITHUB. This repo includes a script to delete multiple items from HathiTrust collections. Moreover, I have used Selenium in Python 3 to import bibliographic information from WORLDCAT to ZOTERO and to collect biographic information from the German National Library data service. These scripts could be adapted to perform a variety of similar tasks in the humanities.

 

---

---