Collections are the tools that people use to make sense of an ever-increasing number of archived web pages. As collections themselves grow, we need tools to make sense of them. Tools that work on the general web, like search engines, are not a good fit for these collections because search engines do not currently represent multiple document versions well. Web archive collections themselves are vast, some containing hundreds of thousands of documents. There are also thousands of collections, many of which cover the same topic. Few collections include standardized metadata. Too many documents from too many collections with not enough metadata makes collection understanding an expensive proposition.
This dissertation establishes a five-process model to assist with web archive collection understanding. This model aims to automatically produce a social media story -- a visualization paradigm with which most web users are already familiar. Each social media story contains surrogates which are summaries of individual documents. These surrogates, when collected together, summarize the overall topic of the story. After applying our storytelling model, they summarize the topic of a web archive collection.
We develop and test a framework to select the best exemplars that represent a collection. We establish that algorithms produced from these primitives select exemplars that are otherwise undiscoverable using conventional search engine methods. We generate story metadata to improve the information scent of a story so users can understand it better. After an analysis showing that existing platforms perform poorly for web archives and a user study establishing the best surrogate type, we generate document metadata for the exemplars with machine learning. We then visualize the story and document metadata together and distribute it to satisfy the information needs of multiple personas who benefit from our model.
Our tools serve as a reference implementation of our Dark and Stormy Archives storytelling model. Hypercane selects exemplars and generates story metadata. MementoEmbed generates document metadata. Raintale visualizes and distributes the story based on the story metadata and the document metadata of these exemplars. By providing understanding at a glance, our stories save users the time and effort of reading thousands of documents and, most importantly, help them understand web archive collections.
