‘Flow Filter’: Introducing an upstream exploratory visualisation and filtering of large and detailed datasets

The scale and diversity of historical digital data that is becoming available, accelerated by Machine Learning methods of preparation and linkage, and the detail with which it is sometimes modelled, has intensified the challenges and opportunities presented by ‘the computational turn’ in humanities research (Berry). The need for bespoke tools and models for graphical argument construction (Drucker) which support humanistic methods of analysis involving nuance, multi-axial zoom (Butterworth) and multi-perspectival possibilities (Viola) have become more acute.

One particularly significant requirement for ‘generous’ tools and interfaces, (Whitelaw), involves a capacity to work across scale: enabling users to delve into the nuance of data within its local context while also zooming out to position it in broader, global contexts—whether singly or comparatively. However, whilst the authors’ design attention is most often given to those bespoke tools for visualising nuance and complexity, such fine-grain insights are likely to be unworkable at the scale of complete datasets, and so need a pre-filter to produce targeted subsets. Having identified this as an essential and generalisable requirement, we subsequently broadened the questions we seek to address. How does someone coming to a richly modelled dataset with some domain knowledge but little familiarity with the schema or properties available for exploration, begin to understand the shape and texture of the data, intuitively and as a productive stage in their research? How can this process be designed as a first stage of increasingly refined exploration, allowing users both to preview possibilities and to reverse and reassess past decisions, while maintaining an overview of the dataset? Without such exploratory tools a researcher may be restricted to well-trodden pathways, driven by prior knowledge and predefined assumptions, with insights into potentially novel connections within a dataset remaining hidden.

The result of these reflections was a set of requirements that re-enforce the need for an exploratory principle of anticipation/preview of comparison/consequence, combined with provisionally and continual iteration for serendipitous discovery, within a large cycle of hypothesis generation and testing. The current range of interfaces available for humanities data is limited and limiting, failing to adequately address these requirements. Existing approaches typically rely on generic search methods, which facet and filter data sets through static menu-based queries and ‘trial-and-error’ workflows.
While this is effective for retrieving specific results, such methods are poorly suited for dynamic exploration, especially by those without prior knowledge of the data. They do not support broader, dynamic exploration of a dataset holistically, in the way necessary to reveal connections, strands and linkages, nor do they allow for interpretative processes (Ciuccarelli & Elli). To meet these challenges, we have adopted a design-based approach of speculative prototyping, or “sandcastle building” (Hindrichs). This method serves as an important mechanism for the construction of argumentation in graphical form (Radzikowska & Rucker) which has informed the design and development of the Flow Filter.

The Flow Filter is a generalisable exploratory data visualisation component that sits upstream within an environment of multiple different downstream components, each with more specialised analytical application. Drawing on principles from the familiar ‘alluvial’ data visualisation form, the Flow Filter functions simultaneously as a visual query builder and a predictive visualisation environment. The interface operates on several levels. It presents a simplified representation of the data structure, for a conceptual overview of the dataset’s organisation. It offers an intuitive interface for serendipitous exploration, encouraging the iterative refinement of research questions. Additionally, its visual rhetorics reveal characteristics and linkages between data attributes in ways that can be shared and communicated across research contexts.

The Flow Filter enables the user to construct "sluice gates" through which data is filtered, with each gate assigned a data attribute, with associated properties that can be selected by the user. This visual logic integrates two aspects of query logic: “OR” logic applies when adding gates, while “AND” logic governs the selection of properties within gates. ‘Flows’ connect these gates, dynamically illustrating relationships and distributions between data attributes. The width of each flow line is proportional to the volume of data passing through it, visually mapping the shape of the data: its structure, densities, and interconnections.

The interface is interactive and responsive to user intuitions and curiosity. The selection of ‘flows’ and ‘bars’ can be modified dynamically, with queries always updatable, enabling researchers not only to view determined search results but also anticipate potential pathways and connections within the dataset. By previewing the breadth or narrowness of downstream ‘flows’, users are guided toward areas of interest for more detailed analysis: whether their interest foregrounds more representative or more anomalous data, or the relationship between the two. As familiarity with the dataset grows, even the non-expert rapidly begins to be able to derive useful insights and formulate questions.

The paper will demonstrate both the use and intrinsic effectiveness of the Flow Filter as a discrete interface, and as the generalisable upstream component that is applied to three datasets which can integrated with their more specific downstream interface components, to enable an expanded scope for iterated exploration.
The three datasets used for the demonstration are: the social and business networks of scientific instrument maker communities in Britain from the mid-eighteenth to mid-nineteenth centuries (around 12,000 individuals); the ‘social clusters’ of residents distributed between 3,500 households in the Saltaire ideal town in the later nineteenth century; and the individuals involved, in varied roles, in the flights from Britain into wartime France in support of resistance networks.

Particular attention will be given to the first of these, for exemplary purposes, around the refinement of broad record sets into meaningful subsets. It will demonstrate how households can be probed and filtered based on attributes such as census year, household size, the ‘identity’ of households (as annotated follow network science analysis), gender, place of birth (at multiple levels of spatial resolution), and occupational types and subtypes (as variously categorised by different classificatory systems). It will show how flowlines can reveal trends like household size distribution over time or the relationship between gender and occupational roles, prior to more sophisticated analysis of selected subsets.

By advancing interactive visualisation methods, Flow Filter contributes to digital humanities research, advancing an intuitive and engaged approach to data exploration which has broad applications in an era of large-scale data and knowledge graph technologies.