Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices

Designing interfaces or applications that move beyond the bounds of a single device screen enables new ways to engage with digital content. Research addressing the opportunities and challenges of interactions with multiple devices in concert is of continued focus in HCI research. To inform the future research agenda of this field, we contribute an analysis and taxonomy of a corpus of 510 papers in the cross-device computing domain. For both new and experienced researchers in the field we provide: an overview, historic trends and unified terminology of cross-device research; discussion of major and under-explored application areas; mapping of enabling technologies; synthesis of key interaction techniques spanning across multiple devices; and review of common evaluation strategies. We close with a discussion of open issues. Our taxonomy aims to create a unified terminology and common understanding for researchers in order to facilitate and stimulate future cross-device research.

Taxonomy

Taxonomy of cross-device design space dimensions: temporal, configuration, relationship, scale, dynamics and space.

Dimension 1: Temporal. Cross-device work can be classified as either synchronous (when interactions happen at the same time) or asynchronous (with a sequential flow of interactions across devices). The large majority of the work in our survey falls into the former category.

Dimension 2: Configuration. This dimension classifies the actual setup of the cross-device system as well as its use of input and output modalities. The main categories within synchronous use are mirrored and  distributed user interfaces. Most active research is done within the distributed UI category, investigating the spatial and logical distribution of interfaces. The asynchronous work is divided across two categories:  interfaces that allow migration across devices, and cross-platform research to make applications run consistently across diverse operating systems. Related taxonomies align with this dimension, in particular Elmqvist’s taxonomy of distributed UIs and Rashid’s focus on multi-device attention switching.

Dimension 3: Relationship. Research addresses different people-to-device relationships. While one person interacting with a single device (1..1) is usually not part of cross-device work, one person interacting with two or more devices (1..m) covers work on cross-device workstations. Collaborative settings fall mostly in two categories: group activities where each person primarily uses a single device (1..1 x 1..1), and collaborative settings with n-people and m-devices (n..m). Examples of the last two categories relate closely to research and studies done in the CSCW community.

Dimension 4: Scale. Interactions can vary across the dimension of scale: from near, to personal, social, and public rooms or buildings. Edward Hall’s proxemics is a commonly used anthropological lens for the scale of interactions, which was later operationalised for cross-device work as proxemic interactions. Scale dimension relates closely to Terrenghi’s taxonomy of display ecosystems across scale and the progression of Weiser’s Tab/Pad/Yard computing.

Dimension 5: Dynamics. Dynamics can vary between setups, and the categories of ad-hoc/mobile, semi-fixed, and fixed spaces closely relate to the phases of cross-device research we introduced earlier. Fixed spaces often include larger-scale wall displays and tabletops, while semi-fixed spaces allow a certain degree of reconfigurability, and ad-hoc/mobile spaces focus on portable devices, allowing dynamic changes and re-configuration.

Dimension 6: Space. The last dimension differentiates between co-located and remote interactions (and corresponds to Johnson’s CSCW matrix). The large majority of cross-device work covers co-located scenarios, but few examples address the challenges of providing cross-device interactions across remotely distributed locations.

Interaction Techniques

Overview of interaction techniques for cross-device computing.

Ontology

Ontology of cross-device research terminology.

We untangle the diverse terminology of cross-device work and map terms out into a single ontology. The main goal of this synthesis is to provide better guidance about scope and specialisations of research within our field. The unified ontology is formed of two parts of key terms used to refer to cross-device literature.

Top part: key term categories of cross-device sub disciplines. The nested categories begin at the bottom with dual-display and multi-monitor work (a), extend to work with multiple mobile devices, tablets or slates (b), further to the category of cross/multi-display (c) and cross/multi/trans-surfaces (d), and finally to cross/multi-device and distributed covering the broadest scope (e). The nested structure in the figure indicates focus areas associated with each term as well as the often included device form factors. It is important to note that a large subset of the terms are used interchangeably.

Bottom part: a list of terms indicating focus areas of research projects (f): interactions and collaboration; interfaces; applications or systems; platform or middleware; environments or ecologies; and computing. We annotated these terms with the most common research focus of papers using each term.