Beyond Audio: Towards a Design Space of Headphones as a Site for Interaction and Sensing

Via Research through Design (RtD), we explore the potential of headphones as a general-purpose input device for both foreground motion-gestures as well as background sensing of user activity. As a familiar wearable device, headphones offer a compelling site for head-situated interaction and sensing. Using emerging sensing modalities such as inertial motion, capacitive touch sensing, and depth cameras, our implemented prototypes explore sensing and interaction techniques that offer a range of compelling capabilities.

User scenarios include context-aware privacy, gestural audio-visual control, and co-opting natural body language as context to drive animated avatars for “camera-off” scenarios in remote work–or even to co-opt (oft-subconscious) head movements such as dodging attacks in video games to enhance the gameplay experience.

Drawing from the literature and other frameworks, we situate our prototypes and related techniques in a design space across the dual dimensions of (1) type of input (touch, mid-air, or head orientation/motion); and (2) the context of user action (application, body, or environment). In particular, interactions that combine multiple inputs and contexts at the same time offer a rich design space of headphone-situated wearable interactions and sensing techniques.