In this project we investigate novel handheld projector systems for indoor spaces. These projection-based devices are “aware” of their environment in ways not demonstrated previously. They offer both spatial awareness, where the system infers location and orientation of the device in 3D space, and geometry awareness, where the system constructs the 3D structure of the world around it, which can encompass the user as well as other physical objects, such as furniture and walls.

Project Overview

This paper presents two novel handheld projector systems for indoor pervasive computing spaces. These projection-based devices are “aware” of their environment in ways not demonstrated previously. They offer both spatial awareness, where the system infers location and orientation of the device in 3D space, and geometry awareness, where the system constructs the 3D structure of the world around it, which can encompass the user as well as other physical objects, such as furniture and walls. Previous work in this area has predominantly focused on infrastructure-based spatial-aware handheld projection and interaction. Our two prototypes offer greater levels of environment awareness, but achieve this using two opposing approaches; the first infrastructure-based and the other infrastructure-less sensing. We highlight a series of interactions that can be implemented at varying scales with these opposing approaches. These include direct touch interactions, as well as in-air gestures, which leverage the shadow of the user for interaction. We describe the technical challenges in realizing these novel systems; and compare them directly by quantifying their location tracking and input sensing capabilities.

Example Interactions enabled by the system: Spatially and geometry-based shadow interactions.

• A and B) User interacts with virtual physics-enabled objects using a real shadow.
• C) Sensing projector pose, user’s hands (rendered green), and implementing a flashlight metaphor to enable writing and painting using the real shadow.
• D) Painting in midair.
• E) Flashlight metaphor implemented by texturing the 3D model and reveal data through the projector. Note projection is automatically corrected for the textured surface.
• F) Debug output showing how physics interactions are enabled within the space through rods raycast onto the 3D model.