Platform for Situated Intelligence and OpenSense: A Tutorial on Building Multimodal Interactive Applications for Research

Platform for Situated Intelligence ( \psi ) is an open-source framework intended to support the rapid development and study of multi-modal, integrative-AI applications. The framework provides infrastructure for constructing and executing pipelines of heterogeneous components that operate over temporal streams of data; a set of development tools for visualization, annotation, and debugging; and an open ecosystem of component technologies. A number of applications and higher-level research platforms have been developed on this framework by various researchers in the ICMI community over the past few years, including OpenSense, a platform specifcally targeted for real-time multimodal data acquisition and behavior perception. In this tutorial, we will walk through how to use Platform for Situated Intelligence and OpenSense for multimodal interaction research, starting from the basic concepts, working together towards a sample application, and showing many hands-on examples of the breadth of research applications that can be targeted.