Beneath the Surface

Microsoft Research Connections funded projects at five universities to explore potential applications of Microsoft Surface technology in education. Topics included input methods for mathematical and scientific diagramming applications, approaches for managing research and teaching materials and facilitating collaborative laboratory work, hybrid applications with other technologies, control systems for mixing music, and gestural vocabularies.

MetaSurfacing with the Microsoft Surface: Manipulating Harvested Project Maps | Jeremy J. Baumberg, University of Cambridge, United Kingdom

By harvesting real data from current electronic lab-books in Microsoft OneNote across the Cambridge NanoPhotonics Centre (NP), our aim was to investigate several approaches to management of research and teaching material facilitating collaborative work. Using multi touch, 2-D and 3-D gestures, and integration with Tablet PCs, we aimed to design and evaluate methods for bringing to the surface (“surfacing”) deeply embedded information to enable “iconic capture” and overviews of research projects’ structure and themes. We anticipated that diverse 2-D and 3-D representations of content allow for rapid visual negotiation through complex multi-component projects. Combined with natural 2-D and 3-D gestures, these concepts are aimed to give rise to novel working practices and support effective management, adoption, and ownership of evolving creative research processes. Our conclusion is that Microsoft Surface and other multi-touch systems can play a strong role within information work environments as a way to explore complex, rich, and evolving data spaces. Together with tablet PCs—which are ideal pen-based input and large vertical displays—multi-touch works well for exploration, mapping, and linking different content and views.

Hands-On-Math | Andries van Dam, Brown University, United States

Having developed pen-based interfaces for more than a decade, including pen-centric systems for mathematics, we believed that the pen was overloaded for many tasks that could naturally be off-loaded onto touch interfaces. Our aim with this project was to explore three areas:

• Technical solutions for sensing pen input on a Microsoft Surface
• The design of a hardware abstraction layer API to unify pen and multi-touch input across multiple platforms
• Multi-touch techniques for exposing additional CAS (Computer Algebra System) functionality that complement our existing pen-centric math sketching UI.

Our expectation was that such a hybrid system would provide a non-trivial example of the potential synergistic benefits of pen and multi-touch computing. We presented a prototype system, Hands-On-Math, which reduces the barriers to accessing computational assistance during math problem solving by unifying CAS functionality with a virtual paper UI. This system contributes novel bi-manual and gestural techniques for managing and writing on virtual note pages in addition to direct manipulation techniques for algebraically transforming mathematical expressions. Pilot studies indicate that, after refinement, a mature version of Hands-On-Math would be a desirable tool for scientific and academic note-taking and ideation.

Garibaldi | Andries van Dam, Brown University, United States

Our goal was to provide a research and visualization tool for a specific large piece of artwork—the Garibaldi Panorama. The work so far has been positively received—the application was on show at the British Library as a centerpiece of their “Growing Knowledge – The Evolution of Research” exhibit that opened in mid-October, 2010. This project also led to the new research project also funded by Microsoft Research, code-named HumBub, which is shorthand for Bubbles for the Humanities. Bubbles are a light-weight, minimal-chrome, window-like container for fragments of code (the Code Bubbles system is also partially sponsored by Microsoft Research), as well as other text and multi-media content.

Creative Production Environments | Mark Bolas, University of Southern California, United States

We focused on two areas that are typically challenging in recording studios: Collaborative Mixing and Simultaneous Multi-Axis Control. Our design process was iterative: we tested and changed our approach based on input from identified partners in a professional mixing environment. This resulted in the discovery and fine-tuning of radically new gestures and interface models that are new both to Microsoft Surface and to recording studios. Windows 7 Multitouch will likely be embraced by vendors of existing digital audio workstations—when this happens, we would like to be ready with new interface ideas that put Multitouch ahead of the pack and brand Surface as the ultimate interface. Additionally, it is hoped that such work is generally useful for signal processing and non-linear editing in other domains.

Putting the Surface in Context | Steven Feiner, Columbia University, United States

Columbia’s Computer Graphics and User Interfaces Lab performed research that addressed two main ways in which the Microsoft Surface could be used in context with other computing devices and systems. First, we explored how the Surface could be used with existing vision-based recognition algorithms, ranging beyond those supported by existing Surface software. Second, we explored how the Surface could be used along with other displays to create a heterogeneous information space in which users can benefit from the complementary interaction and display capabilities of each device. Our Surface botanical field guide was installed in the exhibit,  “Since Darwin: The Evolution of Evolution” (Smithsonian National Museum of Natural History, Washington DC, June–July 2010), and other live demonstrations of this work were presented at various conferences.

Various Projects | Ravin Balakrishnan, University of Toronto, Canada

We have developed several prototypes that we believe would not have been possible to even think about without the Microsoft Surface technology. These include building an adjustable frame to mount the Surface unit in a variety of configurations, a scenario with the Surface Monster tangible, which features fiber optic transmission, an exploration of switching between modalities of typing, touching, and mousing, freeform text input, hand painting, and a concept of tangible actions that are based on the broad idea of making the user’s previous actions tangible and manipulable. While we certainly did not use the Microsoft Surface in the expected way, we believe that our explorations are beneficial to those who will develop for the Microsoft Surface, or use it as a research platform in the future.