How do you take a concept from research to product? In the case of the Microsoft Touch Mouse (opens in new tab), it took a collection of prototypes, collaboration between transatlantic teams, and a lot of user testing. It also helps when the research that launched the project (opens in new tab) won the best-paper award during the Association for Computing Machinery’s 22nd Symposium on User Interface Software and Technology (opens in new tab).
Mouse 2.0: Multi-Touch Meets the Mouse (opens in new tab), a joint effort between Microsoft Research Redmond (opens in new tab), Microsoft Research Cambridge (opens in new tab), and Microsoft’s Applied Sciences Group (opens in new tab), introduced five research prototypes, each exploring a different touch-sensing strategy that influenced the design of different mouse form factors and their interaction possibilities. The research featured extensive user feedback, as well as practical comparisons of different techniques for enabling multitouch on the desktop. The prototypes included three camera-imaging approaches, multiple optical sensors, and the use of capacitive sensors on a curved surface.
Members of the Mouse 2.0 research team expressed the hope that they would be able to refine their prototypes, both ergonomically and in terms of their sensing capabilities, and make a deeper exploration of the interaction techniques specific to this new class of input devices.
GigaPath: Whole-Slide Foundation Model for Digital Pathology
Digital pathology helps decode tumor microenvironments for precision immunotherapy. In joint work with Providence and UW, we’re sharing Prov-GigaPath, the first whole-slide pathology foundation model, for advancing clinical research.
The researchers soon got an opportunity to refine their prototypes. Microsoft Hardware (opens in new tab) decided to get behind the research, and a team was formed to bring a multitouch mouse to market.
New Possibilities for the Humble Mouse
Hrvoje Benko (opens in new tab), researcher with the Adaptive Systems and Interaction (opens in new tab) group at Microsoft Research Redmond, has worked on both the Mouse 2.0 research and the Microsoft Touch Mouse product-development project. He recalls one of the key product decisions: selecting from five prototypes the one that would be the launching point for the new device.
“In the end,” Benko says, “we selected the prototype using capacitive touch sensing to track the position of multiple fingers on its surface. This approach offered the most consistency and flexibility in terms of how we could mount and integrate the sensor, which is important in a small form factor. Plus, unlike camera-based tracking, there are no issues with ambient light, so you reduce the calibration issues. It’s a much more controllable sensor.”
Microsoft Touch Mouse: the final form factor.
Although the choice of prototype simplified some of the technical issues, there were still plenty of challenges when it came to refining the mouse to the point where it was ready for consumer use. The design of the final form factor required sculpting and testing of hundreds of models. The team also examined user interactions and evaluated the kinds of gestures that made sense, developing an entire gestural set focused on enhancing window manipulation and management. At the same time, core technologies, such as firmware and hardware for capacitive sensing, had to be built and optimized for this specific form factor and device functionality.
“The gesture-recognition software is the brains behind all these interactions,” says John Miller (opens in new tab), software architect with the Cambridge Innovation Development (opens in new tab) team at Microsoft Research Cambridge. “Our gestures are multitouch and designed to amplify your experience with Windows 7 (opens in new tab). So they are optimized for window management: docking, moving, minimizing and maximizing, going backward and forward on your webpage, switching between tasks, and so on.”
Getting the Right Touch
Benko and Miller agree that one of the toughest problems they tackled was the requirement that users should be able to operate the device using classic point-and-click interactions, as well as the newly developed set of multitouch gestures. The mouse form itself added complications: The shape encourages a user to rest both palm and curled fingers on the entire touch-sensitive surface, creating constant contact.
“That made everything much, much harder,” Benko smiles ruefully. “Instead of making palm rejection and other issues easier, it added a few more challenges. But at the end of the day, our goal was to have a comfortable, great-looking mouse that people enjoy using, with a nice look and feel that support the gestures, so it was definitely worth the effort.”
Unlike touch-screen devices on which one or two clear touches make user commands easy to interpret, a small sensor surface and the nature of mouse usage creates an entirely different set of problems.
“If you have a touch-sensitive phone,” Miller explains, “you interact by touching the screen, and as soon as you’re done, your finger lifts off the screen. We have completely different issues with the mouse. We have a device that not only has to support gesture touches, but also has to deal with times when the user is just holding it.
A core technical challenge: developing a gesture set that enables clear differentiation between various types of user contact with the touch surface.
“Next, fingers can be very close together when making contact. To the sensor, they can appear as one finger rather than multiple fingers. But if you want to have reliable gesture recognition, you need a way to differentiate between one, two, or three fingers. We had to develop technology that enhances signal processing and reliably tracks contacts.
“And here is one more example,” Miller continues. “Everybody holds the mouse in a slightly different way. Some people hold their fingers flat on the mouse, and people with very small hands will hold the device differently than people with very large hands. So the mouse does not make contact the same way for all users, and they are all going to be performing these gestures in a slightly different way. As a result, there’s a lot more ‘noise’ to handle than from a touch-screen phone or a Tablet PC. We had to deal with a lot more data.”
To mitigate some of these problems, the team set a design goal that gestures should be both intuitive and distinct—the kind that would be hard for a user to perform by accident. This helped simplify the job of the recognition software.
They also developed a tool that recorded sensor data while human testers were using the mouse for actions such as pointing and clicking, multitouch gesturing, and grabbing and releasing the mouse.
“We ended up with data examples of good gestures for mouse usage and unintentional movements,” Benko says, “and this helped us conceive strategies for distinguishing between intentional gestures and incidental movements. It’s what allowed us to develop an engine that’s able to recognize some movements and ignore others.”
Collaboration Delivers a Quality Product
The Microsoft Touch Mouse project is unusual compared with other hardware-development projects, because it is not simply about hardware. Rather, it is a product that combines multiple disciplines in a tightly integrated way, a task that would have been impossible without close collaboration between multiple Microsoft Research and Microsoft hardware-development teams in different locations.
Decisions about the final product, for example, involved testing and evaluation of different prototypes and features by all parties.
Touch Mouse collaborators Hrvoje Benko (left) and John Miller.
“There were a lot of concepts from the original research,” Benko says, “and some of those we decided to leave out. That doesn’t mean they were bad ideas, just that we were being very careful about our choices. It’s how making a product works: You assess the pros and cons of every choice. Both the research and hardware teams were focused on nailing down the core experience, to make sure that everything we included was critical and didn’t distract from the user’s task. Our goal was to deliver a delightful, fluid desktop experience.”
Even though the multitouch-mouse project officially belonged to the hardware team, Microsoft Research remained integral to the development.
“The original Mouse 2.0 paper was just the starting point,” Miller says. “The research efforts didn’t stop there. They continued in tandem with product development. There was a lot of additional research from different parties before we could turn the multitouch-mouse concept into a device that consumers can buy off the shelf.”
For Benko and Miller, one of the most rewarding aspects of this project has been the close collaboration between the hardware team and Microsoft Research in both Cambridge and Redmond. It went beyond technology transfer and was absolutely critical to delivering a successful transition from research prototype to consumer product.
The Microsoft Touch Mouse proves that quality research doesn’t have to address technologies that are many years away from commercialization. Sometimes, it’s about exploring new possibilities. There’s always room for a better mousetrap—make that, a better mouse.
Hardware and devices