Ken Hinckley always thought that searching for information on his computer was like going on a fishing expedition. Every so often, he’d even catch whatever it was he wanted.
Hinckley, a senior researcher in the Adaptive Systems and Interaction group at Microsoft Research Redmond, has been netting better results than usual lately. And now he’s angling to share the secret to his success with Tablet PC users everywhere.
An example of an InkSeine note with links, searches, and screen clippings.
Spotlight: blog post
GraphRAG auto-tuning provides rapid adaptation to new domains
GraphRAG uses LLM-generated knowledge graphs to substantially improve complex Q&A over retrieval-augmented generation (RAG). Discover automatic tuning of GraphRAG for new datasets, making it more accurate and relevant.
That secret is InkSeine, an innovative new inking tool that Hinckley and his team began developing in 2006. InkSeine is designed to help users get the most out of a Tablet PC by unifying ink, search, and gather functions into a fluid workflow uniquely suited to pen interaction. The ink component enables users to capture freeform thoughts and ideas in handwriting on a page; the integrated search functionality gives users quick access to supporting materials and documents with a few simple pen gestures; and the gather element lets users integrate the search results back into their notes—all without breaking the creative flow. As for the name, Hinckley explains: To “seine” is to fish with a net; to fish for results from your ink notes, use InkSeine.
Effective Feb. 15, Microsoft Research is making a prototype of the InkSeine application available as a download. Hinckley, whose group is charged with exploring interface form factors beyond the traditional mouse and keyboard, hopes Tablet PC users will take the bait.
“InkSeine is a powerful and simple tool for thought,” says Hinckley, noting that it is not a supported Microsoft product, but rather a functional demonstration of Microsoft Research innovation. “We hope people will have fun playing with it.”
While InkSeine sports many entertaining, creative features, Microsoft Research takes the technology seriously. In December 2007, an early research prototype was released for testing to about 25 Microsoft Most Valuable Professionals in the Tablet PC world. Hinckley’s team made additional improvements based on that feedback before releasing InkSeine publicly. The next step is for a broader community of Tablet PC users to try InkSeine and contribute to the body of knowledge related to pen computing.
“Designing a really good pen interface is a very difficult problem,” Hinckley explains. “We think we have a good set of ideas, and there are aspects of InkSeine that have been proposed in the research literature for years, but nobody really knows if they work. The best way to advance the state of the art in this area is to try it out in the wild, so to speak. We want people who do drawings, sketches, and notes on their Tablet PCs to tell us what they’re excited about and what may need some rethinking. That’s how we learn the most.”
Users can click links on the InkSeine installation page to send feedback to the InkSeine development team, which also includes Raman Sarin, a senior research software-development engineer who wrote virtually all the code, and Shengdong Zhao, a former Microsoft Research intern who explored the initial ideas and assembled the first prototype. The page also guides users through the installation of InkSeine, which offers the best experience on the Windows Vista operating system but also supports Windows XP Tablet PC Edition.
‘Inking Is Great for Thinking’
The InkSeine project revolves around a desire to invent a new user experience, one that makes it easy for Tablet PC users to think and work in an intuitive, visual, and uninterrupted way. To achieve that, Hinckley and his collaborators took a unique approach, making the pen its focus and ink the premise of its design. The goal: Create a pen-based experience that isn’t a retrofit of using a computer with a mouse.
The search function in use with illustration of the InkSeine gestures available to assist navigation and filtering through search results.
The problem is, functions executed easily with a mouse and a keyboard—such as clicking through drop-down menus, navigating folders to locate information, typing in search queries, and switching between applications—can be onerous on a pen computer, either because of the interface or the limited screen real estate. InkSeine addresses those challenges through the inventive use of pen strokes and simple gestures, coupled with powerful in-place search functionality.
Consider a typical InkSeine usage scenario: As you embark on a project, you create a page on your Tablet PC where you jot down notes and sketch out a few ideas, with information arranged spatially as you see fit. You set up a visual folder where you gather in one place all the documents, e-mail messages, links to Web pages, design diagrams, and other resources related to your project. And whenever you need to locate a piece of information, you simply draw a circle around a handwritten word or phrase to create a query right in your notes, without having to switch to a search application. InkSeine, in effect, goes fishing by searching your ink directly, and within seconds, it presents all the information in your computer with those keywords in it. Or, with a different pen stroke, you can tell InkSeine to search the Web instead. Either way, you can find what you want quickly and easily without leaving the pen-and-ink environment.
“The search interface is the most innovative and unique feature InkSeine has,” Hinckley says. “It’s built on top of Windows Vista Search, so it takes advantage of Microsoft’s great search technologies, but it presents that functionality to users in a completely new way.”
Hinckley, who has used InkSeine extensively since May 2007, hopes users will discover that its tools work as fluid extensions of the creative process and that its gestures quickly become habitual. At a certain point, he says, you stop thinking about the computer and how it operates, and your attention just stays focused on what you’re thinking.
“To me, the information superhighway is a notebook page with plenty of wide-open space,” Hinckley says. “There’s no speed limit to how fast I can fly around with my pen.”
Hinckley has focused on pens as a research topic since long before the InkSeine application was conceived. InkSeine combines some of the most intriguing ideas in pen computing with some of the note-taking technologies found in Microsoft OneNote and Windows Journal, and the prototype complements the offering with the in-place search functionality.
“We’ve taken a large number of these innovations—some that we’ve developed ourselves and some that have flowed out of the academic research community and put them together in a novel way,” says Hinckley, whose 11 years at Microsoft Research have been marked by his enthusiasm for improving and innovating user interfaces. “By restructuring the interface to make all those things easy and painless and effortless, we’re changing how you use the tool. We’re solving all those problems so people can perform whatever task they face in a very intuitive way.”
Search and information retrieval