By Douglas Gantenbein, Senior Writer, Microsoft News Center
Medical imaging today gives surgeons an ability to obtain a virtual peek inside the human body in a way that rivals the campy 1966 movie Fantastic Voyage, in which a team of physicians is miniaturized and injected into a wounded diplomat to save his life.
In today’s operating theater, though, those virtual images can be difficult to retrieve and manipulate. A surgeon can’t use a mouse or keyboard, because they are unsterile and pose a risk of infection. A surgeon can give instructions to someone else, who then can operate a mouse or keyboard, but the surgeon must break concentration to say, “Right … right … up a little … OK.”
GigaPath: Whole-Slide Foundation Model for Digital Pathology
Digital pathology helps decode tumor microenvironments for precision immunotherapy. In joint work with Providence and UW, we’re sharing Prov-GigaPath, the first whole-slide pathology foundation model, for advancing clinical research.
Now, a team at Microsoft Research Cambridge consisting of social scientists, computer scientists, and designers has used the Kinect for Windows hardware and SDK to simplify this process. It enables doctors to use simple hand gestures to change, move, or zoom in on CT scans, MRIs, and other medical images. In initial development, the Kinect for Windows-based system has thrilled surgeons who have seen it and who believe it could help make surgery faster and more accurate. The hope is that these systems will deliver better outcomes to patients when fully field-tested and approved.
Helena Mentis
The project, called Touchless Interaction in Medical Imaging, to bring Kinect for Windows into surgical suites has been guided by Helena Mentis, a Philadelphia native and a postdoctoral researcher with the Socio-Digital Systems group at the Cambridge lab. She has been intrigued for a long time by how technology can improve medical practices. She earned a Ph.D. in Information Sciences and Technology from The Pennsylvania State University, writing a thesis on how expressions of emotion between health-care workers helped foster collaboration in emergency rooms.
Microsoft Research Cambridge’s extensive work in exploring new technology attracted Mentis to that facility. The lab’s researchers also were instrumental in delivering many of Kinect for Windows’ capabilities, so when Mentis heard about Kinect, she soon began to investigate how it might be deployed in medical settings.
“Even before Kinect came on the market, my colleagues and I started to do some fieldwork with an eye on whether something could be developed for surgical theaters,” Mentis says. “The initial idea was to learn where we would want to use the system and what type of systems could interact with it in the surgical area.”
When Kinect for Xbox 360 hit the market in November 2010, Mentis was ready to go. Since then, her team has worked closely with surgeons to test it at several hospitals, particularly vascular surgeons at St. Thomas’ Hospital, located prominently across the Thames River from the British Houses of Parliament.
Vascular surgeons often can avoid performing open-heart surgery by placing stents to open clogged arteries or to support aortas with an aneurysm. But proper placement of the stent requires a great deal of precision, and medical imaging plays a big role in enabling that precision.
Kinect for Windows enables surgeons to use gestures to manipulate medical imagery while in the surgical theater.
By using the Kinect for Windows sensor in conjunction with a large monitor, a surgeon easily can change the image view of a patient’s chest area, look at a new image, or zoom in on details—all without touching a screen, a keyboard, or a mouse. That can help the surgeon determine more quickly the proper placement of a stent.
“They love it,” Mentis says of surgeons’ reactions to seeing Kinect in action. “The idea of being able to control almost anything with just a wave of the hand really speaks to how surgeons want to feel like they have control over things. They want to be able to save as much time as possible and to be able to get a patient in and out of anesthesia as quickly as possible. Those all are things that they’re motivated by, so they think this is the greatest thing ever.”
In the operating room, the setup consists of a Windows PC connected to a fluoroscopy scanner through an external video-capturing device, a Kinect for Windows sensor attached to the PC with a USB cable, and an LCD monitor. The Kinect for Windows sensor is placed below or above one of the computer monitors that the surgeons currently use for visualising medical images.
Key 3-D medical images are acquired before the surgery using a CT (computed tomography) scanner. During surgery, the fluoroscopy scanner captures real-time X-ray images, which are overlaid on one another, registered, and displayed in an image viewer. A surgeon then uses gestures to rotate images, zoom into details, and more.
Although still being tested, the Kinect for Windows sensor, Mentis said, works well in the surgical suite. It’s able to track and follow multiple people, something other camera-based gestural systems cannot do. It also could be “trained” to know which individual in a surgical suite wanted to use the system, which can be a challenge, given that everyone essentially is masked and dressed the same.
To do that, the Kinect sensor is programmed to respond to the command, “Kinect, control,” in conjunction with a raised hand. The sensor then “watches” that individual for instruction, ignoring other people who might themselves be gesturing and talking.
Surgeons also must avoid swinging their arms in too wide of an arc while working with the Kinect for Windows hardware to avoid touching a contaminated object. The system compensates by supporting gesturing in a limited area in front of the surgeon’s body, using the Near Mode feature in Kinect for Windows.
“This is very different than designing for a game, where one can flail about wildly,” Mentis says. “The surgeon’s sterile area is from chest to waist and from shoulder to shoulder. We worked hard to ensure the surgeon could fully use the system while staying within a safe and sterile zone.”
But otherwise, sterility isn’t a particular challenge. If the Kinect for Windows sensor and its accompanying monitor are kept clean and left untouched, the risk of infection is substantially reduced.
In March, Mentis and her team began to test the first iterations of the system with several vascular surgeons in a surgical theater without patients. By the beginning of May, the first field-trial test with patients was under way.
Surgeons who have worked with the research team are eager to give the system a regular place in surgery, once finalized.
“As a surgeon, being able to manipulate information in a touch-free way while operating will have enormous benefits,” says Dr. Neville Dastur, a vascular surgeon at the Brighton and Sussex University Hospitals National Health Service (NHS) Trust. “Breaking the flow of the operation to get someone to do it for you not only distracts from performing a delicate surgery, but it’s also time-consuming.”
Dr. Tom Carrell, vascular surgeon at Guy’s and St. Thomas’ NHS Foundation Trust, says adding Kinect for Windows to the surgical theater could be a big step forward.
“With Kinect, we could revolutionize the way we do complex operations,” Carrell says. “Patients and clinical teams will spend less time in theater, and surgeons will be more in control of the information they need.”
More Possibilities
The Kinect for Windows-based imaging system, Mentis says, already is being evaluated for other new uses. One example is the manipulation of 3-D models of the brain for neurosurgery at Addenbrooke’s Hospital in Cambridge.
“Neurosurgeons are interested in being able to go deep into the brain and extract a tumor or to even do a biopsy for a tumor,” she says. “Unfortunately, tumors never are right on the outside of the brain—they typically are within the brain tissue. The idea is to get in there with the minimal amount of brain damage possible, and better ability to view imaging can help with that.”
Another aim is to develop a touchless interaction-in-surgery tool kit that can be used by any hospital or system interested in applying touchless interaction to an imaging system.
Mentis is working with an extensive team to make the Kinect for Windows-based surgical-suite setup a reality. Partners at Microsoft Research Cambridge included principal researcher Abigail Sellen, senior researcher Antonio Criminisi, designer Robert Corish, and vendor Kenton O’Hara. The team also worked with Mark Rouncefield and Gerardo Gonzalez from Lancaster University’s Computing Department, which is supported by Microsoft Research Connections. The team, which also included participation from King’s College London, hopes that innovations in technology using Kinect for Windows that make surgery easier and faster will obtain appropriate governmental approval.
Graphics and multimedia 
Medical, health and genomics