By Rob Knies, Managing Editor, Microsoft Research
Leonardo da Vinci and Filippo Brunelleschi resound through history as two of the guiding lights of the Italian Renaissance. Leonardo, of course, gifted us with the Mona Lisa and The Last Supper, but he also excelled at mathematics, engineering, anatomy, botany, and a clutch of additional artistic endeavors. Brunelleschi, meanwhile is heralded for his dome atop the Duomo in Florence, but in addition to that engineering feat, he also devised the linear perspective, a revolutionary principle in 15th-century painting.
Men of art, men of science—such wide-ranging expertise is rare 600 years later, but if Lolan Song has her way, such disparate disciplines soon might be converging anew to help preserve some of the world’s greatest treasures.
GigaPath: Whole-Slide Foundation Model for Digital Pathology
Digital pathology helps decode tumor microenvironments for precision immunotherapy. In joint work with Providence and UW, we’re sharing Prov-GigaPath, the first whole-slide pathology foundation model, for advancing clinical research.
Song, senior director of the Microsoft External Research Division for Microsoft Research Asia, well-known as University Relations in the Asia-Pacific region, has joined Baining Guo, assistant managing director of Microsoft Research Asia, in spearheading an effort called eHeritage, which aims to use computing technology to assist and partner with the academic community in the Asia Pacific region to digitize portions of the Asian cultural heritage endangered by deterioration or destruction. And in doing so, she is attempting to recombine the threads of science and art that have come unraveled over the centuries.
“Science and art are in the same family,” Song says, “but over time, that labor division has become more and more specialized, and the arts and science became separate fields. We think that, through new computing technology, we can remarry those two large fields together so that those two can be combined again.”
Guo provides technical advice to the project, receiving support from Ying-Qing Xu, lead researcher in the Internet Graphics Group; Xin Tong, lead researcher and research manager in the same group; Yasuyuki Matsushita, lead researcher in the Visual Computing Group; and Moshe Ben-Ezra, also a lead researcher for the Visual Computing Group. Such teamwork has enabled Microsoft Research Asia to establish itself at the forefront of technological innovation in areas such as computer graphics, computer vision, multimedia, and networking.
The eHeritage project is dedicated to advancing multidisciplinary research worldwide by engaging and partnering with the academic community, focusing on:
- Breakthrough research and innovation that takes advantage of the latest, greatest research work from Microsoft Research to utilize Microsoft tools and technologies to solve difficult problems in high-impact and socially relevant domains.
- Worldwide participation with leading universities and research communities as an active, trusted partner to address problems of concern to academia and throughout the computing industry.
- Community engagement with researchers, technologists, and IT experts in all regions, seeking insights from top innovators through collaborative efforts with Microsoft Research and Microsoft.
Microsoft Research Asia celebrated its 10th anniversary in November, and the lab has demonstrated the quality of its research in top conferences and journals the world over. Having ascended to world-class status, it was only natural that those at the lab began to seek opportunities to make technological contributions to the society at large.
Lolan Song, senior director of the Microsoft External Research Division for Microsoft Research Asia, delivers a talk in May 2007 during the inauguration of UNESCO’s World Heritage Training and Research Institute for the Asia and the Pacific Region.
“About two years ago, we starting thinking about consolidating our work,” Song says, “especially the work related to culture and history, not only to make this effort an internal research project or application, but also to use that work as one of our regional theme projects, in which we would work with our partners in the academic world.
“In the last year, we’ve funded a number of projects—throughout Asia, not only in China. As the economy and culture becomes more globalized, each country or culture is still trying to best demonstrate its own unique identities. Our eHeritage project will provide the opportunity for Asian countries to show their culture and their heritage. That’s why this is so unique.”
The work the eHeritage project is designed to support can be divided into three broad areas:
- Data collection, using sensor networks, databases, instruments, and observation.
- Data processing and visualization, via data modeling, restoration, annotation, and searching.
- Data preservation, within digital museums and using virtual reality.
Microsoft Research Asia’s plan is to apply the latest technologies, in conjunction with academic partners, to aid the preservation, interpretation, and dissemination of cultural and natural heritage for research, education, and protection.
“Our objectives,” Song says, “are, first, to, wherever possible, apply our lab’s work into the area of cultural and natural heritage, so that our research work not only has academic value, but also generates a social impact for Asia—and, also, maybe for the world.
“Second,” she continues, “our own effort is not sufficient. We have to work with our academic partners to make a bigger impact. We might facilitate this by funding some projects in those areas. We want to build a community around this subject, so that we can collaborate together. Maybe, a couple of years from now, we can have a collection of this kind of work that can demonstrate how computing technology can help preserve and educate people. Hopefully, that will make a bigger impact.”
To that end, in July 2008, in Beijing, Microsoft Research Asia held the first eHeritage Workshop, which drew 40 attendees from 23 universities and institutes from across Asia. In addition to China, Japan, South Korea, Singapore, and Australia were represented, sharing the latest research.
“It was the first time we organized this kind of a workshop,” Song recalls. “It had good representation from both the computer-science side and from the domain experts, such as people from the Palace Museum in Beijing and the National Palace Museum in Taipei. We had people coming from the museum on both sides.”
Workshop keynotes were delivered by Katsushi Ikeuchi, professor at the Institute of Industrial Science at the University of Tokyo; James Quo-Ping Lin, researcher and director of the Information Management Center at Taiwan’s National Palace Museum; and Guo.
Ikeuchi, a former professor at Carnegie Mellon University renowned for his work in computer vision, has been working closely with the eHeritage project.
“His work on digitization in Cambodia has been well recognized,” Song says. “Professor Ikeuchi has been one of the advisers and worked with us to lead this eHeritage workshop, from planning to launch to execution. He has been a key person working with us on this project.”
The goal of the workshop was to build a diversified community for cross-discipline collaboration, and the participants came from various backgrounds, including computer science and museum experts. Apparently, the event was a rousing success.
“Because of the launch of this regional project,” Song says, “the head of the Chinese National Commission of the United Nations Educational, Scientific and Cultural Organization (UNESCO), Zhang Xinsheng, who is also a vice minister of Education in China, has given us very positive feedback about our work in this area.”
In fact, Zhang commended the eHeritage effort during a UNESCO gathering in Beijing and specifically mentioned Hsiao-Wuen Hon, managing director of Microsoft Research Asia, and the eHeritage project during an annual event for multinational companies in China, when Zhang presented a special contribution award to Microsoft Research Asia.
“Even though we’ve just started the project,” Song says, “it’s already being recognized as a unique contribution to this area.
The ancient painting Along the River During the Qingming Festival is being digitized and annotated as part of the eHeritage effort.
“Our sister lab, Microsoft Research India, also has done an exciting project called Digital Heritage, led by Vidya Natampally,” Song adds. “For the second eHeritage workshop, scheduled for June in Dun Huang, we have invited Vidya and Curtis Wong from the Next Media group at Microsoft Research Redmond. It is our hope that, with the joint effort from multiple labs within Microsoft Research, we can apply more relevant research work to this area to demonstrate our combined commitment to preserve and promote the world’s heritage.”
One early project already under way, a collaboration with the Palace Museum of Beijing, involves the digitization and annotation of a well-known Chinese painting, Along the River During the Qingming Festival, attributed to the artist Zhang Zeduan of the Song Dynasty, which reigned a thousand years ago. Another, in concert with the National Taiwan University and the National Palace Museum in Taipei, would enable museum-goers to see through weathering to gaze at the original look of a Zhou Dynasty ritual bronze vessel, the 3,000-year-old Mao Kung Ting.
Taipei museum-goers can get a peek at how the bronze vessel Mao Kung Ting appeared 3,000 years ago.
A couple of different graphics projects are capturing attention, hardly surprising given the world-class reputation of the Internet Graphics Group at Microsoft Research Asia.
One, called Appearance Manifolds, simulates the weathering of real-world objects from one image acquisition. They also can be rendered as they originally appeared, before weathering began. From an ancient object’s current appearance, the technology can infer and display weathering differences on different areas, enabling digital recovery of artistic heritage.
The second, Inverse Diffusion Equation, creates a model of light propagation in translucent objects, such as priceless Chinese art objects crafted in jade that are often sealed in museum-display windows, a barrier to all-angle viewing and free interaction.
Tong has developed a technique that enables screen presentation of lifelike, half-translucent images of jade and light, supporting touch-based interaction and full-angle viewing, an achievement heretofore unattainable.
“That effect has attracted attention from the National Palace Museum in Taiwan,” Song notes. “They are interested in applying this technology to their digitization of the works in the museum.”
Also of interest to museums is the lab’s work in gigapixel-image annotation. Museums use high-resolution digital cameras for archiving, management, and research on ancient paintings and drawings.
This camera enables the capture of ultra-high-resolution images that enables informative annotation.
“Using a camera at a reasonable cost,” Song says, “you could produce museum-quality pictures that will ensure the fidelity of the artwork.”
Ben-Ezra has developed a digital camera that takes gigapixel images that creates continuous, perspective, flat images larger than 1.6 gigapixels—at reasonable cost.
The camera takes photographs with more than 10 times the information of the highest-resolution digital cameras commercially available. The camera’s photos are more than six times sharper than images the human eye generates.
The annotation part of the project is designed to enhance the browsing experience by providing useful labels, multimedia and hyperlinks, and, potentially, relevant advertising.
Microsoft Research Asia also is sponsoring several related projects:
3-D Traditional Chinese Painting in Gaming Scenarios
A collaboration with the Beijing Film Academy at the Microsoft Digital Cartoon and Animation Laboratory, this project is designed to connect animation artists with computer scientists to apply Microsoft Research Asia techniques to animation of traditional Chinese painting. The intention is to use a library of brush-stroke shapes to enhance games.
“It is difficult to use computing techniques to simulate the effect of traditional Chinese ink painting,” Song says. “The joint project with the Beijing Film Academy uses animation techniques to portray the peaceful beauty of birds, lotus flowers, and a water pond in a game setting. Although I am not a gamer, I was touched by this game’s use of modern technology to animate traditional Chinese ink painting.”
Analysis of traditional brushstroke techniques enables real-time painting displays and re-animation, thus immersing gamers in the beauty of traditional Chinese culture via play.
“We want to make this simulation 3-D and dynamic,” Song says. “It’s very novel and very interesting.”
Digital Olympic Museum
This project, representing a collaboration with Beihang University of Beijing, aims to use advanced computing technologies to enable users to experience traditional Chinese sports.
In the wake of the XXIX Olympiad, held in Beijing in August 2008, project members are exploring ways to bring the Games and historical athletics data to life.
“We integrated several technologies into this project,” notes Xin Ma, program manager for the University Relations group. “We are producing the Digital Olympics Museum to demonstrate what Asian Olympic Games look like to students, using digital media, motion capture, user-interface design, and other technologies.”
The project promises to fascinate. A paper entitled 3D Modeling, Codec and Protection in Digital Museum, by Yue Qi, Su Cai, and Shen Yang of Beihang University, discusses an ancient game called “cuju” in which two teams of 12-16 players each compete to kick a “ju,” a leather ball filled with feathers, between crescent-shaped goal posts at each end of a field.
If that sounds oddly familiar, it should. The game, which emerged during China’s Warring States Period (476-221 B.C.), has been identified as the predecessor of association football—soccer in the United States—by the Féderation Internationale de Football Association, which organizes and governs the World Cup.
Another feature of the Digital Olympic Museum, which will feature videos and information on previous Olympics, will be a program to teach people ancient Chinese physical exercise.
The Five-Animal Exercise, for example, is based on the movements of the tiger, the deer, the bear, the ape, and the bird. The exercise, created by famous ancient Chinese doctor Hua Tuo in the second century A.D., employs a digital camera to capture the posture of participants as they perform key movements, and only when their posture conforms to requirements are participants allowed to proceed.
Digital Bayon Temple Project
The 12th-century Bayon Temple, part of the famous Angkor Wat ruins in Cambodia, presents an enormous challenge to researchers attempting to create a digital model at a one-centimeter resolution. Microsoft ® provided a Windows ®-based cluster that dramatically reduced the processing time for this huge amount of image data.
The key to this impressive, computationally aspirational endeavor is the creation of modeling algorithms for the structure of the temple complex.
An Asian Bridge
All of this work falls under the University Relations purview, which is to act as a bridge between Microsoft Research and academia in the Asia Pacific region and to contribute to the academic ecosystem.
“With the eHeritage project,” Song says, “we play another role as a bridge, between technology and art. We are good at building communities, and we have strong connections with faculty members and students. This is another way to demonstrate how we can work together to do something that one party alone would not be able to achieve.
“I also think this is a way for University Relations to demonstrate that Microsoft has the social responsibility to help protect and preserve the cultural heritage. That is the kind of non-tangible asset that needs to be preserved as we become more globalized in other areas.”
Song’s group will continue to work closely with academia, governments, and UNESCO across the Asia Pacific region. This fall, for instance, Microsoft Research Asia will sponsor a workshop on digital heritage during the International Conference on Computer Vision, to be held in Kyoto, Japan. The eHeritage effort simply adds to the list of topics on which the Beijing lab has been collaborating over the years.
But it remains special to those research personnel involved. With groups in Microsoft Research Redmond and Microsoft Research Cambridge working on various eScience projects under the direction of Tony Hey, Microsoft corporate vice president of External Research, Song hopes the eHeritage project can make comparable contributions.
“I think this is complementary,” she says, “to make it become a global project that starts from Asia. That would make it a complete, beautiful picture to show what Microsoft technology can do for society.”
Data platforms and analytics 
Graphics and multimedia