Research Collection – Shall we play a game?

From a research point of view, games offer an amazing environment in which to develop new machine learning algorithms and techniques. And we hope, in due course, that those new algorithms will feed back not just into gaming, but into many other domains. Beyond the very technical machine learning techniques themselves, gaming is an environment in which we can explore the relationship between AI and people, and see how they can work in partnership. It’s a very rich environment in which to drive new research ideas.

Dr. Chris Bishop, Technical Fellow and Lab Director, Microsoft Research Cambridge

The history of computer gaming goes back nearly as far as the history of computer science itself. As early as the 1950s, researchers were using games to push the boundaries of computing, driving many advances in computer graphics and human-computer interaction. Games played a role in the earliest research into artificial intelligence, as researchers trained the ancestors of modern AI systems to learn and play games like chess and tic-tac-toe. From the 1960s campus phenomenon of Spacewar! to the insatiable demand for the latest generation of consoles, games endure as a popular and engaging pastime – and, for many future researchers and developers, as a gateway to a career in computer science.

Today, researchers are working to advance the state of the art in gaming in many different ways – enabling more realistic and immersive sound and graphics, using AI to improve gameplay and make gaming more engaging for spectators, and making games more accessible, inclusive and safer to play. Games continue to play a role in advancing artificial intelligence, offering up new challenges for AI models to solve, and using game environments to train language models and autonomous systems. Chess also continues to be fertile ground for AI research; most recently, researchers from Microsoft and their collaborators explore how to better match AI to human skill levels and playing styles.

Listen to this podcast with Dr. Chris Bishop and Microsoft Executive Vice President of gaming Phil Spencer about the intersection of research and gaming at Microsoft.

In February 2021, Microsoft Research hosted the AI and Gaming Research Summit, which brought together researchers and innovators from academia, creators from the gaming industry, Microsoft Research (MSR), and Gaming teams at Microsoft, to share ideas and discuss how AI and machine learning is transforming the landscape of gaming. All recorded talks are now available on the Videos tab at the event site.

This collection highlights much of the research happening across Microsoft to advance the state of the art in gaming, and put games to work to enable new breakthroughs in artificial intelligence.

Games as a research tool

AirSim

Microsoft AirSim (Aerial Informatics and Robotics Simulation) is an open-source robotics simulation platform. From ground vehicles, wheeled robotics, aerial drones, and even static IoT devices, AirSim can capture data for models without costly field operations. AirSim works as a plug-in to Epic Games’ Unreal Engine 4 editor, providing control over building environments and simulating difficult-to-reproduce, real-world events to capture meaningful data for AI models.

Using AirSim, researchers can capture data for machine learning models in a virtual environment, without costly field operations.

Explore more

---

Hybrid Reward Architecture

Games are popular as a testbed for new machine learning techniques because they can be very challenging and allow for easy analysis of new learning techniques in a controlled environment. For reinforcement learning (RL), where the goal is to learn good behavior in a data-driven way, the Arcade Learning Environment (ALE), which provides access to a large number of Atari 2600 games, has been a popular testing environment. The Hybrid Reward Architecture project combines standard reinforcement learning techniques with deep neural networks, with the aim of outperforming humans in Arcade Learning Environment (ALE) games. In 2017, it achieved a perfect score on Ms. Pac-Man.

Explore more

---

Project Malmo

The Malmo platform is a sophisticated AI experimentation platform built on top of Minecraft, and designed to support fundamental research in artificial intelligence.

The Project Malmo platform consists of a mod for the Java version, and code that helps artificial intelligence agents sense and act within the Minecraft environment. The two components can run on Windows, Linux, or Mac OS, and researchers can program their agents in any programming language they’re comfortable with.

Minecraft is ideal for artificial intelligence research for the same reason it is appealing to the millions of fans who enter its virtual world every day. Unlike other computer games, Minecraft offers its users endless possibilities, ranging from simple tasks, like walking around looking for treasure, to complex ones, like building a structure with a group of teammates.

Explore more

---

Super Phoenix (Suphx)

Mahjong is a popular multi-player imperfect-information game, but it is very challenging for AI research due to its complex playing/scoring rules and rich hidden information. In 2019, Microsoft researchers designed Super Phoenix (Suphx), an AI for Mahjong based on deep reinforcement learning with some newly introduced techniques including global reward prediction, oracle guiding, and run-time policy adaptation. Suphx has demonstrated stronger performance than most top human players in terms of stable rank and is rated above 99.99% of all the officially ranked human players in the Tenhou platform. This is the first time that a computer program outperformed most top human players in Mahjong.

Explore more

---

TextWorld

Text-based games may seem primitive next to the beautifully rendered graphics of today, but to succeed at even the simplest such game, humans use a special set of skills. We seamlessly and simultaneously comprehend language descriptions, plan our next moves, visualize the environment, remember important information, and generalize past experiences to new situations. AI agents don’t yet possess these capabilities, but they are key to general intelligence. Microsoft TextWorld is an open-source, extensible platform that both generates and simulates text-based games. It is a popular testing ground for AI, used by researchers to train, evaluate, analyze, and improve reinforcement learning (RL) agents that learn from and interact with the world through language.

Explore more

---

Research to improve the gaming experience

Project Paidia

Project Paidia (opens in new tab) is a research project in close collaboration between Microsoft Research Cambridge and Ninja Theory (opens in new tab). Its focus is to drive state of the art research in reinforcement learning to enable novel applications in modern video games, in particular: agents that learn to collaborate with human players.

In contrast to traditional approaches to crafting the behavior of bots, non-player characters, or other in-game characters, reinforcement learning does not require a game developer to anticipate a wide range of possible game situations and map out and code all required behaviors. Instead, with reinforcement learning, game developers control a reward signal which the game character then learns to optimize while responding fluidly to all aspects of a game’s dynamics. The result is nuanced situation and player-aware emergent behavior that would be challenging or prohibitive to achieve using traditional Game AI.

Explore more

---

RoomAlive Toolkit

RoomAlive Toolkit is an open source SDK that transforms any room into an immersive, augmented, magical entertainment experience. RoomAlive Toolkit presents a unified, scalable approach for interactive projection mapping that dynamically adapts content to any room. Users can touch, shoot, stomp, dodge and steer projected content that seamlessly co-exists with their existing physical environment.

Explore more

---

SeeingVR

SeeingVR is a research toolkit for making virtual reality more accessible to people with low vision – moderate to severe visual impairments that cannot be corrected fully by glasses. It includes a number of enhancements that provide visual and audio augmentations to existing VR experiences – including magnification, edge enhancement, brightness and contrast adjustment, text augmentation and text-to-speech. The toolkit also enables VR developers to add capabilities such as object descriptions and object labeling and highlighting. SeeingVR is designed to be used with the Unity VR development platform.

SeeingVR includes enhancements that help developers make VR more accessible to people with low vision.

Explore more

---

Project Triton

Project Triton renders immersive acoustics that responds to the shape and materials of 3D environments in games and augmented/virtual reality (AR/VR). It is unique in modeling the true wave physics of sound, including diffraction, producing natural effects like smooth occlusion and scene-dependent reverberation. Detailed simulation is performed offline, yielding a lightweight runtime that scales from desktop to mobile phones. Incubated over a decade in Microsoft Research, Project Triton ships in major game titles like Gears of War, Sea of Thieves, and Borderlands 3. Plugin integrations for Unity and Unreal Engine are available as part of Project Acoustics.

Explore more

---

TrueMatch

Developed by Microsoft researchers and based on reinforcement learning, the TrueMatch matchmaking system uses AI to decide which people should play together in online multiplayer games.

TrueMatch dynamically adjusts players’ matchmaking allowances based on their location and skill level. If there are many players nearby, then TrueMatch shrinks the allowances so that they only get fair matches with low ping. If there are few players nearby, TrueMatch expands the allowances. Previous systems expand the allowance only after a player has waited a long time in the queue. TrueMatch starts the search pre-expanded, cutting wait time in half in low population regions. TrueMatch can also give players a personalized estimate of their wait time, before they queue up to play.

TrueMatch has launched in Gears of War 4, Gears of War 5, and Halo 5.

Explore more

---

TrueSkill

The TrueSkill ranking system is a skill based ranking system for Xbox Live developed at Microsoft Research. The purpose of a ranking system is to both identify and track the skills of gamers in a game (mode) in order to be able to match them into competitive matches. TrueSkill has been used to rank and match players in many different games, from Halo 3 to Forza Motorsport 7.

The classic TrueSkill ranking system only uses the final standings of all teams in a match in order to update the skill estimates (ranks) of all players in the match. The TrueSkill 2 ranking system also uses the individual scores of players in order to weight the contribution of each player to each team. As a result, TrueSkill 2 is much faster at figuring out the skill of a new player.

An improved version of the TrueSkill ranking system, named TrueSkill 2 (opens in new tab), launched with Gears of War 4 (opens in new tab) and was later incorporated into Halo 5 (opens in new tab).

Explore more

---

Watch For

Developed by Microsoft researchers as part of a 2017 internal company hackathon, Watch For identifies and surfaces the most interesting parts of a livestream in real time. It recognizes text, symbols, objects, players and other elements that signal exciting moments in a game, and uses AI to learn over time as more streams and content are fed into the model. After winning the hackathon, the researchers worked with product groups to integrate the technology into HypeZone, which was part of the company’s Mixer game streaming service. The technology was subsequently incorporated into MSN’s Esports hub, and was also used by the Cloud9 professional esports organization to derive insights to support match preparation. Based on customer feedback and the team’s combined AI expertise and end-to-end distributed systems knowledge, the Watch For team is continuing their exploration in this area, and working on tools for enabling non-experts to efficiently extract video data.

Opens in a new tab