Automatic and Scalable Fault Detection for Mobile Applications

This paper describes the design, implementation, and evaluation of Vanar Sena, an automated fault finder for mobile applications (“apps”). The techniques in Vanar Sena are driven by a study of 25 million real-world crash reports of Windows Phone apps reported in2012. Our analysis indicates that a modest number of root causes are responsible for many observed failures, but that they occur in a wide range of places in an app, requiring a wide coverage of possible execution paths. Vanar Sena adopts a “grey box” testing method, instrumenting the app binary to achieve both coverage and speed. Vanar Sena runs on cloud servers: the developer uploads the app binary; Vanar Sena then runs several app “monkeys” in parallel to emulate user, network, and sensor data behavior, returning a detailed report of crashes and failures. We have tested Vanar Sena with 3000 apps from the Windows Phone store, finding that 1108 of them had failures; Vanar Sena uncovered 2969 distinct bugs in existing apps, including 1227 that were not previously reported. Because we anticipate Vanar Sena being used in regular regression tests, testing speed is important. Vanar Sena uses two techniques to improve speed. First, it uses a “hit testing” method to quickly emulate an app by identifying which user interface controls map to the same execution handlers in the code. Second, it generates a Processing Completed event to accurately determine when to start the next interaction. These features are key benefits of Vanar Sena’s grey box philosophy.