When students are stuck with a math problem, searching online for help can be challenging. Typing complex math problems also poses a significant barrier. We look to reimagine input modalities through camera to make search easily accessible on mobile devices without the need to input a query in a search box. Image as an input is particularly challenging, and we develop new ways to understand user intent and provide appropriate knowledge and actions geared towards learning and understanding concepts. Microsoft Math Solver is a mobile and web app that allows users to scan printed or handwritten math problems or use digital ink to write the problem on screen, and instantly get help understanding how to solve the problem with step-by-step description, online video lectures, and links to web sites with similar problems.
State-of-the-art Math OCR models are used to digitize printed or handwritten images to math expressions. Math expressions are processed in a query analyzer that generates a parse tree corresponding to the input expression. This step is used to normalize the input while handling any noisy text or errors. The normalized expression is then processed by variety of numerical solver engines that generate solutions and steps to solve the problem. In addition, we perform entity linking to extract key mathematical concepts related to the input problem. Features from the expression tree are used to train this multi-label classifier. These entities are used to identify related online video lectures for the input problem. We also perform a math expression search over top math domain sites to determine websites with similar problems. The results are merged and rendered in a unified SERP view on the client with rich math layout rendering.
Download the mobile experience at math.microsoft.com. (opens in new tab)