Microsoft’s Experimentation Platform (ExP) provides a platform used by product teams across Microsoft to run 1,000s of A/B tests every month. From a product perspective this means that we have a big responsibility both as a steward for data-driven decision-making and as an innovative center of excellence. As a result ExP’s product team must prioritize effectively to maximize the impact of investments, balancing long-term goals with ongoing customer feedback. In this post we describe some of the strategies and processes that we use to build a world–class scalable experimentation platform with a core focus on trustworthy A/B testing.
Set your sights on a long-term vision and measure your progress
ExP’s mission is to accelerate innovation through trustworthy analysis and A/B testing. Trustworthiness is at the core of our platform, in terms of both giving users the ability to run successful A/B tests and empowering them to make high quality decisions. To align our investments with long-term priorities for our customers including product teams throughout Microsoft such as Office, Bing, xBox, and Azure, we revisit our product vision and strategy on an annual basis. This includes using the OKR (objectives and key results) framework to set new goals that measure our impact which includes tracking adoption, engagement, and experiment quality. As a product team we also invest in testing our product by running our own A/B tests to protect the customer experience by assessing the impact of new features on the end-user experience.
Keep a pulse on your customers
A key component of our product development process is to stay in touch with our customers to track their success, collect targeted feedback, and triage new asks. This requires a multi-faceted approach as outlined here:
- Regularly engage to accelerate the A/B testing flywheel: As a platform team we have several ways to engage with our customers to accelerate their A/B testing flywheel journey [1]. To ensure that our customers are successful at ramping up A/B testing for their products, we support their continuous and iterative development process by organizing periodic joint initiative reviews that include participants from product, data science, and engineering. To prepare for each initiative review, we collaborate with our customers to write a document that summarizes highlights and lowlights, checks in on progress, and captures opportunities for improvement and new feature requests [2]. These documents are reviewed in a meeting with dedicated reading and commenting time during the first half followed by discussion of any feedback, comments, and decision in the second half to encourage a high-quality conversation.
- Create opportunities to collect continuous feedback: In addition to holding regular business reviews, our product team also organizes a monthly customer council meeting in which we present prototypes and get product feedback on new or existing features. The customer council meetings help us get feedback to quickly iterate and serve to supplement other avenues of feedback including focus groups, cognitive design walkthroughs, and one-on-one usability studies. These additional feedback mechanisms are scheduled on an as-needed basis to get more focused feedback that goes beyond in-app feedback surveys and quantitative results from A/B tests.
- Prioritize and triage customer asks: To ensure that customer feature requests are prioritized appropriately we hold a weekly triage meeting in which feature asks are discussed and the team decides on a priority relative to our existing roadmap and customer commitments. Regardless of the outcome, a key component of this process is to close the loop with the customer to ensure alignment and maintain trust.
Create a culture that fosters cross-team alignment and collaboration
In addition to having a fantastic team, one of the keys to successfully building a world class product is to have the right process. For our product team, this means carefully balancing roadmap prioritization by considering the return on investment of each project before we make a commitment to our customers. To manage our backlog, ExP Product Managers work within a trio that includes Data Scientist and Engineering representation. In collaboration with their trios, ExP Product Managers spend approximately 50% of their time on project management and execution and the remaining 50% on forward-looking planning which includes focusing on the product vision for their product area, maintaining a long-term backlog, and developing new design prototypes. This gives the team the right balance and ensures that everyone stays aligned.
From a planning and execution perspective, we’ve created a continuous planning process in partnership with our engineering and data science teams that starts with backlog grooming within each product area and culminates in a cross-team backlog grooming exercise to set priorities and determine alignment across all teams. By revisiting prioritization on regular basis using an agile Kanban-like process that considers return on investment, we can react quickly to shift resources to address changes in scope and priorities and stay focused on the highest impact deliverables for our customers.
To manage specific projects, we use Azure DevOps to track ideas and feature requests from their inception until launch. We follow a lightweight process with variable detail depending on the size of the project which includes the following steps: (1) product spec with job stories and UX mocks, (2) dev design with work item breakdown, (3) detailed UX design, (4) implementation, and (5) A/B testing and customer feedback. Following this sequencing enables us to collect customer feedback and ensures that everyone is aligned and can review each other’s work. Before a feature is ready to be launched, we schedule a bug bash and work directly with pilot customers to get feedback with an initial Beta release before rolling out the feature more broadly with an A/B test using our own platform. To monitor progress across the team and learn whether we are on-track, we use Power BI to visualize the status of work. Looking beyond the launch of specific features, we also use Power BI to report on feature usage and product engagement, leverage the customer feedback process outline above and come up with our own innovative ideas to help us determine where to invest next.
Continuously invest in a world class product that powers trustworthy A/B testing
If you are a frequent reader of our blog, you have probably noticed some of our posts which highlight investments in trustworthy A/B testing such as diagnosing sample ratio mismatch [3] and alerting [4]. We take pride in our product development process to continuously invest in a world class product and we will continue to share new and interesting A/B testing features on this blog as we advance our mission to accelerate innovation through trustworthy analysis and A/B testing.
– Sebastian Kohlmeier, Microsoft Experimentation Platform
References
[1] A. Fabijan, B. Arai, P. Dmitriev, and L. Vermeer, “It takes a Flywheel to Fly: Kickstarting and Keeping the A/B testing Momentum.” https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/it-takes-a-flywheel-to-fly-kickstarting-and-keeping-the-a-b-testing-momentum/.
[2] B. Porter, “The Beauty of Amazon’s 6-Pager.» https://www.linkedin.com/pulse/beauty-amazons-6-pager-brad-porter/.
[3] A. Fabijan, T. Blanarik, M. Caughron, K. Chen, R. Zhang, A. Gustafson, V. Kavitha Budumuri, and S. Hunt, “Diagnosing Sample Ratio Mismatch in A/B Testing.” https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/.
[4] A. Agrawal, and J. Lu, “Alerting in Microsoft’s Experimentation Platform (ExP).” https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/alerting-in-microsofts-experimentation-platform-exp/.