Knowing what we don’t know in NCAA Football ratings: Understanding and using structured uncertainty

There is a great deal of uncertainty in the skills of teams in NCAA football, which makes ranking teams and choosing postseason matchups difficult. Despite this, standard approaches (e.g., the BCS system) estimate a single ranking of teams and use it to make decisions about postseason matchups. In this work, we argue for embracing uncertainty in rating NCAA football teams. Specifically, we (1) develop a statistical model that infers uncertainty in and correlations between team skills based on game outcomes, (2) make proposals for how to communicate the inferred uncertainty, and (3) show how to make decisions about postseason matchups that are principled in the face of the uncertainty. We apply our method to 14 years of NCAA football data and show that it produces interesting recommendations for postseason matchups, and that there are general lessons to be learned about choosing postseason matchups based on our analysis.