Revisiting Popularity and Demographic Biases in Recommender Evaluation and Effectiveness

Recommender systems are susceptible to popularity bias and can disproportionately recommend popular items. Groups that are underrepresented in the training data may also receive less relevant recommendations from these algorithms compared to others. Ekstrand et al. investigate how recommender performance varies according to popularity and demographics, and find statistically significant differences in recommendation utility between binary genders on two datasets, and significant effects based on age on one. Here we reproduce those results and extend them with additional analyses. We find statistically significant differences in recommender performance by both age and gender. We observe that recommendation utility steadily degrades for older users, and is lower for women than men. We also find that the utility is higher for users from countries with more representation in the dataset. Total usage and the popularity of consumed content are strong predictors of recommender performance and also vary significantly across demographic groups.