AI Evaluation

This research area focuses on core methods for evaluating AI systems, including development of quantitative metrics, mixed methods approaches that combine qualitative with quantitative insights, and contextual meta-evaluation methods.

Publications

Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
A. Olteanu, S.-L. Blodgett, A. Balayn, A. Wang, F. Diaz, F. du Pin Calmon, M. Mitchell, M. D. Ekstrand, R. Binns, S. Barocas
arxiv, 2025
Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy
A. Deviyani, F. Diaz
NAACL 2025
Offline Evaluation of Set-Based Text-to-Image Generation
N. Arabzadeh, F. Diaz, J. He
SIGIR-AP 2024
Distributionally-Informed Recommender System Evaluation
M. D. Ekstrand, B. Carterette, F. Diaz
ACM Transactions on Recommender Systems, March 2024
On Evaluating Session-Based Recommendation with Implicit Feedback
F. Diaz
Perspectives on the Evaluation of Recommender Systems Workshop, RecSys 2021
Mixed Method Development of Evaluation Metrics
P. Chandar, F. Diaz, C. Hosey, B. St. Thomas
KDD 2021
Estimation of Fair Ranking Metrics with Incomplete Judgments
O. Kirnap, F. Diaz, A. J. Biega, M. Ekstrand, B. Carterette, E. Yilmaz
WWW 2021
Beyond Accuracy: Grounding Evaluation Metrics for Human-Machine Learning Systems
P. Chandar, F. Diaz, B. St. Thomas
NeurIPS 2020
Evaluating Stochastic Rankings with Expected Exposure
F. Diaz, B. Mitra, M. D. Ekstrand, A. J. Biega, B. Carterette
CIKM 2020
Mixed methods for evaluating user satisfaction
J. Garcia-Gathright, C. Hosey, B. St. Thomas, B. Carterette, F. Diaz
RecSys 2018
Understanding and Evaluating User Satisfaction with Music Discovery
J. Garcia-Gathright, B. St. Thomas, C. Hosey, Z. Nazari, F. Diaz
SIGIR 2018
Auditing Search Engines for Differential Performance Across Demographics
R. Mehrotra, A. Anderson, F. Diaz, A. Sharma, H. Wallach, E. Yilmaz
WWW 2017
A Study of Realtime Summarization Metrics
M. Ekstrand, R. McCreadie, V. Pavlu, F. Diaz
CIKM 2016
Online and social media data as an imperfect continuous panel survey
F. Diaz, M. Gamon, J. Hofman, E. Kiciman, D. Rothschild
PLoS ONE, January 2016
Contextual and Dimensional Relevance Judgments for Reusable SERP-level Evaluation
P. Golbus, I. Zitouni, J. Kim, A. Hassan, F. Diaz
WWW 2014