AI Evaluation

This research area focuses on core methods for evaluating AI systems, including development of quantitative metrics, mixed methods approaches that combine qualitative with quantitative insights, and contextual meta-evaluation methods.

Publications

Evaluation of Information Access Agents under Simulated AI Marketplace Dynamics
T.-E. Kim, A. Salemi, H. Zamani, F. Diaz
SIGIR, 2026
Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation
X. He, T.-E. Kim, M. Fröebe, J. Arguello, B. Mitra, F. Diaz
SIGIR, 2026
Diversification as Risk Minimization
R. Takehi, F. Diaz, T. Sakai
WSDM, 2026
Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor
A. Olteanu, S.-L. Blodgett, A. Balayn, A. Wang, F. Diaz, F. du Pin Calmon, M. Mitchell, M. D. Ekstrand, R. Binns, S. Barocas
NeurIPS, 2025
Tip of the Tongue Query Elicitation for Simulated Evaluation
Y. He, T.-E. Kim, F. Diaz, J. Arguello, B. Mitra
SIGIR 2025
Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy
A. Deviyani, F. Diaz
NAACL 2025
Offline Evaluation of Set-Based Text-to-Image Generation
N. Arabzadeh, F. Diaz, J. He
SIGIR-AP 2024
Distributionally-Informed Recommender System Evaluation
M. D. Ekstrand, B. Carterette, F. Diaz
ACM Transactions on Recommender Systems, March 2024
On Evaluating Session-Based Recommendation with Implicit Feedback
F. Diaz
Perspectives on the Evaluation of Recommender Systems Workshop, RecSys 2021
Mixed Method Development of Evaluation Metrics
P. Chandar, F. Diaz, C. Hosey, B. St. Thomas
KDD 2021
The Benchmark Lottery
M. Dehghani, Y. Tay, A. Gritsenko, Z. Zhao, N. Houlsby, F. Diaz, D. Metzler, O. Vinyals
arxiv, 2021
Estimation of Fair Ranking Metrics with Incomplete Judgments
O. Kirnap, F. Diaz, A. J. Biega, M. Ekstrand, B. Carterette, E. Yilmaz
WWW 2021
Beyond Accuracy: Grounding Evaluation Metrics for Human-Machine Learning Systems
P. Chandar, F. Diaz, B. St. Thomas
NeurIPS 2020
Overview of the TREC 2020 Fair Ranking track
A. Biega, F. Diaz, M. D. Ekstrand, S. Kohlmeier
TREC 2020
Evaluating Stochastic Rankings with Expected Exposure
F. Diaz, B. Mitra, M. D. Ekstrand, A. J. Biega, B. Carterette
CIKM 2020
Overview of the TREC 2019 Fair Ranking track
A. Biega, F. Diaz, M. D. Ekstrand, S. Kohlmeier
TREC 2020
Mixed methods for evaluating user satisfaction
J. Garcia-Gathright, C. Hosey, B. St. Thomas, B. Carterette, F. Diaz
RecSys 2018
Understanding and Evaluating User Satisfaction with Music Discovery
J. Garcia-Gathright, B. St. Thomas, C. Hosey, Z. Nazari, F. Diaz
SIGIR 2018
Auditing Search Engines for Differential Performance Across Demographics
R. Mehrotra, A. Anderson, F. Diaz, A. Sharma, H. Wallach, E. Yilmaz
WWW 2017
Overview of the TREC 2016 Real-Time Summarization track
J. Lin, A. Roegiest, L. Tan, R. McCreadie, E. Voorhees, F. Diaz
TREC 2016
A Study of Realtime Summarization Metrics
M. Ekstrand, R. McCreadie, V. Pavlu, F. Diaz
CIKM 2016
Online and social media data as an imperfect continuous panel survey
F. Diaz, M. Gamon, J. Hofman, E. Kiciman, D. Rothschild
PLoS ONE, January 2016
Overview of the TREC 2015 Temporal Summarization track
J Aslam, F Diaz, M Ekstrand-Abueg, R McCreadie, V Pavlu, T Sakai
TREC 2015
Overview of the TREC 2014 Web track
K Collins-Thompson, C Macdonald, P Bennett, F Diaz, E Voorhees
TREC 2014
Overview of the TREC 2014 Temporal Summarization track
J Aslam, F Diaz, M Ekstrand-Abueg, R McCreadie, V Pavlu, T Sakai
TREC 2014
Contextual and Dimensional Relevance Judgments for Reusable SERP-level Evaluation
P. Golbus, I. Zitouni, J. Kim, A. Hassan, F. Diaz
WWW 2014
Overview of the TREC 2013 Web track
K Collins-Thompson, P Bennett, F Diaz, C Clarke, E Voorhees
TREC 2013