This is a lecture-based course intended for first or second year graduate students; advanced undergraduates with permission of instructor. Material will be split into two parts—measurement and experimentation—and in two contexts—offline and online. Measurement will cover measurement modeling, core constructs of interest in NLP tasks, and a series of deep dives into NLP-specific classes of metrics—sequences, rankings, and multi-turn interaction. Online metrics are mature in IR but there is sufficient literature in the broader NLP community to support studying implicit metrics there. Experimentation will cover the statistical analysis of measured properties of NLP systems. The offline experimentation component will cover basic principles of data set construction and hypothesis testing. The online experimentation component will cover basic principles in A/B testing and hypothesis testing there. This course will help students interrogate existing evaluation methods, laying a foundation for the development of new evaluation methods.

• Be able to select or design an appropriate evaluation measure for a task in both offline and online contexts.
• Be able to test hypotheses about relative performance of systems in both offline and online contexts.

One course in applied ML (e.g., 11667: Large Language Models Methods and Application, 11737 Multilingual Natural Language Processing, 11742 Search Engines, 11751 Speech Recognition and Understanding, 11777 Multimodal Machine Learning, 11796 Question Answering).

The course will be lecture and discussion-based.

Projects will focus on a deep understanding of the measurement and evaluation of a specific evaluation domain, a relevant property, and analysis of metrics. Each homework will include a project-oriented question, making the final report and presentation more tractable. Projects will be evaluated according to clearly stated assumptions both in the metrics as well as analyses. With the permission of all instructors, you may evaluate the performance of an algorithm-oriented project in another course; in this case, the algorithmic and evaluation work should decouple and be assessable independently.

Each homework will focus on questions of measurement with respect to project metrics, building toward a complete profile of the metric by the end of the semester. Homeworks will be due and released at the beginning of class.

Each student will be assigned one lecture to act as scribe and prepare notes to share with the rest of the class two weeks after the lecture. Notes should be written in a prose style and will be evaluated for completeness, including content and references.

1: Introduction

• overview of quantitative evaluation
• course structure
• expectations

Chapter 1

2: Introduction to measurement

• basics of AI system decisions
• evaluation constructs
• metric properties

Chapter 2

3: Evaluating evaluation metrics

• validity
• reliability
• sensitivity

Chapter 3

4: Foundations and basics of offline evaluation

• common evaluation criteria in offline evaluation
• basic offline metrics

Chapter 4

5: Foundations and basics of online evaluation

• common evaluation criteria in online evaluation
• basic online metrics

Chapter 5

6: Elicitation of evaluation data

• rating guidelines
• types of raters
• types of labels

Chapter 6

7: Foundations and basic constructs of simulated evaluation

• LLM-as-a-judge
• simulated populations

8: The Benchmark Problem (Ben Recht seminar)

9: Evaluating sequences (offline)

• hand-crafted metrics
• learned metrics

Chapter 7

10: Evaluating sequences (online)

• behavioral signals for sequences

Chapter 8

11: Evaluating rankings (offline)

• hand-crafted utility metrics
• diversity metrics
• fairness metrics
• ranked list preference

Chapter 9

12: Evaluating rankings (online)

• behavioral signals for rankings
• session metrics
• user feedback models
• interleaving

Chapter 10

13: Evaluating multi-turn systems (offline)

• turn-level evaluation
• dialog-level evaluation
• reference-free metrics

Chapter 11

14: Evaluating multi-turn systems (online)

• conversation properties
• crowd labeling
• implicit labels

Chapter 12

15: Multi-task evaluation

• multi-task evaluation benchmarks
• combining multiple evaluations

Chapter 13

16: Evaluation Harnesses and Tools (Thomas Serban von Davier)

17: Population-level metrics

• disaggregated evaluation
• quality of service harms
• worst-case evaluation

Chapter 14

18: From measurement to experimentation

• summarizing measurement
• comparing systems

Chapter 15

19: Dataset construction (offline)

• selecting contexts
• selecting decisions
• validating datasets

Chapter 16

20: Dataset construction (online)

• traffic splitting
• randomization units
• A/A tests

Chapter 17

21: Hypothesis testing (offline)

• testing statistical significance
• fundamentals of significance testing
• applications of significance testing

Chapter 18

22: Hypothesis testing (online)

• fixed horizon testing
• error rates
• sequential testing
• segmentation
• internal and external validity

Chapter 19

23: Efficient data collection (offline)

• active learning for evaluation
• metric decomposition and estimation

Chapter 20

24: Efficient data collection (online)

• off-policy evaluation

Chapter 21

Project Presentations 1

Project Presentations 2