Unexpected news events, such as natural disasters or other human tragedies, create a large volume of dynamic text data from official news media as well as less formal social media. Automatic real-time text summarization has become an important tool for quickly transforming this overabundance of text into clear, useful information for end-users including affected individuals, crisis responders, and interested third parties. Despite the importance of real-time summarization systems, their evaluation is not well understood as classic methods for text summarization are inappropriate for real-time and streaming conditions. The TREC 2013-2015 Temporal Summarization (TREC-TS) track was one of the first evaluation campaigns to tackle the challenges of real-time summarization evaluation, introducing new metrics, ground-truth generation methodology and dataset. In this paper, we present a study of TREC-TS track evaluation methodology, with the aim of documenting its design, analyzing its effectiveness, as well as identifying improvements and best practices for the evaluation of temporal summarization systems.
bibtex
Copied!
@inproceedings{ekstrand:ts-eval,
year = {2016},
url = {http://doi.acm.org/10.1145/2983323.2983653},
title = {A Study of Realtime Summarization Metrics},
series = {CIKM '16},
publisher = {ACM},
pages = {2125--2130},
numpages = {6},
location = {Indianapolis, Indiana, USA},
isbn = {978-1-4503-4073-1},
doi = {10.1145/2983323.2983653},
booktitle = {Proceedings of the 25th ACM International on Conference on Information and Knowledge Management},
author = {Ekstrand-Abueg, Matthew and McCreadie, Richard and Pavlu, Virgil and Diaz, Fernando},
address = {New York, NY, USA},
acmid = {2983653}
}