图 书 馆
测试集的可重用性度量Measuring the Reusability of Test Collections |
|
| 课程网址: | http://videolectures.net/wsdm2010_carterette_mtro/ |
| 主讲教师: | Ben Carterette |
| 开课单位: | 特拉华大学 |
| 开课时间: | 2010-10-12 |
| 课程语种: | 英语 |
| 中文简介: |  |
| 课程简介: | While test collection construction is a time-consuming and expensive process, the true cost is amortized by reusing the collection over hundreds or thousands of experiments. Some of these experiments may involve systems that retrieve documents not judged during the initial construction phase, and some of these systems may be “hard” to evaluate: depending on which judgments are missing and which judged documents were retrieved, the experimenter’s confidence in an evaluation could potentially be very low. We propose two methods for quantifying the reusability of a test collection for evaluating new systems. The proposed methods provide simple yet highly effective tests for determining whether an existing set of judgments is useful for evaluating a new system. Empirical evaluations using TREC datasets confirm the usefulness of our proposed reusability measures. In particular, we show that our methods can reliably estimate confidence intervals that are indicative of collection reusability. |
| 关 键 词: | 测试集; 统计; 置信度 |
| 课程来源: | 视频讲座网 |
| 最后编审: | 2019-10-24：lxf |
| 阅读次数: | 52 |