|
高效学习选择抽样的标记源准确性
Efficiently Learning the Accuracy of Labeling Sources for Selective Sampling |
|
| |
| 课程网址: |
http://videolectures.net/kdd09_donmez_elalsss/
|
| 主讲教师: |
Pinar Donmez
|
| 开课单位: |
卡内基梅隆大学
|
| 开课时间: |
信息不详。欢迎您在右侧留言补充。
|
| 课程语种: |
英语
|
| 中文简介: |
许多可扩展的数据挖掘任务依赖于主动学习来提供最有用的、准确标记的实例。但是，如果有多个具有不同但未知可靠性的标记源（“oracle”或“专家”），该怎么办？随着诸如亚马逊的Mechanical Turk等价格低廉且可扩展的在线注释工具的出现，标签制作过程变得更容易受到噪音的影响，而且事先不知道每个贴标机的准确度。本文正是解决了这样一个挑战：如何共同学习标记源的准确性，并为手头的主动学习任务获得最有用的标签，从而最大限度地减少总的标记工作。更具体地说，我们将IETHRESH（区间估计阈值）作为一种策略来智能地选择具有最高估计标记精度的专家。IETHRESH估计每个专家的可靠性的置信区间，并筛选出其估计上限置信区间低于阈值的置信区间——该置信区间共同优化了预期的准确度（平均值），需要更好地估计专家的准确度（方差）。我们的框架足够灵活，能够处理各种不同的噪声级，并且优于基线，例如询问所有可用的专家和随机选择专家。尤其是，IETHRESH实现了给定的准确度，不到所有贴标签专家发出的查询的一半，也不到UCI蘑菇一号等数据集上随机选择专家所需查询的三分之一。结果表明，我们的方法自然地平衡了勘探和开发，因为它获得了哪些专家需要依赖的知识，并以不断增加的频率进行选择。
|
| 课程简介: |
Many scalable data mining tasks rely on active learning to provide the most useful accurately labeled instances. However, what if there are multiple labeling sources (`oracles' or `experts') with different but unknown reliabilities? With the recent advent of inexpensive and scalable online annotation tools, such as Amazon's Mechanical Turk, the labeling process has become more vulnerable to noise - and without prior knowledge of the accuracy of each individual labeler. This paper addresses exactly such a challenge: how to jointly learn the accuracy of labeling sources and obtain the most informative labels for the active learning task at hand minimizing total labeling effort. More specifically, we present IEThresh (Interval Estimate Threshold) as a strategy to intelligently select the expert(s) with the highest estimated labeling accuracy. IEThresh estimates a confidence interval for the reliability of each expert and filters out the one(s) whose estimated upper-bound confidence interval is below a threshold - which jointly optimizes expected accuracy (mean) and need to better estimate the expert's accuracy (variance). Our framework is flexible enough to work with a wide range of different noise levels and outperforms baselines such as asking all available experts and random expert selection. In particular, IEThresh achieves a given level of accuracy with less than half the queries issued by all-experts labeling and less than a third the queries required by random expert selection on datasets such as the UCI mushroom one. The results show that our method naturally balances exploration and exploitation as it gains knowledge of which experts to rely upon, and selects them with increasing frequency.
|
| 关 键 词: |
数据挖掘任务; 主动学习; 在线标注工具; 置信区间估计
|
| 课程来源: |
视频讲座网
|
| 最后编审: |
2019-12-26：cwx
|
| 阅读次数: |
38
|
|
图 书 馆