图 书 馆
一般众包任务的统计质量估计Statistical Quality Estimation for General Crowdsourcing Tasks |
|
| 课程网址: | http://videolectures.net/kdd2013_baba_crowdsourcing_tasks/ |
| 主讲教师: | Yukino Baba |
| 开课单位: | 东京大学 |
| 开课时间: | 2013-09-27 |
| 课程语种: | 英语 |
| 中文简介: |  众包的请求者和平台提供者面临的最大挑战之一是质量控制，这是指期望既不一定很有能力也没有动力的人群工作者获得高质量的结果。解决此问题的常用方法是引入冗余，即要求多个工作人员从事相同的任务。对于简单的多项选择任务，已经提出了几种统计方法来汇总多个答案。但是，这些方法不能始终应用于具有非结构化响应格式的更一般的任务，例如文章编写，程序编码和徽标设计，这些格式在大多数众包市场中都占多数。在本文中，我们针对此类一般众包任务提出了一种无监督的统计质量估计方法。我们的方法基于两个阶段的过程。首先，要求多个工作人员在创建阶段中从事相同的任务，然后在检查阶段中，另一组工作人员对每个工件进行审查和评分。我们对每个作者的能力和每个审阅者的偏见进行建模，并使用项目响应理论中的分级响应模型提出一个两阶段概率生成模型。使用几个常规众包任务的实验表明，该方法优于流行的投票汇总方法，这表明我们的方法可以以较低的成本提供高质量的结果。 |
| 课程简介: | One of the biggest challenges for requesters and platform providers of crowdsourcing is quality control, which is to expect high-quality results from crowd workers who are neither necessarily very capable nor motivated. A common approach to tackle this problem is to introduce redundancy, that is, to request multiple workers to work on the same tasks. For simple multiple-choice tasks, several statistical methods to aggregate the multiple answers have been proposed. However, these methods cannot always be applied to more general tasks with unstructured response formats such as article writing, program coding, and logo designing, which occupy the majority on most crowdsourcing marketplaces. In this paper, we propose an unsupervised statistical quality estimation method for such general crowdsourcing tasks. Our method is based on the two-stage procedure; multiple workers are first requested to work on the same tasks in the creation stage, and then another set of workers review and grade each artifact in the review stage. We model the ability of each author and the bias of each reviewer, and propose a two-stage probabilistic generative model using the graded response model in the item response theory. Experiments using several general crowdsourcing tasks show that our method outperforms popular vote aggregation methods, which implies that our method can deliver high quality results with lower costs. |
| 关 键 词: | 质量估计; 模型生成 |
| 课程来源: | 视频讲座网 |
| 数据采集: | 2020-12-16：zyk |
| 最后编审: | 2020-12-16：zyk |
| 阅读次数: | 50 |