图 书 馆
选择性标签问题:在不可观测的情况下评估算法预测The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables |
|
| 课程网址: | http://videolectures.net/kdd2017_lakkaraju_selective_labels_probl... |
| 主讲教师: | Himabindu Lakkaraju |
| 开课单位: | 视频讲座网 |
| 开课时间: | 2017-10-09 |
| 课程语种: | 英语 |
| 中文简介: |  |
| 课程简介: | Evaluating whether machines improve on human performance is one of the central questions of machine learning. However, there are many domains where the data is {em selectively labeled} in the sense that the observed outcomes are themselves a consequence of the existing choices of the human decision-makers. For instance, in the context of judicial bail decisions, we observe the outcome of whether a defendant fails to return for their court appearance only if the human judge decides to release the defendant on bail. Comparing the performance of humans and machines on data with this type of bias can lead to erroneous estimates and wrong conclusions. Here we propose a novel framework for evaluating the performance of predictive models on selectively labeled data. We develop an evaluation methodology that is robust to the presence of unmeasured confounders (unobservables). We propose a metric that allows us to evaluate the effectiveness of any given black-box predictive model and benchmark it against the performance of human decision-makers. We also develop an approach called emph{contraction} which allows us to compute this metric without resorting to counterfactual inference by exploiting the heterogeneity of human decision-makers. Experimental results on real world datasets spanning diverse domains such as health care, insurance, and criminal justice demonstrate the utility of our evaluation metric in comparing human decisions and machine predictions. Experiments on synthetic data also show that our contraction technique produces accurate estimates of our evaluation metric. |
| 关 键 词: | 评估机器; 机器学习; 标记数据 |
| 课程来源: | 视频讲座网 |
| 数据采集: | 2022-11-30：chenxin01 |
| 最后编审: | 2022-11-30：chenxin01 |
| 阅读次数: | 30 |