图 书 馆
肮脏的一打：在线控制实验中十二个常见的度量解释陷阱A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments |
|
| 课程网址: | http://videolectures.net/kdd2017_gupta_dirty_dozen/ |
| 主讲教师: | Somit Gupta |
| 开课单位: | 微软 |
| 开课时间: | 2017-10-09 |
| 课程语种: | 英语 |
| 中文简介: |  |
| 课程简介: | Online controlled experiments (e.g., A/B tests) are now regularly used to guide product development and accelerate innovation in software. Product ideas are evaluated as scientific hypotheses, and tested in web sites, mobile applications, desktop applications, services, and operating systems. One of the key challenges for organizations that run controlled experiments is to come up with the right set of metrics. Having good metrics, however, is not enough. In our experience of running thousands of experiments with many teams across Microsoft, we observed again and again how incorrect interpretations of metric movements may lead to wrong conclusions about the experiment’s outcome, which if deployed could hurt the business by millions of dollars. Inspired by Steven Goodman’s twelve p-value misconceptions [4], in this paper, we share twelve common metric interpretation pitfalls which we observed repeatedly in our experiments. We illustrate each pitfall with a puzzling example from a real experiment, and describe processes, metric design principles, and guidelines that can be used to detect and avoid the pitfall. With this paper, we aim to increase the experimenters’ awareness of metric interpretation issues, leading to improved quality and trustworthiness of experiment results and better data-driven decisions. |
| 关 键 词: | 在线控制; 产品创意; 科学假设 |
| 课程来源: | 视频讲座网 |
| 数据采集: | 2023-03-27：chenxin01 |
| 最后编审: | 2023-05-22：chenxin01 |
| 阅读次数: | 32 |