图 书 馆
在网络上运行控制实验时要避免的七个陷阱Seven Pitfalls to Avoid when Running Controlled Experiments on the Web |
|
| 课程网址: | http://videolectures.net/kdd09_crook_spawrcew/ |
| 主讲教师: | Thomas Crook |
| 开课单位: | 认知实验室 |
| 开课时间: | 2009-09-14 |
| 课程语种: | 英语 |
| 中文简介: |  |
| 课程简介: | Controlled experiments, also called randomized experiments and A/B tests, have had a profound influence on multiple fields, including medicine, agriculture, manufacturing, and advertising. While the theoretical aspects of offline controlled experiments have been well studied and documented, the practical aspects of running them in online settings, such as web sites and services, are still being developed. As the usage of controlled experiments grows in these online settings, it is becoming more important to understand the opportunities and pitfalls one might face when using them in practice. A survey of online controlled experiments and lessons learned were previously documented in Controlled Experiments on the Web: Survey and Practical Guide (Kohavi, et al., 2009). In this follow-on paper, we focus on pitfalls we have seen after running numerous experiments at Microsoft. The pitfalls include a wide range of topics, such as assuming that common statistical formulas used to calculate standard deviation and statistical power can be applied and ignoring robots in analysis (a problem unique to online settings). Online experiments allow for techniques like gradual ramp-up of treatments to avoid the possibility of exposing many customers to a bad (e.g., buggy) Treatment. With that ability, we discovered that its easy to incorrectly identify the winning Treatment because of Simpsons paradox. |
| 关 键 词: | 控制实验; 网络调查; 实际指导; 统计公式 |
| 课程来源: | 视频讲座网 |
| 最后编审: | 2020-06-15：heyf |
| 阅读次数: | 35 |