图 书 馆
辍学的PAC贝叶斯分析A PAC-Bayesian Analysis of Dropouts |
|
| 课程网址: | http://videolectures.net/nipsworkshops2013_mcallester_dropouts/ |
| 主讲教师: | David McAllester |
| 开课单位: | 芝加哥丰田技术研究所 |
| 开课时间: | 2014-10-06 |
| 课程语种: | 英语 |
| 中文简介: |  |
| 课程简介: | Intuitively, a neural network that is robust to dropout perturbations should have better generalization properties - it should perform better on novel inputs. Stochastic model perturbation is the fundamental concept underlying PAC-Bayesian generalization theory. This talk will briefly summarize PAC-Bayesian generalization theory and give a regularization bound for a simple form of dropout training as a straightforward application. For a regularization bound involving an L2 penalty for model weights, dropouts reduce the regularization penalty by a factor of 1-alpha where alpha is the dropout rate. The bound then expresses a trade-off between the dropout rate and the training loss. While this regularization bound in intriguing, it may not be the right analysis. An alternative analysis involves variance reduction - the standard motivation for bagging. There are good reasons to believe that a certain general PAC-Bayes variance bound is significantly tighter than the general PAC-Bayes regularization bound. Unfortunately the variance bound is opaque - it does not involve explicit regularization and is difficult to compare with regularization bounds. Also, unlike regularization bounds, there is no obvious method for designing algorithms that minimize the variance bound. A compelling variance-based PAC-Bayesian analysis of dropouts remains an open problem. |
| 关 键 词: | 随机模型; 贝叶斯泛化理论; 最小化方差 |
| 课程来源: | 视频讲座网 |
| 数据采集: | 2022-11-08：chenjy |
| 最后编审: | 2022-11-08：chenjy |
| 阅读次数: | 43 |