图 书 馆
加速随机梯度下降Speeding Up Stochastic Gradient Descent |
|
| 课程网址: | http://videolectures.net/eml07_bengio_ssg/ |
| 主讲教师: | Yoshua Bengio |
| 开课单位: | 蒙特利尔大学 |
| 开课时间: | 2007-12-29 |
| 课程语种: | 英语 |
| 中文简介: |  |
| 课程简介: | n order to tackle large-scale learning problems whose solution necessarily involves a large model with many tunable parameters, difficult non-convex optimization has to be performed efficiently. Computational complexity arguments strongly suggest that deep architectures will be necessary to represent the kind of complex functions that AI involves. Unfortunately, this involves difficult optimization problems and efficient approximate iterative optimization becomes key to obtain good generalization, and not so much the regularization techniques that have been so well studied in the last two decades. Furthermore, because of the size of the data sets involved in such tasks, it is imperative that computation scale no more than linearly with respect to the number of training examples. In many cases, the algorithm to beat is stochastic gradient descent, and the comparisons have to be made by looking at the curve of test error versus computation time. Following recent interest in online versions of second-order optimization methods, we present computational tricks that yield a linear time variant of natural gradient optimization. Another issue, that is particularly difficult to address in the optimization of multi-layer neural networks, is how to parallelize efficiently. SMP machines becoming cheaper and easier to use, we compare and discuss different strategies for exploiting parallelization of training for multi-layer neural networks, showing that naive approaches fail but those taking into account the communication bottleneck yield impressive speed-ups. |
| 关 键 词: | 大型模型; 迭代优化; 随机梯度 |
| 课程来源: | 视频讲座网 |
| 最后编审: | 2019-04-10：lxf |
| 阅读次数: | 77 |