Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent
Author(s)Nicol N. Schraudolph
Contributor(s)The Pennsylvania State University CiteSeerX Archives
Full recordShow full item record
AbstractWe propose a generic method for iteratively approximating various second-order gradient steps -- Newton, Gauss-Newton, Levenberg-Marquardt, and natural gradient -- in linear time per iteration, using special curvature matrix-vector products that can be computed in O(n). Two recent acceleration techniques for online learning, matrix momentum and stochastic meta-descent (SMD), in fact implement this approach. Since both were originally derived by very different routes, this o ers fresh insight into their operation, resulting in further improvements to SMD.