Accession Number:

AD1105415

Title:

Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations

Descriptive Note:

Journal Article - Open Access

Corporate Author:

Institute of High Performance Computing Agency for Science, Technology and Research Connexis North Singapore

Personal Author(s):

• Li,Qianxiao
• Tai,Cheng
• E,Weinan

Report Date:

2019-03-01

Pagination or Media Count:

47.0

Abstract:

We develop the mathematical foundations of the stochastic modified equations SME framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with small noise parameters. We prove that this approximation can be understood mathematically as an weak approximation, which leads to a number of precise and useful results on the approximations of stochastic gradient descent SGD, momentum SGD and stochastic Nesterovs accelerated gradient method in the general setting of stochastic objectives. We also demonstrate through explicit calculations that this continuous-time approach can uncover importantanalytical insights into the stochastic gradient algorithms under consideration that may not be easy to obtain in a purely discrete-time setting.

Descriptors:

• stochastic processes
• differential equations
• algorithms
• momentum
• gradients
• approximations (mathematics)
• Heuristic methods
• theorems
• eigenvalues

Subject Categories:

• Statistics and Probability
• Theoretical Mathematics
• Numerical Mathematics

Distribution Statement:

APPROVED FOR PUBLIC RELEASE