WeChat Mini Program
Old Version Features

Convergence and Dynamical Behavior of the ADAM Algorithm for Nonconvex Stochastic Optimization

SIAM Journal on Optimization(2021)

Inst Polytech Paris

Cited 74|Views37
Abstract
Adam is a popular variant of stochastic gradient descent forfinding a local minimizer of a function. In the constant stepsize regime,assuming that the objective function is differentiable and nonconvex, weestablish the convergence in the long run of the iterates to a stationarypoint under a stability condition. The key ingredient is the introductionof a continuous-time version of Adam, under the form of a nonautonomousordinary differential equation. This continuous-time system is a relevantapproximation of the Adam iterates, in the sense that the interpolatedAdam process converges weakly toward the solution to the ODE. Theexistence and the uniqueness of the solution are established. We furthershow the convergence of the solution toward the critical points of theobjective function and quantify its convergence rate under a Łojasiewicz assumption.Then, we introduce a novel decreasing stepsize version of Adam.Under mild assumptions, it is shown that the iterates are almost surely boundedand converge almost surely to critical points of the objective function. Finally,we analyze the fluctuations of the algorithm by means of a conditional central limit theorem.
More
Translated text
Key words
stochastic approximation,dynamical systems,adaptive gradient methods
PDF
Bibtex
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Data Disclaimer
The page data are from open Internet sources, cooperative publishers and automatic analysis results through AI technology. We do not make any commitments and guarantees for the validity, accuracy, correctness, reliability, completeness and timeliness of the page data. If you have any questions, please contact us by email: report@aminer.cn
Chat Paper
Summary is being generated by the instructions you defined