The Adam optimizer combines momentum and RMSProp. Its update rule uses $$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$$ and $$v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$$. What are the default values of $$\beta_1$$ and $$\beta_2$$?

Question

Accepted Answer

Default Adam hyperparameters from the original paper (Kingma & Ba, 2014): $$\beta_1=0.9$$ (momentum decay), $$\beta_2=0.999$$ (RMSProp decay), $$\epsilon=10^{-8}$$, $$\alpha=0.001$$.