What is the vanishing gradient problem?

Question

Accepted Answer

In deep networks, gradients shrink as they propagate back through many layers via the chain rule: $$\frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial W_n} \cdot \prod_{i=1}^{n}\frac{\partial h_i}{\partial h_{i-1}}$$. With sigmoid activations, this product approaches 0. ReLU and residual connections help.