The gradient of sigmoid $$\sigma(x) = \frac{1}{1+e^{-x}}$$ can be expressed as:

Question

MLQuiz · Accepted Answer

The elegant self-referential derivative: $$\sigma'(x) = \sigma(x)(1-\sigma(x))$$. Maximum at x=0 where $$\sigma'(0)=0.25$$. This small max gradient causes the **vanishing gradient problem** in deep networks.