Изменения

* Стохастический градиентный спуск [http://www.machinelearning.ru/wiki/index.php?title=%D0%9C%D0%B5%D1%82%D0%BE%D0%B4_%D1%81%D1%82%D0%BE%D1%85%D0%B0%D1%81%D1%82%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%B3%D1%80%D0%B0%D0%B4%D0%B8%D0%B5%D0%BD%D1%82%D0%B0]: <tex>w^{(0)}</tex> - начальные весы сети, <tex>w^{(k+1)}=w^{(k)}-\mu\frac{\partial L(w^{(k)})}{\partial w}</tex>;

* Momentum [https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Momentum]: <tex> \Delta w:=\alpha \Delta w-\eta \nabla Q_{i}(w)</tex>, <tex> w:=w+\Delta w</tex> или <tex> w:=w-\eta \nabla Q_{i}(w)+\alpha \Delta w</tex>;

* NAG (Nesterov accelerated gradient)[https://jlmelville.github.io/mize/nesterov.html#nag], : <tex> w^{(k+1)} = w^{(k)}-v^{(k)}; v^{(k+1)}=\gamma v^{(k)}+\mu\frac{\partial L(w^{(k)}-v^{(k)})}{\partial w}</tex> *Adagrad [http://akyrillidis.github.io/notes/AdaGrad]: <tex>g_{i,(k)}=\frac{\partial L(w_i^{(k)})}{\partial w_i}, w_i^{(k+1)}=w_i^{(k)}-\frac{\mu}{\sqrt{G^{(k)}_i, i+\epsilon}}g_{i,(k)}</tex>, где G - диагональная матрица, элементы которой, суммы квадратов координат градиента к k-ой итерации алгоритма; *RMSProp[https://towardsdatascience.com/a-look-at-gradient-descent-and-rmsprop-optimizers-f77d483ef08b]: <tex>E^{(k)}[g_i^2] = \gamma E^{(k-1)}[g_i^2]+(1-\gamma)g^2_{i, (k)}, w^{(k+1)}_i = w_i^{(k)}-\frac{\mu}{\sqrt{E^{(k)}[g_i^2]+\epsilon}}g_{i, (k)}</tex>;  *Adadelta[https://arxiv.org/abs/1212.5701]:<tex>w^{(k+1)}=w^{(k)}-\mu(Q''(w^{(k)})^{-1}Q'(w^{(k)}</tex>, вычисление матрицы Q вторых производных довольно сложная задача, поэтому вместо неё можно брать приближение: <tex>w^{(k+1)}=w^{(k)}-\frac{RMS^{(k-1)}[\delta w_i}{RMS^{(k)}[g_i]}g_i^{(k)}</tex>, где <tex>RMS^{(k)}[x_i]=\sqrt{E^{(k)}[x^2_i]+\epsilon}</tex>; *Adam[https://arxiv.org/pdf/1412.6980.pdf]..: <tex> w^{(k+1)}_i = w_i^{(k)}-\frac{\mu}{\sqrt{\hat{b}^2_{(k)}+\epsilon}}\hat{m}_{(k)}</tex>, где <tex>\hat{m}_{(k)}=\frac{\gamma_1 E^{(k-1)}[g_i]+(1-\gamma_1)g_{i,(k)}}{1-\gamma_1^k}</tex> и <tex>\hat{b}^2_{(k)}= \frac{\gamma_2 E^{(k-1)}[g^2_i]+(1-\gamma_2)g:2_{i,(k)}}{1-\gamma_2^k}</tex>.