Backpropagation

A simple example

$f(x,y,z) = (x + y)z$

computational graph

$q = x + y$

$f = qz$

${\partial q \over \partial x} = 1, {\partial q \over \partial y} = 1$

${\partial f \over \partial q} = z, {\partial f \over \partial z} = q$

• Chain rule

  ${\partial f \over \partial x} = {\partial f \over \partial q}{\partial q \over \partial x}$

  ${\partial f \over \partial y} = {\partial f \over \partial q}{\partial q \over \partial y}$

  Upstream gradient * Local gradient

ex) x = -2, y = 5, z = -4

${\partial f \over \partial f}$부터 $\partial f \over \partial z$, $\partial f \over \partial q$, $\partial f \over \partial x$, $\partial f \over \partial y$를 구한다

Backpropagation ⇒ Backward 방향으로 loss 전달, gradient 계산

Another example

$f(w, x) = {1 \over 1 + e^{-(w_0x_0 + w_1x_1 + w_2)}}$

⇒ sigmoid function의 형태

${1\over {1 + e^{-1}}} = {(1 + e^{-1})}^{-1}$

ex)

주요 함수들의 local gradients

• $f(x) = e^x$
• $f(x) = ax$
• $f(x) = {1 \over x}$
• $f_e(x) = e + x$

${df \over dx} = e^x$

${df \over dx} = a$

${df \over dx} = -{1 \over x^2}$

${df \over dx} = 1$