next up previous
Next: Boltzmann machines I [3 Up: NNA_Exercises_2009 Previous: Decision Boundaries of Backprop

Tangent propagation [2* P]

Consider a multilayer network with arbitrary feed-forward topology, which is to be trained by minimizing the tangent propagation error function

$\displaystyle \tilde{E} = E + \lambda \Omega$ (7)

where $ \lambda$ is a regularization coefficient,

$\displaystyle \Omega = \frac{1}{2}\sum_n \sum_k \left( \sum_{i=1}^D J_{nki}\tau_{ni}\right)^2$ (8)

and $ J_n$ denotes the Jacobian matrix for input $ {\bf x}_n$ . Show that the regularization term $ \Omega$ can be written as a sum over patterns of terms of the form

$\displaystyle \Omega_n = \frac{1}{2} \sum_k \left.\left( \mathcal{G}y_k\right)^2\right\vert _{{\bf x}_n}$ (9)

where $ \mathcal{G}$ is a differential operator defined by

$\displaystyle \mathcal{G} \equiv \sum_i \tau_i \frac{\partial}{\partial x_i}.$ (10)

By acting on the forward propagation equations

$\displaystyle z_j = h(a_j),~~~~~~~~~~ a_j =\sum_i w_{ji}z_i$ (11)

with the operator $ \mathcal{G}$ , show that $ \Omega_n$ can be evaluated by forward propagation using the following equations:

$\displaystyle \alpha_j = h'(a_j)\beta_j,~~~~~~~~~~ \beta_j = \sum_i w_{ji}\alpha_i,$ (12)

where we have defined the new variables

$\displaystyle \alpha_j \equiv \mathcal{G}z_j,~~~~~~~~~~ \beta_j \equiv \mathcal{G}a_j.$ (13)

Now show that the derivatives of $ \Omega_n$ with respect to a weight $ w_{rs}$ in the network can be written in the form

$\displaystyle \frac{\partial\Omega_n}{\partial w_{rs}} = \sum_k \alpha_k \{ \phi_{kr}z_s + \delta_{kr}\alpha_s\}$ (14)

where we have defined

$\displaystyle \delta_{kr} \equiv \frac{\partial y_k}{\partial a_r},~~~~~~~~~~ \phi_{kr} \equiv \mathcal{G}\delta_{kr}.$ (15)

Write down the backpropagation equations for $ \delta_{kr}$ , and hence derive a set of backpropagation equations for the evaluation of the $ \phi_{kr}$ .


next up previous
Next: Boltzmann machines I [3 Up: NNA_Exercises_2009 Previous: Decision Boundaries of Backprop
Haeusler Stefan 2010-01-19