Theory Of Cortical Plasticity

To show the explicit dependence on the learning rate, we rewrite the random modification equations in the form:
and the deterministic differential equations,
The convergence of the solution will be shown in two steps; First we show that the solution of the averaged deterministic equation converges, and then we use theorem 3A.1 to show the convergence of the solution of the random differential equation to the solution of its averaged deterministic equation.
The deterministic differential equations represent a negative gradient of the risk. Therefore, in order to show convergence of the solution, we only need to show that the risk is bounded from below. This will assure that the solution converges to a local minimum of the risk.
We can assume that m the synaptic weight vector lies in the space spanned by the random variable d. When we replace the random variable d with a training set d 1, , d n, this assumption says that m ? Span{ d 1, , d n}. This implies that there is a ? > 0, so that ? m Var( m d) ? ?? m? 2 > 0.
To show that the vector
is bounded we assume that none of its components is zero (since zero is definitely bounded), and multiply both sides of the above equation by
, this implies:

which implies that
.