Theory Of Cortical Plasticity

Appendix 3A: Convergence of the Solution of the Random Differential Equations

Overview

To show the explicit dependence on the learning rate, we rewrite the random modification equations in the form:

and the deterministic differential equations,

The convergence of the solution will be shown in two steps; First we show that the solution of the averaged deterministic equation converges, and then we use theorem 3A.1 to show the convergence of the solution of the random differential equation to the solution of its averaged deterministic equation.

3A.1 Convergence of the Deterministic Equation

The deterministic differential equations represent a negative gradient of the risk. Therefore, in order to show convergence of the solution, we only need to show that the risk is bounded from below. This will assure that the solution converges to a local minimum of the risk.

We can assume that m the synaptic weight vector lies in the space spanned by the random variable d. When we replace the random variable d with a training set d 1, , d n, this assumption says that m ? Span{ d 1, , d n}. This implies that there is a ? > 0, so that ? m Var( m d) ? ?? m? 2 > 0.

To show that the vector is bounded we assume that none of its components is zero (since zero is definitely bounded), and multiply both sides of the above equation by , this implies:

which implies that .

UNLIMITED FREE
ACCESS
TO THE WORLD'S BEST IDEAS

SUBMIT
Already a GlobalSpec user? Log in.

This is embarrasing...

An error occurred while processing the form. Please try again in a few minutes.

Customize Your GlobalSpec Experience

Category: IC Electronic Filters
Finish!
Privacy Policy

This is embarrasing...

An error occurred while processing the form. Please try again in a few minutes.