4.4 Back Propagation Algorithm and Its Variants

One of the most popular algorithms for neural network training is back propagation (BP), proposed by Rumelhart, Hinton, and Williams in 1986 [9]. The basic concept of back propagation training was discussed in Chapter 3, when we introduced MLP. In this section, variants of BP are presented. Back propagation is a stochastic algorithm based upon the steepest descent principle [10], in which the weights of the neural network are updated along the negative gradient direction in the weight space. The update formulae are given by

(4.32)

(4.33)

The basic back propagation algorithm suffers from slower convergence and possible weight oscillation. Sample-by-sample training in which E _{T r} and are computed with one data sample at a time leads to a stochastic process. Since may change between samples, w can also oscillate, since the present and past updates of w may partially cancel each other out. To remedy such a situation, we can keep ? small and add a momentum term. By keeping ? small, the training process tends to use more epochs and the updating of w becomes...