Backpropagation-of-Error Methods

In this chapter we consider iterative algorithms for solving systems of linear equations. If A is an invertible real matrix, B ^?1 is an approximation of A ^?1, and we wish to find x for which A x = b, we could proceed as follows: having obtained an approximate solution x ^k, view B ^?1( b ? A x ^k) as an approximation of the error x ? x ^k = A ^?1( b ? A x ^k) and take as the next approximation

For this to be a practical method, we need B to be a good approximation of A and also to be easily inverted. For a more general M by N matrix A, the system A x = b need not have a solution, in which case we may wish to calculate a least-squares solution. The Landweber algorithm, with iterative step

converges to a least-squares solution for any ? in the interval (0 , 2 /L), where L is the largest eigenvalue of Q = A ^T A. Algorithms of this type are sometimes called backpropagation-of-error methods because the error is transformed by A ^T back into the space of N -dimensional vectors. Throughout this chapter we shall assume that Q is invertible so that the system A x = b has a unique...