14.1 Steepest, But Slow, Descent

Our approach to iterative methods in Chapter 12 was based, at least implicitly, on dynamical systems. The solution of the linear system

where A is a d d real nonsingular matrix and b ? ? ^d, was formulated as an iterated map

where h : ? ^d ? ? ^d. The convergence of this recursive procedure was a consequence of basic features of the map h: its contractivity (in the spirit of Section 7.1) and fixed points. Indeed, much of the effort required to design, analyse and understand methods of this kind is a reflection of the tension between mathematical attributes of the map h, which ensure convergence to the right limit, and numerical desiderata that each iteration should be cheap and that convergence should occur rapidly.

The basic pattern of one-step stationary iteration (14.2) can be generalized by the inclusion of past values of x ^{[ k ]} or by allowing h to vary. In this chapter we intend to adopt a different point of departure altogether and view the problem from the standpoint of the theory of optimization. The main underlying idea is to restate (14.1) as the minimization of some function f : ? ^d ? ? and apply an optimization algorithm.

Let us assume for the time being that the matrix A in (14.1) is symmetric and positive definite.