Overview

Frequently, the solution of a linear least-squares problem using the normal equations of Section 10.2.2,

(A.1)

fails because a zero pivot occurs in the matrix calculation because C is singular. If the matrix is sufficiently close to singular, the answer becomes extremely sensitive to round-off errors, in which case you typically get fitted A _? s with very large amplitudes that are delicately balanced to almost precisely cancel out. Here is an example of a nearly singular matrix:

(A.2)

This arises when the data do not clearly distinguish between two or more of the basis functions provided.

The solution is to use singular value decomposition (SVD). When some combination of basis functions is irrelevant to the fit, SVD will drive the amplitudes of these basis functions down to small values rather than pushing them up to delicately canceling infinities.

How does SVD work?

First, we need to restate the least-squares problem slightly differently. In least-squares, we want to minimize

(A.3)

where

(A.4)

In matrix form, we can write

(A.5)

where X is the design matrix given by

(A.6)

and

(A.7)

The problem is to find A which minimizes Equation (A.5).

Any rectangular matrix can be written in reduced SVD form as follows ^[1] (see any good linear algebra text for a proof ):

(A.9)

where the columns of U are orthonormal and are the eigenvectors of X X ^T. The columns of V are orthonormal and are the eigenvectors of X ^T