5.2 Principal Component Analysis Based on Covariance Matrices

Dillon and Goldstein (1984) provide the following formal definition of principal components analysis (PCA):

Principal components analysis transforms the original set of variables into a smaller set of linear combinations that account for most of the variance in the original set. The purpose of PCA is to determine factors (i.e., principal components) in order to explain as much of the total variation in the data as possible.

Principal component analysis was developed by Hotelling (1933) after its origination by Pearson (1901). Other significant contributers to the development PCA include Rao (1964), Jolliffe (1986), and Jackson (1991).

Specifically, suppose we have X = ( X ₁, X ₂,..., X _p) ^T being a vector of p multivariate variables distributed as a multivariate normal distribution, that is, X ~ N( ?, ?), where

is the vector of population means, and

is the population covariance matrix.

Because ? is a covariance matrix, it is at least a positive semidefinite matrix. All eigenvalues of ? should be greater or equal to zero. That is, if ? = ( ? ₁, ? ₂,..., ? _k,..., ? _p) ^T is the vector of eigenvalues of ?, where ? ₁ is the largest eigenvalue, ? ₂ is the second largest eigenvalue, etc., then ? ₁ ? ? ₂ = ... ? ? _k =... ? ?