Projections, Projection Matrices, and Idempotent Matrices
Consider the following problem: Which point on the line through the origin and point a in Figure I.3 is the closest to point b? The answer is point p in Figure
I.4. The line through b and p is perpendicular to the subspace spanned by a, and therefore the distance between b and any other point in this subspace is larger than the distance between b and p. Point p is called the projection of b on the subspace spanned by a. To find p, let p = c ■ a, where c is a scalar. The distance between b and p is now ||b – c ■ a\; consequently, the problem is to find the scalar c that minimizes this distance. Because ||b – c ■ a\ is minimal if and only if
||b — c ■ a||2 = (b — c ■ a)T(b — c ■ a) = bTb — 2c ■ aTb + c2aTa
is minimal, the answer is c = aTb/aTa; hence, p = (aTb/aTa) ■ a.
Similarly, we can project a vector y in Rn on the subspace of Rn spanned by a basis [x,xk} as follows. Let X be the n x к matrix with columns
x1,…,xk. Any point p in the column space R(X) of X can be written as p = Xb, where b є Rk. Then the squared distance between y and p = Xb is
IIy – XbII2 = (y – Xb)T(y – Xb)
= yTy – bTXTy – yTXb + bTXTXb = yTy – 2bTXTy + bTXTXb, (I.39)
where the last equality follows because yTXb is a scalar (or, equivalently, a 1 x 1 matrix); hence, yTXb = (yTXb)T = bTXTy. Given Xand y, (I.39) is a quadratic function of b. The first-order condition for a minimum of (I.39) is given by
2 XTy + 2 XT Xb = 0,
which has the solution
b = (XT X)-1XTy.
Thus, the vectorp in R(X) closest to y is
p = X(XT X)-1XTy, (I.40)
which is the projection of y on R(X).
Matrices of the type in (I.40) are called projection matrices:
Definition I.11: Let A be an n x k matrix with rankk. Then the n x n matrix P = A( AT A)-1 AT is called a projection matrix: For each vector x in Rn, Px is the projection ofx on the column space of A.
Note that this matrix P is such that PP = A(ATA)-1 ATA(ATA)-1 AT) = A(ATA)-1 AT = P. This is not surprising, though, because p = Px is already in R(A); hence, the point in R(A) closest to p is p itself.
Definition I.12: Ann x n matrix M is called idempotent if MM = M.
Thus, projection matrices are idempotent.