6.3 Orthogonal projections

Big idea

Projection onto a line took a vector apart into a piece along one direction and a piece perpendicular to it. Replace the line by any subspace and the same split exists, with the same formula run once per basis direction.

Two facts make it worth the name. The split is unique: there is exactly one way to write a vector as something in the subspace plus something orthogonal to the subspace. And the piece inside the subspace is the closest point of that subspace to the original vector, closer than any other point in it.

That second fact is what turns projection into an approximation tool. If a vector is out of reach because it sits outside a subspace, the projection is the best available stand-in, and the distance to it measures how far out of reach the original was. The rest of the chapter uses nothing else.

Decoder

The orthogonal decomposition of a vector with respect to a subspace is unique, and the component in the subspace is independent of the orthogonal basis used to compute it.

Two separate claims. First: one vector, one subspace, exactly one way to split into “inside” plus “perpendicular”. Second: the formula uses a basis, but the answer does not depend on which orthogonal basis you fed it. Pick a different orthogonal basis for the same subspace and every intermediate number changes while the final vector does not.

That matters in practice because the next section manufactures orthogonal bases, and you need to know that a different starting order does not produce a different projection.

Definitions and results

The decomposition. Let $W$ be a subspace of $\mathbb{R}^n$. Every $\mathbf{y}$ in $\mathbb{R}^n$ can be written in exactly one way as

$$ \mathbf{y} = \hat{\mathbf{y}} + \mathbf{z} $$

with $\hat{\mathbf{y}}$ in $W$ and $\mathbf{z}$ in $W^{\perp}$. The vector $\hat{\mathbf{y}}$ is the orthogonal projection of $\mathbf{y}$ onto $W$.

The formula. If $\{\mathbf{u}_1, \dots, \mathbf{u}_p\}$ is an orthogonal basis for $W$, then

$$ \hat{\mathbf{y}} = \frac{\mathbf{y} \cdot \mathbf{u}_1}{\mathbf{u}_1 \cdot \mathbf{u}_1}\mathbf{u}_1 + \cdots + \frac{\mathbf{y} \cdot \mathbf{u}_p}{\mathbf{u}_p \cdot \mathbf{u}_p}\mathbf{u}_p $$

One line-projection per basis vector, summed. The basis must be orthogonal for this to be valid; with a general basis the terms interfere and the sum is wrong.

Already inside. If $\mathbf{y}$ is already in $W$, the projection returns $\mathbf{y}$ itself and $\mathbf{z} = \mathbf{0}$. Projecting twice is the same as projecting once.

Best approximation. For $\mathbf{y}$ in $\mathbb{R}^n$ and $W$ a subspace, the projection $\hat{\mathbf{y}}$ is the point of $W$ closest to $\mathbf{y}$: for every other $\mathbf{w}$ in $W$,

$$ \|\mathbf{y} - \hat{\mathbf{y}}\| < \|\mathbf{y} - \mathbf{w}\| $$

The reason is the right triangle. Write $\mathbf{y} - \mathbf{w} = (\mathbf{y} - \hat{\mathbf{y}}) + (\hat{\mathbf{y}} - \mathbf{w})$. The first piece is in $W^{\perp}$, the second is in $W$, so they are orthogonal and the Pythagorean identity applies. The squared distance to $\mathbf{w}$ is the squared distance to $\hat{\mathbf{y}}$ plus a nonnegative extra term, which is zero only when $\mathbf{w} = \hat{\mathbf{y}}$.

Distance to a subspace. The number $|\mathbf{z}| = |\mathbf{y} - \hat{\mathbf{y}}|$ is the distance from $\mathbf{y}$ to $W$. It is zero exactly when $\mathbf{y}$ lies in $W$.

Orthonormal version. If the basis is orthonormal, every denominator is $1$ and the formula becomes $\hat{\mathbf{y}} = (\mathbf{y} \cdot \mathbf{u}_1) \mathbf{u}_1 + \cdots + (\mathbf{y} \cdot \mathbf{u}_p)\mathbf{u}_p$. Collecting those columns into a matrix $U$ gives the compact form $\hat{\mathbf{y}} = UU^T\mathbf{y}$. Read the two products in order: $U^T$ computes the coordinates, $U$ rebuilds the vector from them. Note the contrast with $U^TU = I_p$ from the previous section; $UU^T$ is $n \times n$ and is the identity only when the columns span all of $\mathbb{R}^n$.

Two extreme cases. If $W$ is all of $\mathbb{R}^n$, then every vector is already inside and the projection is the identity. If $W$ is the zero subspace, every projection is the zero vector and the distance to $W$ is just $\|\mathbf{y}\|$. Both agree with the formula, which has $n$ terms in the first case and none in the second.

Dimensions add up. Since every vector splits into a part in $W$ and a part in $W^{\perp}$, and the only vector in both is zero, the two subspaces satisfy $\dim W + \dim W^{\perp} = n$. That gives a quick sanity check on any projection setup: a plane in $\mathbb{R}^3$ has a line as its complement, so the orthogonal part of every projection onto that plane is a multiple of one fixed vector.

Worked examples

Projecting onto a plane

Let $W = \text{Span}\{\mathbf{w}_1, \mathbf{w}_2\}$ with

$$ \mathbf{w}_1 = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}, \qquad \mathbf{w}_2 = \begin{bmatrix} 1 \\ -1 \\ 1 \end{bmatrix} $$

These are orthogonal, since $1 - 1 + 0 = 0$, so the projection formula applies directly. Project $\mathbf{y} = (3, 1, 4)$.

Square lengths: $\mathbf{w}_1 \cdot \mathbf{w}_1 = 2$ and $\mathbf{w}_2 \cdot \mathbf{w}_2 = 3$. Numerators: $\mathbf{y} \cdot \mathbf{w}_1 = 3 + 1 + 0 = 4$ and $\mathbf{y} \cdot \mathbf{w}_2 = 3 - 1 + 4 = 6$. So the weights are $4/2 = 2$ and $6/3 = 2$, and

$$ \hat{\mathbf{y}} = 2\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} + 2\begin{bmatrix} 1 \\ -1 \\ 1 \end{bmatrix} = \begin{bmatrix} 4 \\ 0 \\ 2 \end{bmatrix}, \qquad \mathbf{z} = \begin{bmatrix} 3 \\ 1 \\ 4 \end{bmatrix} - \begin{bmatrix} 4 \\ 0 \\ 2 \end{bmatrix} = \begin{bmatrix} -1 \\ 1 \\ 2 \end{bmatrix} $$

Check that $\mathbf{z}$ is in $W^{\perp}$: $\mathbf{z} \cdot \mathbf{w}_1 = -1 + 1 + 0 = 0$ and $\mathbf{z} \cdot \mathbf{w}_2 = -1 - 1 + 2 = 0$. Both zero, so the split is the right one. The distance from $\mathbf{y}$ to the plane is $\|\mathbf{z}\| = \sqrt{1 + 1 + 4} = \sqrt{6}$, about $2.449$.

Testing the best-approximation claim

Keep $W$ and $\mathbf{y}$ from above and try a different point of $W$. Take $\mathbf{w} = \hat{\mathbf{y}} + \mathbf{w}_1 = (5, 1, 2)$, which is in $W$ because it is a combination of the two spanning vectors. Then $\mathbf{y} - \mathbf{w} = (-2, 0, 2)$, with length $\sqrt{8}$, about $2.828$.

That is larger than $\sqrt{6}$, as the best-approximation result requires. The Pythagorean bookkeeping shows exactly where the extra came from: $8 = 6 + 2$, and $2 = \|\mathbf{w}_1\|^2$ is the squared distance from $\mathbf{w}$ back to $\hat{\mathbf{y}}$ inside the plane.

The orthonormal shortcut

Use the orthonormal pair

$$ \mathbf{e}_1 = \frac{1}{3}\begin{bmatrix} 2 \\ 1 \\ 2 \end{bmatrix}, \qquad \mathbf{e}_2 = \frac{1}{3}\begin{bmatrix} 1 \\ 2 \\ -2 \end{bmatrix} $$

and project $\mathbf{y} = (9, 0, 0)$ onto their span. With unit vectors the coordinates are just dot products: $\mathbf{y} \cdot \mathbf{e}_1 = 18/3 = 6$ and $\mathbf{y} \cdot \mathbf{e}_2 = 9/3 = 3$. So

$$ \hat{\mathbf{y}} = 6\mathbf{e}_1 + 3\mathbf{e}_2 = (4, 2, 4) + (1, 2, -2) = (5, 4, 2) $$

and $\mathbf{z} = (9,0,0) - (5,4,2) = (4, -4, -2)$.

Check orthogonality: $\mathbf{z} \cdot \mathbf{e}_1 = (8 - 4 - 4)/3 = 0$ and $\mathbf{z} \cdot \mathbf{e}_2 = (4 - 8 + 4)/3 = 0$. The distance from $\mathbf{y}$ to the plane is $\|\mathbf{z}\| = \sqrt{16 + 16 + 4} = 6$.

The same answer comes from $UU^T\mathbf{y}$ with $U = [\mathbf{e}_1 ; \mathbf{e}_2]$, and computing it that way is worth doing once: the matrix $UU^T$ is built from the subspace alone, so it projects every vector in $\mathbb{R}^3$ at the cost of one matrix-vector product.

Practice

The main drill is the subspace formula: given an orthogonal basis and a target, produce the projection, the orthogonal part, and the distance.

The single-direction version is the building block, and it is worth staying fluent in it.

Videos

Watch for the derivation that the error vector must be orthogonal to every column of the basis matrix. That condition, written out, is the equation the next sections solve.

Quiz

Five items on computing projections onto subspaces and the distances they measure.