6.2 Orthogonal sets

Big idea

Finding coordinates in a basis has meant solving a system. Write the target as an unknown combination of the basis vectors, row reduce, read the weights. The work grows with the size of the basis and has to be redone for every new target.

If the basis vectors happen to be mutually orthogonal, that work disappears. Each weight comes from a single dot product, and the weights do not interact: changing one basis vector changes only its own coordinate. A system of equations collapses into a list of independent divisions.

Orthogonality also splits a vector cleanly. Given one direction, any vector breaks into the part along that direction and the part perpendicular to it, and the split is unique. That decomposition is the engine of the rest of the chapter; the general version in the next section is the same formula with more directions.

Decoder

The word “projection” gets used for both a vector and an operation. The projection of $\mathbf{y}$ onto a line is the vector on that line closest to $\mathbf{y}$; projecting is the act of computing it. When a section says “the component of $\mathbf{y}$ orthogonal to $\mathbf{u}$”, it means what is left after the projection is subtracted.

“Orthogonal set” and “orthonormal set” differ by one requirement. Both demand that every pair of distinct members be orthogonal. Orthonormal adds that every member have length $1$. Any orthogonal set of nonzero vectors becomes orthonormal after each vector is divided by its own length, which changes directions not at all.

Definitions and results

Orthogonal set. A set $\{\mathbf{u}_1, \dots, \mathbf{u}_p\}$ is orthogonal when $\mathbf{u}_i \cdot \mathbf{u}_j = 0$ for every $i \neq j$. Only distinct pairs are tested; a vector is not required to be orthogonal to itself, and could not be unless it were zero.

Orthogonal sets are independent. An orthogonal set of nonzero vectors is linearly independent. Dot a dependence relation with $\mathbf{u}_i$: every term but one dies, leaving $c_i(\mathbf{u}_i \cdot \mathbf{u}_i) = 0$, and since $\mathbf{u}_i$ is nonzero, $c_i = 0$. So such a set is automatically a basis for the subspace it spans, called an orthogonal basis.

Coordinates in an orthogonal basis. If $\{\mathbf{u}_1, \dots, \mathbf{u}_p\}$ is an orthogonal basis for a subspace $W$ and $\mathbf{y}$ is in $W$, then

$$ \mathbf{y} = c_1\mathbf{u}_1 + \cdots + c_p\mathbf{u}_p, \qquad c_j = \frac{\mathbf{y} \cdot \mathbf{u}_j}{\mathbf{u}_j \cdot \mathbf{u}_j} $$

No row reduction, no back substitution, and each $c_j$ is computed without reference to the others.

Projection onto a line. For nonzero $\mathbf{u}$, split $\mathbf{y}$ as

$$ \hat{\mathbf{y}} = \frac{\mathbf{y} \cdot \mathbf{u}}{\mathbf{u} \cdot \mathbf{u}}\mathbf{u}, \qquad \mathbf{z} = \mathbf{y} - \hat{\mathbf{y}} $$

Then $\hat{\mathbf{y}}$ is a multiple of $\mathbf{u}$ and $\mathbf{z}$ is orthogonal to $\mathbf{u}$, and no other split of $\mathbf{y}$ has both properties. The vector $\hat{\mathbf{y}}$ is the orthogonal projection of $\mathbf{y}$ onto the line through $\mathbf{u}$, and it depends on the line, not on which vector along it you used: replacing $\mathbf{u}$ by $5\mathbf{u}$ leaves $\hat{\mathbf{y}}$ unchanged, because the scalar appears squared below and once above.

Orthonormal set. An orthogonal set of unit vectors. For an orthonormal basis, the coordinate formula loses its denominators: $c_j = \mathbf{y} \cdot \mathbf{u}_j$.

Matrices with orthonormal columns. Let $U$ be $m \times n$ with orthonormal columns. Then $U^TU = I_n$, because entry $(i,j)$ of $U^TU$ is column $i$ dotted with column $j$. Three consequences follow directly: $\|U\mathbf{x}\| = \|\mathbf{x}\|$, and $(U\mathbf{x}) \cdot (U\mathbf{y}) = \mathbf{x} \cdot \mathbf{y}$, and therefore $U\mathbf{x}$ and $U\mathbf{y}$ are orthogonal exactly when $\mathbf{x}$ and $\mathbf{y}$ are. Such a map preserves lengths and angles; it moves the picture without distorting it.

Edge cases worth knowing. A set containing the zero vector can still be orthogonal, since the zero vector is orthogonal to everything, but it is dependent and useless as a basis, and the coordinate formula divides by zero on it. That is why statements about orthogonal bases say “nonzero”. A set with one vector is orthogonal by default: there are no distinct pairs to test.

Orthogonal matrix. The name is reserved for a square matrix with orthonormal columns. For those, $U^TU = I$ forces $U^{-1} = U^T$, so the rows are orthonormal too. The name is a historical mismatch worth noting: orthogonal columns are not enough, they must also be unit vectors.

Worked examples

Coordinates without solving a system

The set

$$ \mathbf{u}_1 = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}, \qquad \mathbf{u}_2 = \begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}, \qquad \mathbf{u}_3 = \begin{bmatrix} 1 \\ 1 \\ -2 \end{bmatrix} $$

is orthogonal: $\mathbf{u}_1 \cdot \mathbf{u}_2 = 1 - 1 + 0 = 0$, $\mathbf{u}_1 \cdot \mathbf{u}_3 = 1 + 1 - 2 = 0$, and $\mathbf{u}_2 \cdot \mathbf{u}_3 = 1 - 1 + 0 = 0$. Three nonzero orthogonal vectors in $\mathbb{R}^3$ are independent, so they form a basis.

Write $\mathbf{y} = (6, 2, 1)$ in that basis. The square lengths are $\mathbf{u}_1 \cdot \mathbf{u}_1 = 3$, $\mathbf{u}_2 \cdot \mathbf{u}_2 = 2$, and $\mathbf{u}_3 \cdot \mathbf{u}_3 = 6$. The numerators are $\mathbf{y} \cdot \mathbf{u}_1 = 6 + 2 + 1 = 9$, $\mathbf{y} \cdot \mathbf{u}_2 = 6 - 2 + 0 = 4$, and $\mathbf{y} \cdot \mathbf{u}_3 = 6 + 2 - 2 = 6$. So the weights are $9/3 = 3$, $4/2 = 2$, and $6/6 = 1$.

Check by rebuilding: $3(1,1,1) + 2(1,-1,0) + 1(1,1,-2) = (3+2+1, ; 3-2+1, ; 3+0-2) = (6, 2, 1)$. Correct, and no system was solved.

Splitting a vector along a line

Project $\mathbf{y} = (7, 6)$ onto the line through $\mathbf{u} = (4, 2)$. The numerator is $28 + 12 = 40$ and the denominator is $16 + 4 = 20$, so the weight is $2$ and

$$ \hat{\mathbf{y}} = 2\begin{bmatrix} 4 \\ 2 \end{bmatrix} = \begin{bmatrix} 8 \\ 4 \end{bmatrix}, \qquad \mathbf{z} = \begin{bmatrix} 7 \\ 6 \end{bmatrix} - \begin{bmatrix} 8 \\ 4 \end{bmatrix} = \begin{bmatrix} -1 \\ 2 \end{bmatrix} $$

Check both required properties. The first piece is a multiple of $\mathbf{u}$ by construction. The second satisfies $\mathbf{z} \cdot \mathbf{u} = -4 + 4 = 0$.

The Pythagorean identity gives a third check: $\|\mathbf{y}\|^2 = 49 + 36 = 85$, while $\|\hat{\mathbf{y}}\|^2 = 64 + 16 = 80$ and $|\mathbf{z}|^2 = 1 + 4 = 5$. The two parts account for the whole, $80 + 5 = 85$.

Orthonormal columns preserve length

Let

$$ \mathbf{e}_1 = \frac{1}{3}\begin{bmatrix} 2 \\ 1 \\ 2 \end{bmatrix}, \qquad \mathbf{e}_2 = \frac{1}{3}\begin{bmatrix} 1 \\ 2 \\ -2 \end{bmatrix} $$

Each has square length $(4 + 1 + 4)/9 = 1$ and $(1 + 4 + 4)/9 = 1$, and their dot product is $(2 + 2 - 4)/9 = 0$. So the pair is orthonormal and the $3 \times 2$ matrix $U$ with these columns satisfies $U^TU = I_2$.

Take $\mathbf{x} = (3, -3)$. Then

$$ U\mathbf{x} = 3\mathbf{e}_1 - 3\mathbf{e}_2 = (2,1,2) - (1,2,-2) = (1, -1, 4) $$

Compare lengths: $\|\mathbf{x}\|^2 = 9 + 9 = 18$ and $|U\mathbf{x}|^2 = 1 + 1

• 16 = 18$. Equal, as promised. Note that $U$ is not square, so it is not an orthogonal matrix; the length-preserving property needs only orthonormal columns.

The cost comparison is worth making explicit. Finding coordinates in a general basis for a subspace of $\mathbb{R}^n$ means row reducing an augmented matrix, and the work grows with the cube of the number of basis vectors. In an orthogonal basis it is one dot product and one division per coordinate, so the work grows linearly in the number of basis vectors. That gap is why the next two sections spend effort manufacturing orthogonal bases rather than using whatever basis arrived first.

Practice

First the recognition and construction work: deciding whether a set is orthogonal or orthonormal, and normalizing one that is not.

Then the projection formula itself, one direction at a time, including reading off the orthogonal leftover.

Videos

The first half covers orthonormal sets and what $Q^TQ = I$ buys you; the Gram-Schmidt half belongs to a later section. Watch for the argument that orthonormal columns make the inverse free.

Quiz

Five items on orthogonal sets, coordinates in an orthogonal basis, and projection onto a line.