7.1 Diagonalization of symmetric matrices

Big idea

Symmetric matrices are where eigenvalue work stops going wrong. A real symmetric matrix always has real eigenvalues, always has enough eigenvectors to diagonalize, and always admits a diagonalizing matrix whose columns are orthonormal. None of that is conditional. You never test a symmetric matrix for diagonalizability; it passes.

One short fact drives all of it: eigenvectors of a symmetric matrix that belong to different eigenvalues are perpendicular, not merely independent. Independence is all that plain diagonalization needs. Perpendicularity is the extra, and it is what upgrades the change-of-basis matrix $P$ to an orthogonal matrix, so that $P^{-1} = P^{T}$ and inverting costs a transpose.

The result is a factorization $A = PDP^{T}$ that reads as a change of coordinates: turn into the eigenvector coordinate system with $P^{T}$, scale each axis by its eigenvalue with $D$, turn back with $P$. The rest of the chapter is that sentence pointed at different questions: which direction stretches the most, when a quadratic expression stays positive, how to factor a matrix that is not even square.

Decoder

A matrix is orthogonally diagonalizable when an orthogonal $P$ and a diagonal $D$ exist with $A = PDP^{-1}$.

Plainly: you can find a full set of eigenvectors for $A$ that are mutually perpendicular and each of length one. Put those eigenvectors in the columns of $P$, put their eigenvalues in the matching positions on the diagonal of $D$, and the factorization holds. The word “orthogonally” describes $P$, not some new kind of diagonalizing. Because $P$ has orthonormal columns, $P^{-1}$ is just $P^{T}$, which is why the factorization is normally written $A = PDP^{T}$.

“Orthogonal matrix” is a misleading name and it costs people points. The columns are not just perpendicular; they are also unit vectors. A matrix with perpendicular columns of length $3$ is not an orthogonal matrix. Normalize first.

Definitions and results

Symmetric matrix. A square matrix with $A^{T} = A$. The entry in row $i$, column $j$ equals the entry in row $j$, column $i$, so the array mirrors across the main diagonal.

Orthogonal matrix. A square matrix $P$ whose columns form an orthonormal set. That is the same as $P^{T}P = I$, so $P^{-1} = P^{T}$. Such a $P$ preserves dot products and lengths: $(P\mathbf{x}) \cdot (P\mathbf{y}) = \mathbf{x} \cdot \mathbf{y}$ and $\|P\mathbf{x}\| = \|\mathbf{x}\|$. Geometrically it is a rotation, a reflection, or a combination of the two.

Eigenvectors come out perpendicular. Let $A$ be symmetric with $A\mathbf{u} = \lambda\mathbf{u}$ and $A\mathbf{v} = \mu\mathbf{v}$ for $\lambda \neq \mu$. Then

$$ \lambda (\mathbf{u} \cdot \mathbf{v}) = (A\mathbf{u})^{T}\mathbf{v} = \mathbf{u}^{T}A^{T}\mathbf{v} = \mathbf{u}^{T}A\mathbf{v} = \mu (\mathbf{u} \cdot \mathbf{v}) $$

so $(\lambda - \mu)(\mathbf{u} \cdot \mathbf{v}) = 0$, and since the eigenvalues differ, $\mathbf{u} \cdot \mathbf{v} = 0$. The symmetry is used exactly once, at $A^{T} = A$.

Which matrices are orthogonally diagonalizable. Exactly the symmetric ones. One direction takes a line: if $A = PDP^{T}$ then $A^{T} = (P^{T})^{T}D^{T}P^{T} = PDP^{T} = A$, because a diagonal matrix is its own transpose. The harder direction is the content of the spectral theorem.

Spectral theorem. For a real symmetric $n \times n$ matrix $A$: every eigenvalue is real; counted with multiplicity there are $n$ of them; each eigenspace has dimension equal to the multiplicity of its eigenvalue, so there are never too few eigenvectors; eigenspaces for different eigenvalues are mutually perpendicular; and $A$ factors as $PDP^{T}$ with $P$ orthogonal. The list of eigenvalues with multiplicity is called the spectrum of $A$.

Building $P$. Find a basis for each eigenspace. Inside an eigenspace of dimension two or more, the basis that row reduction hands you is usually not orthogonal, so run Gram-Schmidt within that eigenspace. Across different eigenspaces you get orthogonality for free. Normalize every vector, then stack them as the columns of $P$. The column order is your choice, as long as $D$ uses the same order.

Spectral decomposition. With orthonormal eigenvectors $\mathbf{u}_1, \dots, \mathbf{u}_n$ and matching eigenvalues,

$$ A = \lambda_1 \mathbf{u}_1\mathbf{u}_1^{T} + \lambda_2 \mathbf{u}_2\mathbf{u}_2^{T} + \cdots + \lambda_n \mathbf{u}_n\mathbf{u}_n^{T} $$

Each $\mathbf{u}_i\mathbf{u}_i^{T}$ is $n \times n$ and has rank $1$; it is the matrix that projects a vector orthogonally onto the line through $\mathbf{u}_i$. So a symmetric matrix is a weighted sum of projections, one per eigendirection, and the weights are the eigenvalues. Watch the order: $\mathbf{u}^{T}\mathbf{u}$ is the number $1$, while $\mathbf{u}\mathbf{u}^{T}$ is a whole matrix.

Two free checks. The trace equals the sum of the eigenvalues and the determinant equals their product. Both are one subtraction away and they catch most sign slips.

Worked examples

A two by two, start to finish

Take

$$ A = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix} $$

The characteristic polynomial is $(3 - \lambda)^2 - 1 = \lambda^2 - 6\lambda + 8$, so the eigenvalues are $\lambda = 4$ and $\lambda = 2$. Trace check: $4 + 2 = 6$, matching $3 + 3$. Determinant check: $4 \cdot 2 = 8 = 9 - 1$.

For $\lambda = 4$, solve $(A - 4I)\mathbf{x} = \mathbf{0}$. The matrix $\begin{bmatrix} -1 & 1 \\ 1 & -1 \end{bmatrix}$ gives $x_1 = x_2$, so $(1, 1)$ spans the eigenspace. For $\lambda = 2$, the matrix $\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$ gives $x_1 = -x_2$, so $(1, -1)$ spans that one. Their dot product is $1 - 1 = 0$, as the theory promised. Normalize both by dividing by $\sqrt{2}$:

$$ P = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}, \qquad D = \begin{bmatrix} 4 & 0 \\ 0 & 2 \end{bmatrix} $$

Check the factorization. First $DP^{T} = \frac{1}{\sqrt{2}}\begin{bmatrix} 4 & 4 \ 2 & -2 \end{bmatrix}$. Then

$$ PDP^{T} = \frac{1}{2}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}\begin{bmatrix} 4 & 4 \\ 2 & -2 \end{bmatrix} = \frac{1}{2}\begin{bmatrix} 6 & 2 \\ 2 & 6 \end{bmatrix} = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix} $$

That is $A$, so the factorization is correct.

A repeated eigenvalue

Take

$$ B = \begin{bmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 2 \end{bmatrix} $$

Every row sums to $4$, so $B(1,1,1) = (4,4,4)$ and $\lambda = 4$ is an eigenvalue with eigenvector $(1,1,1)$. The trace is $6$, so the other two eigenvalues sum to $2$. Look at $B - I$: every entry is $1$, so it has rank $1$ and its null space has dimension $2$. That makes $\lambda = 1$ an eigenvalue of multiplicity $2$, and $4 + 1 + 1 = 6$ closes the trace.

The eigenspace for $\lambda = 1$ is the solution set of $x_1 + x_2 + x_3 = 0$, a plane. Row reduction offers the basis $(1, -1, 0)$ and $(1, 0, -1)$, which is not orthogonal: their dot product is $1$. Apply Gram-Schmidt. Keep $\mathbf{w}_1 = (1, -1, 0)$ and subtract its component from the second vector:

$$ \mathbf{w}_2 = (1, 0, -1) - \frac{1}{2}(1, -1, 0) = \left(\tfrac{1}{2}, \tfrac{1}{2}, -1\right) $$

Scale by $2$ to clear fractions: $(1, 1, -2)$. Check the pair: $1 - 1 + 0 = 0$. Check against the other eigenspace: $(1,1,1) \cdot (1,-1,0) = 0$ and $(1,1,1) \cdot (1,1,-2) = 0$, both free by the perpendicularity result. Normalize by $\sqrt{3}$, $\sqrt{2}$, $\sqrt{6}$:

$$ P = \begin{bmatrix} 1/\sqrt{3} & 1/\sqrt{2} & 1/\sqrt{6} \\ 1/\sqrt{3} & -1/\sqrt{2} & 1/\sqrt{6} \\ 1/\sqrt{3} & 0 & -2/\sqrt{6} \end{bmatrix}, \qquad D = \begin{bmatrix} 4 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$

Check one column of $P^{T}P$: the second column of $P$ dotted with itself is $\tfrac{1}{2} + \tfrac{1}{2} + 0 = 1$, and dotted with the third it is $\tfrac{1}{\sqrt{12}} - \tfrac{1}{\sqrt{12}} + 0 = 0$. Check the factorization on a vector instead of multiplying three matrices: $P^{T}(1,1,1) = (\sqrt{3}, 0, 0)$, then $D$ makes it $(4\sqrt{3}, 0, 0)$, then $P$ returns $(4, 4, 4)$, which is $B(1,1,1)$.

Splitting a matrix into projections

Use the two by two above. Its unit eigenvectors are $\mathbf{u}_1 = \tfrac{1}{\sqrt{2}}(1,1)$ and $\mathbf{u}_2 = \tfrac{1}{\sqrt{2}}(1,-1)$, so

$$ \mathbf{u}_1\mathbf{u}_1^{T} = \frac{1}{2}\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}, \qquad \mathbf{u}_2\mathbf{u}_2^{T} = \frac{1}{2}\begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix} $$

Weight them by the eigenvalues:

$$ 4 \cdot \frac{1}{2}\begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix} + 2 \cdot \frac{1}{2}\begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix} = \begin{bmatrix} 2 & 2 \\ 2 & 2 \end{bmatrix} + \begin{bmatrix} 1 & -1 \\ -1 & 1 \end{bmatrix} = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix} $$

Back to $A$. Read the two pieces: $A$ stretches anything along $(1,1)$ by $4$ and anything along $(1,-1)$ by $2$, and every other vector is a mix of those two behaviors.

Practice

First the eigenvectors themselves, since every step above starts by solving $(A - \lambda I)\mathbf{x} = \mathbf{0}$ and reading off a basis.

Next the factorization. These ask you to assemble $P$ and $D$ and to confirm that $A = PDP^{-1}$ holds.

Last the matrices that make $P^{-1} = P^{T}$ work: recognizing orthonormal columns, and using the transpose as the inverse.

Videos

Watch for the argument that a real symmetric matrix cannot have a complex eigenvalue, and for the moment the factorization is rewritten with $P^{T}$ in place of $P^{-1}$. That swap is the whole advantage of symmetry.

Quiz

Six items on finding eigenvectors, orthogonally diagonalizing a small symmetric matrix, and recognizing orthogonal matrices.