7.2 Quadratic forms

Big idea

A quadratic form is a polynomial in several variables where every term has degree exactly two: squares and cross products, no linear part, no constant. Any such expression can be written as $\mathbf{x}^{T}A\mathbf{x}$ for exactly one symmetric matrix $A$. That single move hands the whole subject to the machinery of the previous section.

Cross terms are what make a form hard to read. With them, you cannot tell at a glance whether the expression can go negative or where it is largest. Without them, the answer is obvious: a sum of squares with positive coefficients is positive, and a mixed sign means the form takes both signs. So the goal is to get rid of the cross terms.

An orthogonal change of variable does it. Because $A$ is symmetric, $A = PDP^{T}$ with $P$ orthogonal; substituting $\mathbf{x} = P\mathbf{y}$ turns the form into $\lambda_1 y_1^2 + \cdots + \lambda_n y_n^2$. The variables change but the geometry does not, because $P$ only rotates or reflects. The eigenvalues survive the substitution untouched, which is why the sign pattern of the eigenvalues is the complete answer to whether a form is always positive, always negative, or neither.

Decoder

A change of variable $\mathbf{x} = P\mathbf{y}$ with $P$ orthogonal produces a quadratic form with no cross-product term.

The sentence hides a substitution. Start with $Q(\mathbf{x}) = \mathbf{x}^{T}A\mathbf{x}$ and replace every $\mathbf{x}$ by $P\mathbf{y}$:

$$ (P\mathbf{y})^{T}A(P\mathbf{y}) = \mathbf{y}^{T}(P^{T}AP)\mathbf{y} = \mathbf{y}^{T}D\mathbf{y} $$

Since $D$ is diagonal, $\mathbf{y}^{T}D\mathbf{y}$ has only squared terms. So the new variables $y_1, \dots, y_n$ are just the coordinates of the same old point measured along the eigenvector directions instead of along the standard axes, and $\mathbf{y} = P^{T}\mathbf{x}$ converts back. The point did not move. The ruler did.

Definitions and results

Quadratic form. A function $Q(\mathbf{x}) = \mathbf{x}^{T}A\mathbf{x}$ on $\mathbb{R}^n$ with $A$ symmetric. Written out, it is a sum of terms $a_{ii}x_i^2$ and $2a_{ij}x_ix_j$.

Matrix of a form. Given the expression, build $A$ by putting the coefficient of $x_i^2$ in the $(i,i)$ entry and half the coefficient of $x_ix_j$ in both the $(i,j)$ and the $(j,i)$ entries. Splitting the cross coefficient in half is what makes $A$ symmetric, and the symmetric choice is unique. Going the other way, expand $\mathbf{x}^{T}A\mathbf{x}$ and each off-diagonal pair contributes twice.

Cross-product term. Any term with two different variables. A form has none exactly when its matrix is diagonal.

Change of variable. For an invertible $P$, setting $\mathbf{x} = P\mathbf{y}$ replaces $A$ by $P^{T}AP$. This is not similarity in general, but when $P$ is orthogonal, $P^{T}AP = P^{-1}AP$, so it is both a change of variable and a similarity, and the eigenvalues are unchanged.

Principal axes theorem. Every quadratic form can be written without cross terms by an orthogonal change of variable. Take $P$ from the orthogonal diagonalization of $A$; then $Q = \lambda_1 y_1^2 + \cdots + \lambda_n y_n^2$. The columns of $P$ are called the principal axes of the form; in two variables they are the axes of the ellipse or hyperbola that the level curves trace.

Classification. Let $A$ be the matrix of $Q$. The form is

• positive definite when $Q(\mathbf{x}) > 0$ for every $\mathbf{x} \neq \mathbf{0}$, which happens exactly when all eigenvalues of $A$ are positive;
• negative definite when $Q(\mathbf{x}) < 0$ for every $\mathbf{x} \neq \mathbf{0}$, exactly when all eigenvalues are negative;
• indefinite when $Q$ takes both positive and negative values, exactly when $A$ has eigenvalues of both signs;
• positive semidefinite when $Q(\mathbf{x}) \geq 0$ always, exactly when no eigenvalue is negative, and negative semidefinite in the mirror case.

Why the eigenvalue test works. In the new variables $Q = \sum \lambda_i y_i^2$, and $\mathbf{y} = \mathbf{0}$ only when $\mathbf{x} = \mathbf{0}$. If every $\lambda_i > 0$ the sum is positive unless all $y_i$ vanish. If some $\lambda_i < 0$, choose $\mathbf{y}$ to be the $i$th standard basis vector and $Q$ comes out negative. A zero eigenvalue gives directions where the form vanishes without the vector being zero: $x_1^2 + 2x_1x_2 + x_2^2 = (x_1 + x_2)^2$ is zero all along the line $x_2 = -x_1$, and its matrix has eigenvalues $2$ and $0$.

Do not read the diagonal. Signs of the diagonal entries do not classify a form. A matrix with $1$ in both diagonal slots can still be indefinite if the cross term is large enough, as the second worked example shows. Only the eigenvalues decide.

Level sets. For a positive definite form in two variables, the curve $Q(\mathbf{x}) = c$ with $c > 0$ is an ellipse whose axes point along the eigenvectors, with semi-axis length $\sqrt{c/\lambda}$ in the eigenvector direction for $\lambda$. A large eigenvalue means a short axis. An indefinite form gives hyperbolas.

Worked examples

From expression to matrix, and back without cross terms

Take $Q(\mathbf{x}) = 5x_1^2 + 4x_1x_2 + 5x_2^2$. The diagonal entries are $5$ and $5$; the cross coefficient $4$ splits into $2$ and $2$:

$$ A = \begin{bmatrix} 5 & 2 \\ 2 & 5 \end{bmatrix} $$

Expand to confirm: $\mathbf{x}^{T}A\mathbf{x} = 5x_1^2 + 2x_1x_2 + 2x_2x_1 + 5x_2^2$, and the two middle terms add to $4x_1x_2$.

The characteristic polynomial is $(5 - \lambda)^2 - 4$, so $\lambda = 7$ and $\lambda = 3$. Both are positive, so the form is positive definite. The eigenvectors are $(1,1)$ for $7$ and $(1,-1)$ for $3$, so take

$$ P = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} $$

and the change of variable $\mathbf{x} = P\mathbf{y}$ gives $Q = 7y_1^2 + 3y_2^2$.

Check at a point. At $\mathbf{x} = (2, 1)$ the original expression gives $5(4) + 4(2) + 5(1) = 33$. The new coordinates are $\mathbf{y} = P^{T}\mathbf{x} = \tfrac{1}{\sqrt{2}}(3, 1)$, so $7y_1^2 + 3y_2^2 = 7 \cdot \tfrac{9}{2} + 3 \cdot \tfrac{1}{2} = \tfrac{63 + 3}{2} = 33$. The two agree.

An indefinite form with a harmless-looking matrix

Take $Q(\mathbf{x}) = x_1^2 + 6x_1x_2 + x_2^2$, so

$$ A = \begin{bmatrix} 1 & 3 \\ 3 & 1 \end{bmatrix} $$

Both diagonal entries are positive, but the eigenvalues are $1 \pm 3$, that is $4$ and $-2$. Opposite signs, so the form is indefinite.

Confirm directly instead of trusting the rule. Along the eigenvector $(1,1)$: $1 + 6 + 1 = 8$, positive. Along $(1,-1)$: $1 - 6 + 1 = -4$, negative. A single form takes both signs, and the two witnesses are exactly the eigenvector directions, where the form equals the eigenvalue times the squared length: $4 \cdot 2 = 8$ and $-2 \cdot 2 = -4$.

In the new variables the form is $4y_1^2 - 2y_2^2$, and the level curves $Q = c$ are hyperbolas opening along the $(1,1)$ direction.

Three variables

Take $Q(\mathbf{x}) = 2x_1^2 + 2x_2^2 + 2x_3^2 + 2x_1x_2 + 2x_1x_3 + 2x_2x_3$. Each cross coefficient is $2$, so each splits into $1$:

$$ A = \begin{bmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 2 \end{bmatrix} $$

This is the matrix diagonalized in the previous section, with eigenvalues $4$, $1$, $1$ and orthonormal eigenvectors $\tfrac{1}{\sqrt{3}}(1,1,1)$, $\tfrac{1}{\sqrt{2}}(1,-1,0)$, $\tfrac{1}{\sqrt{6}}(1,1,-2)$. All eigenvalues are positive, so the form is positive definite, and in the new variables it is $4y_1^2 + y_2^2 + y_3^2$.

Check at $\mathbf{x} = (1, 2, 3)$. The original expression gives $2(1 + 4 + 9) + 2(2) + 2(3) + 2(6) = 28 + 22 = 50$. The new coordinates are the dot products with the three unit eigenvectors: $y_1 = 6/\sqrt{3}$, $y_2 = -1/\sqrt{2}$, $y_3 = -3/\sqrt{6}$. Then

$$ 4y_1^2 + y_2^2 + y_3^2 = 4 \cdot \frac{36}{3} + \frac{1}{2} + \frac{9}{6} = 48 + \frac{1}{2} + \frac{3}{2} = 50 $$

Same value, as it must be: the form did not change, only the coordinates used to describe the point.

Practice

Everything here starts with eigenvalues, since their signs classify the form. These drills are the characteristic polynomial and its roots.

Then the factorization that removes the cross terms. Assembling $P$ and $D$ for a symmetric matrix is exactly the work of an orthogonal change of variable.

Videos

Watch for the tests that decide positive definiteness without computing every eigenvalue, and for the link between a positive definite matrix and a function sitting at a minimum. The second idea is what the next section pushes on.

Quiz

Five items on eigenvalues, classification by sign pattern, and the matrix of a quadratic form.