6.7 Inner product spaces

Big idea

Every result in this chapter was proved from four properties of the dot product: symmetry, additivity, scalars pulling out, and positivity of a vector against itself. No proof opened up a vector and looked at its entries.

So the results hold for any operation with those four properties, on any vector space. Declare such an operation on polynomials, on functions, or on $\mathbb{R}^n$ with the entries weighted unequally, and you inherit length, distance, orthogonality, Gram-Schmidt, and best approximation without proving anything again.

The payoff is that approximation problems far from geometry become projection problems. Finding the polynomial of low degree closest to a given function, or the best fit that trusts some measurements more than others, is projection onto a subspace once the right inner product is in place.

Decoder

An inner product on a vector space is a function assigning a scalar to each ordered pair of vectors, subject to the symmetry, bilinearity, and positive definiteness axioms.

Unpacked into the four rules, with $\langle \mathbf{u}, \mathbf{v} \rangle$ the notation for the product: swapping the arguments changes nothing; the product distributes over a sum in the first argument; a scalar in the first argument comes out front; and $\langle \mathbf{u}, \mathbf{u} \rangle$ is positive except for the zero vector, where it is zero. Symmetry extends the middle two rules to the second argument for free.

“Positive definite” is the rule doing real work. It is what makes $\sqrt{\langle \mathbf{u}, \mathbf{u} \rangle}$ a usable length, and it is the rule most easily broken by a careless definition, so check it first.

Definitions and results

Inner product space. A vector space together with an inner product. The dot product makes $\mathbb{R}^n$ one, but it is not the only choice for that space.

Weighted inner product on $\mathbb{R}^n$. Fix positive weights $w_1, \dots, w_n$ and define

$$ \langle \mathbf{u}, \mathbf{v} \rangle = w_1u_1v_1 + w_2u_2v_2 + \cdots + w_nu_nv_n $$

The four rules hold, and positivity needs every weight to be strictly positive. A zero or negative weight breaks the definition: a nonzero vector could then have length zero or an imaginary length.

Evaluation inner product on polynomials. Fix distinct numbers $t_0, \dots, t_n$ and define $\langle p, q \rangle = p(t_0)q(t_0) + \cdots + p(t_n)q(t_n)$. This is the dot product of the two vectors of sampled values. Positivity holds on polynomials of degree at most $n$: such a polynomial vanishing at all $n + 1$ sample points must be the zero polynomial. Use fewer sample points than that and the rule fails.

Inner product on continuous functions. On $C[a,b]$, define $\langle f, g \rangle = \int_a^b f(t)g(t)\,dt$. The evaluation product is its discrete cousin, summing over sample points instead of integrating.

What carries over. Once an inner product is fixed: $\|\mathbf{u}\| = \sqrt{\langle \mathbf{u}, \mathbf{u} \rangle}$; the distance between two vectors is the length of their difference; two vectors are orthogonal when their inner product is zero; the Pythagorean identity holds; Gram-Schmidt produces orthogonal bases; and the projection onto a subspace with orthogonal basis $\{\mathbf{u}_1, \dots, \mathbf{u}_p\}$ is

$$ \hat{\mathbf{y}} = \frac{\langle \mathbf{y}, \mathbf{u}_1 \rangle}{\langle \mathbf{u}_1, \mathbf{u}_1 \rangle}\mathbf{u}_1 + \cdots + \frac{\langle \mathbf{y}, \mathbf{u}_p \rangle}{\langle \mathbf{u}_p, \mathbf{u}_p \rangle}\mathbf{u}_p $$

and it is still the closest point of the subspace, now measured in the new length.

Cauchy-Schwarz inequality. $|\langle \mathbf{u}, \mathbf{v} \rangle| \leq |\mathbf{u}| , |\mathbf{v}|$, with equality only when one vector is a multiple of the other. This is what lets you define an angle in any inner product space: the ratio on the left over the right never exceeds $1$, so it is a valid cosine.

Triangle inequality. $|\mathbf{u} + \mathbf{v}| \leq |\mathbf{u}| + |\mathbf{v}|$, a consequence of Cauchy-Schwarz.

Orthogonality depends on the product. Two vectors orthogonal under one inner product are usually not orthogonal under another. There is no such thing as perpendicular without a stated inner product.

Worked examples

A weighted product on the plane

Define $\langle \mathbf{u}, \mathbf{v} \rangle = 4u_1v_1 + 5u_2v_2$ on $\mathbb{R}^2$. Both weights are positive, so this is an inner product.

Take $\mathbf{u} = (1, 1)$ and $\mathbf{v} = (5, -4)$. Then

$$ \langle \mathbf{u}, \mathbf{v} \rangle = 4(1)(5) + 5(1)(-4) = 20 - 20 = 0 $$

so they are orthogonal here. Under the ordinary dot product $\mathbf{u} \cdot \mathbf{v} = 5 - 4 = 1$, which is not zero. The same pair of arrows is perpendicular in one inner product and not in the other.

Lengths: $\|\mathbf{u}\|^2 = 4 + 5 = 9$, so $\|\mathbf{u}\| = 3$, and $\|\mathbf{v}\|^2 = 4(25) + 5(16) = 100 + 80 = 180$, so $\|\mathbf{v}\| = 6\sqrt{5}$.

Check the Pythagorean identity, which must hold since they are orthogonal here. $\mathbf{u} + \mathbf{v} = (6, -3)$, and $|\mathbf{u} + \mathbf{v}|^2 = 4(36) + 5(9) = 144 + 45 = 189$. And $9 + 180 = 189$. It holds.

Checking the two inequalities

Same product, with $\mathbf{u} = (1,1)$ and $\mathbf{w} = (1,0)$. Then $\langle \mathbf{u}, \mathbf{w} \rangle = 4$, while $\|\mathbf{u}\| = 3$ and $\|\mathbf{w}\| = 2$. Cauchy-Schwarz says $4 \leq 6$, which holds, and the strict inequality is expected since $\mathbf{w}$ is not a multiple of $\mathbf{u}$.

For the triangle inequality, $\mathbf{u} + \mathbf{w} = (2, 1)$ has $\|\mathbf{u} + \mathbf{w}\|^2 = 4(4) + 5(1) = 21$, so the length is $\sqrt{21}$, about $4.58$, and the bound is $3 + 2 = 5$. It holds, with room to spare because the two vectors point in different directions.

Orthogonal polynomials from sampled values

On polynomials of degree at most $2$, define $\langle p, q \rangle = p(-1)q(-1) + p(0)q(0) + p(1)q(1)$. Three distinct sample points, so this is an inner product on that space.

Start from the basis $1$, $t$, $t^2$ and orthogonalize. First, $\langle 1, t \rangle = -1 + 0 + 1 = 0$, so those two are already orthogonal; keep both. Next, $\langle t^2, 1 \rangle = 1 + 0 + 1 = 2$ and $\langle 1, 1 \rangle = 3$, while $\langle t^2, t \rangle = -1 + 0 + 1 = 0$. So the third vector loses only its constant component:

$$ r(t) = t^2 - \frac{2}{3} $$

Check: $\langle r, 1 \rangle = \left(1 - \tfrac{2}{3}\right) + \left(0 - \tfrac{2}{3}\right) + \left(1 - \tfrac{2}{3}\right) = \tfrac{1}{3} - \tfrac{2}{3} + \tfrac{1}{3} = 0$, and $\langle r, t \rangle = -\tfrac{1}{3} + 0 + \tfrac{1}{3} = 0$. So $\{1, t, t^2 - \tfrac{2}{3}\}$ is an orthogonal basis.

Now the best approximation question: which polynomial of degree at most $1$ is closest to $t^2$ in this inner product? Project onto the span of $1$ and $t$. The coefficients are $\langle t^2, 1 \rangle / \langle 1, 1 \rangle = 2/3$ and $\langle t^2, t \rangle / \langle t, t \rangle = 0/2 = 0$, so the answer is the constant $2/3$.

The error is $t^2 - 2/3$, whose squared length is $\left(\tfrac{1}{3}\right)^2 + \left(\tfrac{2}{3}\right)^2 + \left(\tfrac{1}{3}\right)^2 = \tfrac{6}{9} = \tfrac{2}{3}$. That is the distance, squared, from $t^2$ to every line in this space, and no choice of slope and intercept beats it.

Practice

The computation inside any of these products is still a sum of products of numbers, so stay fast at the underlying arithmetic.

Lengths and distances come from the product by the same square root, whichever product is in force.

And the orthogonal-set machinery transfers unchanged, which is what makes the whole generalization worth having.

Quiz

Six items on inner product rules, weighted products, lengths, and orthogonality.