4.8 Lagrange Multipliers

Big idea

The previous section looked for the highest point of a surface with the inputs free to roam. Most real optimization is not like that. You maximize output subject to a fixed budget, minimize surface area subject to a fixed volume, find the point of a curve closest to a target. The constraint is an equation the inputs must satisfy, and it cuts the domain down to a curve or a surface.

One route is substitution: solve the constraint for one variable, put it into the objective, and optimize what remains with one-variable methods. That works when the constraint is easy to solve and becomes unpleasant or impossible when it is not, and it also destroys any symmetry the problem had. Lagrange’s method avoids solving the constraint at all.

The geometric idea is worth carrying rather than the formula. Draw the level curves of the objective $f$ and the constraint curve $g = k$ in the same plane. Walk along the constraint curve and watch the values of $f$ you cross. As long as the curve is cutting across level curves of $f$, the value is still rising or still falling, so you are not at an extremum yet. The value stops changing exactly where the constraint curve runs tangent to a level curve of $f$. At a point of tangency the two curves share a normal direction, and the gradients are normal to their own level sets, so $\nabla f$ and $\nabla g$ must be parallel there. Introducing a scalar $\lambda$ to record the ratio turns that geometric statement into the equation $\nabla f = \lambda \nabla g$.

The resulting system has one equation per variable plus the constraint itself, which is exactly enough to pin down the candidates. It finds candidates only, with no test for which are maxima and which are minima. On a closed bounded constraint set you settle that by evaluating $f$ at every candidate and comparing.

Decoder

If $f$ has a constrained extremum at a point of the curve or surface $g = k$, and $\nabla g$ is not the zero vector there, then $\nabla f = \lambda \nabla g$ at that point for some number $\lambda$.

The multiplier $\lambda$ is not the answer to anything you were asked. It is a ratio that appears because two parallel vectors differ by a scalar factor, and once it has helped you solve for the point you throw it away. Reporting $\lambda$ as the maximum value is a standard slip.

Read the statement as necessary, not sufficient. Points satisfying the system may be maxima, minima or neither, in the same way that a vanishing derivative in one variable only produces candidates. When the constraint set is closed and bounded, a continuous $f$ must attain a maximum and a minimum on it, so the largest value among the candidates is the maximum and the smallest is the minimum, with no further test needed. When the constraint set runs off to infinity, you have to think about behavior far out before declaring anything.

The hypothesis $\nabla g \ne \mathbf{0}$ matters. Where the constraint’s gradient vanishes, the constraint set can have a corner or a cusp, the tangency argument breaks, and extrema can hide at such points without satisfying the equations. Check for them separately when the constraint is anything other than a smooth curve.

Definitions and results

Constrained optimization problem. Maximize or minimize $f(x,y)$ subject to $g(x,y) = k$, or $f(x,y,z)$ subject to $g(x,y,z) = k$. The constraint equation must be written with the variable part on one side and the constant on the other.

The method of Lagrange multipliers. Solve the system

$$ \nabla f = \lambda \nabla g, \qquad g = k $$

for all unknowns, including $\lambda$. In two variables this reads $f_x = \lambda g_x$, $f_y = \lambda g_y$, $g(x,y) = k$: three equations in three unknowns.

Reading the answer. Evaluate $f$ at every solution point. If the constraint set is closed and bounded, the largest value is the constrained maximum and the smallest is the constrained minimum.

Why the gradients align. At a constrained extremum the directional derivative of $f$ along the constraint set is zero, so $\nabla f$ is perpendicular to the constraint set. But $\nabla g$ is also perpendicular to it, since the set is a level set of $g$. Two vectors perpendicular to the same curve or surface at a point are parallel.

Degenerate cases. If $\nabla g = \mathbf{0}$ at a point of the constraint set, that point is a candidate that the system may miss, and it must be examined by hand. If $\nabla f = \mathbf{0}$, the system is satisfied with $\lambda = 0$.

Solving the system. The equations are usually nonlinear, so there is no routine method. Two reliable moves: eliminate $\lambda$ by dividing one equation by another, watching for the division-by-zero cases separately, or solve each equation for $\lambda$ and set the expressions equal. Substitute into the constraint last.

Two constraints. To optimize $f$ subject to both $g = k_1$ and $h = k_2$, the constraint set is the intersection curve, and the condition becomes

$$ \nabla f = \lambda \nabla g + \mu \nabla h $$

with the two constraints appended. In three variables this is five equations in five unknowns. Geometrically, $\nabla f$ must lie in the plane spanned by the two constraint normals, which is the same as saying $\nabla f$ is perpendicular to the intersection curve.

Relation to unconstrained problems. Dropping the constraint recovers the previous section: with no $g$ to match, the condition collapses to $\nabla f = \mathbf{0}$.

Worked examples

A product on a circle

Maximize and minimize $f(x,y) = xy$ subject to $x^2 + y^2 = 8$.

Here $g = x^2 + y^2$, so $\nabla f = \langle y, x\rangle$ and $\nabla g = \langle 2x, 2y\rangle$. The system is

$$ y = 2\lambda x, \qquad x = 2\lambda y, \qquad x^2 + y^2 = 8 $$

Substituting the first into the second gives $x = 2\lambda(2\lambda x) = 4\lambda^2 x$. If $x = 0$ then the first equation forces $y = 0$, which fails the constraint, so $x \ne 0$ and $4\lambda^2 = 1$, giving $\lambda = \pm 1/2$ and therefore $y = \pm x$.

With $y = x$ the constraint gives $2x^2 = 8$, so $x = \pm 2$ and the points are $(2,2)$ and $(-2,-2)$, both with $f = 4$. With $y = -x$ the same arithmetic gives $(2,-2)$ and $(-2,2)$, both with $f = -4$. The circle is closed and bounded, so the maximum is $4$ and the minimum is $-4$.

Check with a substitution: on the circle write $x = 2\sqrt2\cos t$ and $y = 2\sqrt2 \sin t$, so $f = 8\cos t \sin t = 4\sin 2t$, which oscillates between $-4$ and $4$. The methods agree.

The closest point on a line

Find the point of the line $x + 2y = 10$ nearest the origin.

Minimize the squared distance $f(x,y) = x^2 + y^2$, which has its minimum at the same place as the distance and avoids a square root. With $g = x + 2y$,

$$ 2x = \lambda, \qquad 2y = 2\lambda, \qquad x + 2y = 10 $$

The first two give $y = \lambda = 2x$. Substituting into the constraint, $x + 4x = 10$, so $x = 2$ and $y = 4$. The nearest point is $(2,4)$, at squared distance $4 + 16 = 20$ and distance $2\sqrt5$.

Check with the point-line distance formula: the distance from the origin to $x + 2y - 10 = 0$ is $|{-10}|/\sqrt{1 + 4} = 10/\sqrt5 = 2\sqrt5$. It matches. Note also that $\nabla f = \langle 4,8\rangle$ at the solution is parallel to $\nabla g = \langle 1,2\rangle$, so the segment from the origin to the point is perpendicular to the line, as it should be.

Three variables with one constraint

Among positive numbers $x$, $y$, $z$ with $x + y + z = 12$, maximize the product $f = xyz$.

With $g = x + y + z$, the gradients give

$$ yz = \lambda, \qquad xz = \lambda, \qquad xy = \lambda $$

Setting the first two equal gives $yz = xz$, and since $z > 0$ we may cancel it to get $y = x$. Setting the last two equal gives $xz = xy$, and cancelling $x > 0$ gives $z = y$. So all three variables are equal, and the constraint forces $x = y = z = 4$. The product is $64$.

The constraint region here, with the positivity requirement, is not closed: pushing one variable toward zero keeps the sum at $12$ while driving the product toward $0$. So the single candidate is the maximum, and there is no minimum. This is a case where the geometry of the constraint set has to be considered and not just the algebra.

Two constraints at once

Maximize and minimize $f(x,y,z) = x + y + z$ on the curve where the cylinder $x^2 + y^2 = 2$ meets the plane $x + z = 1$.

Set $g = x^2 + y^2$ and $h = x + z$. The gradients are $\nabla f = \langle 1,1,1\rangle$, $\nabla g = \langle 2x, 2y, 0\rangle$ and $\nabla h = \langle 1, 0, 1\rangle$. The condition $\nabla f = \lambda \nabla g + \mu\nabla h$ reads

$$ 1 = 2\lambda x + \mu, \qquad 1 = 2\lambda y, \qquad 1 = \mu $$

The third equation gives $\mu = 1$, so the first becomes $2\lambda x = 0$. The second equation forbids $\lambda = 0$, since $1 = 0$ is false, so $x = 0$. The cylinder constraint then gives $y^2 = 2$ and $y = \pm\sqrt2$, and the plane constraint gives $z = 1 - x = 1$.

The two candidates are $(0, \sqrt2, 1)$ with $f = 1 + \sqrt2$ and $(0,-\sqrt2,1)$ with $f = 1 - \sqrt2$. The intersection curve is an ellipse, closed and bounded, so those are the maximum and the minimum.

Check by parametrizing: on the cylinder take $x = \sqrt2\cos t$, $y = \sqrt2\sin t$, and the plane gives $z = 1 - x$. Then $f = x + y + z = x + y + 1 - x = y + 1 = \sqrt2 \sin t + 1$, which ranges over $[1 - \sqrt2,\, 1 + \sqrt2]$. The extremes match, and the cancellation of $x$ explains why the answers came out on the $y$-axis.

Practice

Start with one constraint. Write $\nabla f = \lambda\nabla g$ component by component, eliminate the multiplier, substitute into the constraint, and compare the values of $f$ at the candidates you find.

Then two constraints. The objective’s gradient is now a combination of two constraint gradients, so the system grows by one equation and one unknown, and the constraint set is the curve where two surfaces meet.

Quiz

Five items on setting up and solving the multiplier system, identifying constrained maxima and minima, and the two-constraint version.