LocusBlog

Standard deviation

Standard deviation is the square root of the variance, and variance is the mean of the squared deviations from the mean. Written out, the standard deviation of a whole population of $N$ values is

$$\sigma = \sqrt{\frac{1}{N}\sum_{i=1}^{N}(x_i - \mu)^2}$$

and for a sample of $n$ values drawn from a larger population it is

$$s = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})^2}$$

Two formulas, one idea, and a denominator that differs by one. OpenStax Introductory Statistics puts both in section 2.7. The difference is small on large data sets and large on small ones, and choosing the wrong one is the most common way this calculation goes wrong.

Why squares, and why the root comes back off

The obvious measure of spread is the average distance from the mean. It is useless, because it is always zero:

$$\sum_{i=1}^{n}(x_i - \bar{x}) = \sum x_i - n\bar{x} = n\bar{x} - n\bar{x} = 0$$

That is not a coincidence about a particular data set. It is what defining $\bar{x}$ as $\frac{1}{n}\sum x_i$ guarantees. Positive and negative deviations cancel exactly, so something has to remove the signs before averaging.

Squaring is the choice that survives. It kills the sign, it is differentiable everywhere where the absolute value is not, and it makes variances of independent quantities add, which is the property the rest of statistics is built on. What squaring costs is interpretability: if the data is in centimetres, every deviation is in centimetres, and every squared deviation is in centimetres squared. Variance carries that unit. A variance of 4 cm$^2$ cannot be marked on the same axis as a mean of 5 cm, and it cannot be compared to a range in centimetres.

Taking the square root puts the unit back. Standard deviation is in centimetres, sits on the same axis as the mean, and can be read as a typical distance from centre. That is the only reason the root is there, and it is enough of a reason.

What the $n-1$ is correcting

Deviations in the sample formula are measured from $\bar{x}$, not from the true population mean $\mu$, because $\mu$ is unknown. There is an exact identity relating the two sums:

$$\sum(x_i - \mu)^2 = \sum(x_i - \bar{x})^2 + n(\bar{x} - \mu)^2$$

The cross term vanishes because $\sum(x_i - \bar{x}) = 0$, the fact from the last section. Since the added piece is a square, it is never negative: the sum of squared deviations about $\bar{x}$ is always the smaller of the two. That is the heart of it. The sample mean is, by construction, the number that minimises the sum of squared deviations for that particular sample, and it was computed from the same data it is being compared against, so it sits closer to the data than the true mean does. Dividing that too-small sum by $n$ therefore underestimates $\sigma^2$ systematically, not occasionally. Take expectations to see by how much. The first expectation is $E\big[\sum(x_i - \mu)^2\big] = n\sigma^2$, and the second is

$$E\big[n(\bar{x} - \mu)^2\big] = n\operatorname{Var}(\bar{x}) = n \cdot \frac{\sigma^2}{n} = \sigma^2$$

Subtracting leaves $E\big[\sum(x_i - \bar{x})^2\big] = n\sigma^2 - \sigma^2 = (n-1)\sigma^2$. Dividing by $n-1$ instead of $n$ cancels exactly that factor, which is Bessel’s correction.

Now the part that is usually stated too strongly. The correction makes $s^2$ an unbiased estimator of $\sigma^2$. It does not make $s$ an unbiased estimator of $\sigma$, because the square root is a concave function and the expectation of a concave function is not the function of the expectation. $E[s] < \sigma$ still, for any $n$. For normally distributed data the shortfall at $n = 8$ is about 3.5 percent, and there is no fix by changing the denominator to another whole number, because the size of the remaining bias depends on the shape of the distribution. The honest summary is that $n-1$ fixes the variance and leaves the standard deviation slightly low, and everyone uses it anyway because the alternative is worse.

The same eight numbers, computed both ways

Take the data set 2, 4, 4, 4, 5, 5, 7, 9. The sum is 40 and there are 8 values, so the mean is $40/8 = 5$.

Deviations from the mean, in order: $-3$, $-1$, $-1$, $-1$, $0$, $0$, $2$, $4$. They sum to zero, which is the first check and catches an arithmetic slip in the mean.

Squared deviations: 9, 1, 1, 1, 0, 0, 4, 16. Their sum is 32. Everything below comes from that one number.

Treating the eight values as the entire population:

$$\sigma^2 = \frac{32}{8} = 4, \qquad \sigma = \sqrt{4} = 2$$

Treating them as a sample of eight drawn from something larger:

$$s^2 = \frac{32}{7} = 4.571429, \qquad s = \sqrt{4.571429} = 2.138090$$

The sum of squares can be had a second way, without computing a single deviation. Expanding $\sum(x_i - \bar{x})^2$ and simplifying gives the computational formula:

$$\sum(x_i - \bar{x})^2 = \sum x_i^2 - \frac{\left(\sum x_i\right)^2}{n}$$

Here $\sum x_i^2 = 4 + 16 + 16 + 16 + 25 + 25 + 49 + 81 = 232$ and $\left(\sum x_i\right)^2/n = 40^2/8 = 1600/8 = 200$, so the sum of squares is $232 - 200 = 32$. Same 32, and it arrived without the mean ever being subtracted from anything. That is why the form is worth knowing: it needs one pass over the data accumulating $\sum x$ and $\sum x^2$, and it does not propagate rounding error from a non-terminating mean. On a data set whose values are large and close together it can lose precision to cancellation, so it is a convenience, not an upgrade.

Using $n$ on a sample

Suppose those eight numbers are a sample and the population divisor gets used by habit. The reported standard deviation is 2 instead of 2.138090, and the reported variance is 4 instead of 4.571429.

The size of the error is fixed by $n$ alone. The variance comes out low by the factor $\frac{n-1}{n}$, which at $n = 8$ is $7/8$, so the variance is 12.5 percent too small. The standard deviation is low by $\sqrt{7/8} = 0.935414$, about 6.5 percent. At $n = 5$ the variance would be 20 percent low; at $n = 100$, one percent. So the mistake matters most exactly where small samples are common, and it fades where nobody would have worried about it.

The rule for which to use is not about the size of the data set. It is about what the data set is. Eight measurements that are all the objects there are - eight planets, the eight members of a committee - are a population, and $n$ is correct. Eight measurements standing in for a larger group you did not measure are a sample, and $n-1$ is correct. Calculators encode this as $\sigma_x$ versus $s_x$, and spreadsheets as STDEV.P versus STDEV.S; the two buttons are not interchangeable and the labels are the only warning you get.

From here the standard deviation goes straight into the z-score $(x - \mu)/\sigma$, into the empirical rule about how much of a bell-shaped data set lies within one, two and three standard deviations, and into the standard error $\sigma/\sqrt{n}$, which is the standard deviation of the sample mean rather than of the data. Each of those is the same quantity wearing a different job. Practice on standard deviation and variance mixes population and sample wording deliberately, and it is worth being fast at the mean first, since every deviation in the calculation depends on getting it right.

References