LocusBlog

Why we rate problems like chess players

A student sits down to practice quadratics. The worksheet has twenty problems. Two are trivial for her. Three are out of reach. The other fifteen sit somewhere in between, and nobody knows where. That is the normal state of a math worksheet.

Locus gives every student a rating and every problem a rating. Same scale. Same units. When a student attempts a problem in ranked mode, both ratings move. It is the Elo system from chess, pointed at a different kind of match.

The idea is not ours. Klinkenberg, Straatemeier and van der Maas built it into Math Garden and published the results in 2011. Their system targeted a 0.75 success rate and ran on 3,648 children answering 3.5 million arithmetic problems over ten months. Pelanek surveyed the approach across adaptive educational systems in 2016 and recommended it.

One update, worked out

Here is a problem.

$$\text{Solve for } x\text{:} \quad \frac{1}{x - 1} + \frac{1}{x + 1} = 1$$

The answer is $x = 1 + \sqrt{2}$ or $x = 1 - \sqrt{2}$. A student types that free-form. A computer algebra system checks it for equivalence, so 1+sqrt(2) and 1+2^(1/2) both pass. A grader that accepts one exact string rates typing, not algebra.

Say the problem sits at 1600 and the student sits at 1200. The expected score is the standard Elo formula:

$$E = \frac{1}{1 + 10^{(1600 - 1200)/400}} = \frac{1}{11} = 0.091$$

She has about a 9 percent chance. If she solves it, with $K = 32$ (Locus scales $K$ up a little for a fast correct answer, which we ignore here):

$$\text{student:} \quad 1200 + 32 \times (1 - 0.091) = 1229$$

She gains 29 points. If she misses it:

$$\text{student:} \quad 1200 + 32 \times (0 - 0.091) = 1197$$

She loses 3 points.

The problem’s rating does not move on her attempt. A problem is rated against many solvers, so a 32-point swing on one cohort would make its rating jitter instead of converge. Locus keeps the author’s difficulty as an anchor and runs a nightly recalibration: it collects the ranked attempts on each problem, compares what happened to what the ratings predicted, and nudges a separate calibrated rating by at most 8 points per sweep, never more than 300 points from the authored value.

The asymmetry is the design. Missing a problem four hundred points above you costs almost nothing. Beating it moves you a lot. Now compare an even match. Put this problem near 1200:

$$\text{Solve for } x\text{:} \quad \frac{x + 3}{x - 2} = 5$$

The answer is $x = \frac{13}{4}$. Against a 1200 student, $E = 0.5$, and the update is plus or minus 16 either way. A coin flip is worth more information than a blowout, so it moves the rating more.

The problem side is where the rating earns its keep. Pelanek’s guidance is to set a new item’s difficulty to a neutral value and let student answers calibrate it. Locus is more conservative: the author’s difficulty is the starting point and the calibration is bounded. The effect is the same in direction. When we add a problem that we think is easy and it turns out to eat 1500-rated students, its calibrated rating climbs a few points a night until the corpus says so, and nobody has to file a bug. The bound means a wrong authored difficulty can be off by 300 points and no more.

What the research supports

The interesting question is not how to update a rating. It is what gap to aim for when picking the next problem.

Math Garden aimed at a 0.75 success rate. Pelanek describes a geography practice system that also uses a target success rate, with 75 percent as the example. Wilson, Shenhav, Straccia and Cohen approached the same question from the other end in 2019. For a broad class of stochastic-gradient-descent learners on binary classification tasks, they derive an optimal training error rate of 15.87 percent, so accuracy near 85 percent.

That result is narrower than the headline. It assumes Gaussian noise. The authors report the optimum shifts to 82 percent under Laplacian noise and 75 percent under Cauchy. It is a theorem about gradient descent, not a law about children. Read it as support for a range rather than a constant.

Translated to rating points, the range is easy to state. A problem 300 points below a student gives an 85 percent expected score. A problem 190 points below gives 75 percent. So the target band is roughly 190 to 300 points below the student’s rating. Locus does not aim there yet. The ranked draw centers on the student’s rating and shifts it by a random offset of up to 200 points either way, so many draws land inside that band and some land above it. The band is the target the research gives us. The draw is a first approximation of it, and we have not tuned the offset against our own outcome data.

Bjork and Bjork supply the reason the band has a floor. Difficulties are desirable because they trigger the encoding and retrieval processes that support learning. Their limit is explicit: if the learner lacks the background knowledge or skills to respond successfully, the difficulty becomes undesirable. They name adaptive schedules tuned to a learner’s past successes as a good instance of getting this right.

They also make a point worth handing to any teacher who reads student feedback. Conditions that make performance improve fastest during practice often fail to support long-term retention. A student reporting that a session felt hard is not evidence that the difficulty was set wrong. It may be evidence of the opposite.

Where it breaks

One number per student is a strong assumption. In Pelanek’s formulation the Elo update shares its functional form with the Rasch model from item response theory, differing in how parameters are estimated. Rasch assumes a single skill. A student can be fluent at factoring and lost at logarithms, and one rating averages that into a lie. Per-topic ratings reduce the damage. They do not remove it, because topics are a human-drawn boundary and skills do not respect it.

Cold start is real. A new problem’s rating is noise for its first attempts, and the fix is a tuning problem with no clean answer. Pelanek states it plainly: a small $K$ converges too slowly, a large $K$ is unstable because it overweights the last few attempts. His recommended starting point is an uncertainty function that shrinks the update as evidence accumulates, $U(n) = \frac{a}{1 + bn}$, rather than a fixed $K$. New problems and new students both move fast, then settle.

The thing Elo does that item response theory does not is track a moving target. IRT models were built for testing, where skill is assumed constant across the test. A student practicing is a student whose skill is changing during the measurement. That is the case Elo was built for, and it is the reason to prefer it here even where an IRT model would fit the data slightly better.

The last limit is the one worth being honest about. A rating records whether an answer was right. It records nothing about why a wrong answer was wrong. A student who drops a negative sign and a student who does not know what a rational equation is both score zero, and both move the same number of points. Elo puts a student in front of the right problem. It does not tell you what to say when they miss it.

That is also why practice mode on Locus is unrated. Rating every attempt turns practice into performance, and a student who is afraid of losing points will choose problems she can already do. The band we spent all this arithmetic locating is the one where she misses roughly one attempt in four, and nobody walks into that band voluntarily while the scoreboard is running.

References