LocusBlog

Why we rate problems like chess players

A student sits down to practice quadratics. The worksheet has twenty problems. Two are trivial for her. Three are out of reach. The other fifteen sit somewhere in between, and nobody knows where. That is the normal state of a math worksheet.

Locus gives every student a rating and every problem a rating. Same scale. Same units. When a student attempts a problem in ranked mode, both ratings move. It is the Elo system from chess, pointed at a different kind of match.

The idea is not ours. Klinkenberg, Straatemeier and van der Maas built it into Math Garden and published the results in 2011. Their system targeted a 0.75 success rate and ran on 3,648 children answering 3.5 million arithmetic problems over ten months. Pelanek surveyed the approach across adaptive educational systems in 2016 and recommended it.

One update, worked out

Here is a problem.

Solve for x:  1/(x - 1) + 1/(x + 1) = 1

The answer is x = 1 + sqrt(2) or x = 1 - sqrt(2). A student types that free-form. A computer algebra system checks it for equivalence, so 1+sqrt(2) and 1+2^(1/2) both pass. A grader that accepts one exact string rates typing, not algebra.

Say the problem sits at 1600 and the student sits at 1200. The expected score is the standard Elo formula:

E = 1 / (1 + 10^((1600 - 1200) / 400)) = 1 / 11 = 0.091

She has about a 9 percent chance. If she solves it, with K = 32:

student:  1200 + 32 * (1 - 0.091) = 1229
problem:  1600 + 32 * (0.091 - 1) = 1571

She gains 29 points. The problem loses 29. If she misses it:

student:  1200 + 32 * (0 - 0.091) = 1197
problem:  1600 + 32 * (0.091 - 0) = 1603

She loses 3 points.

The asymmetry is the design. Missing a problem four hundred points above you costs almost nothing. Beating it moves you a lot. Now compare an even match. Put this problem near 1200:

Solve for x:  (x + 3)/(x - 2) = 5

The answer is x = 13/4. Against a 1200 student, E = 0.5, and the update is plus or minus 16 either way. A coin flip is worth more information than a blowout, so it moves the rating more.

The problem half of that update matters more than the student half. Nobody has to guess how hard a problem is. Pelanek’s guidance is to set a new item’s difficulty to a neutral value and let student answers calibrate it. Difficulty stops being an author’s opinion and becomes a measurement. When we add a problem that we think is easy and it turns out to eat 1500-rated students, the rating says so within a few hundred attempts, and nobody has to file a bug.

What the research supports

The interesting question is not how to update a rating. It is what gap to aim for when picking the next problem.

Math Garden aimed at a 0.75 success rate. Pelanek describes a geography practice system that also uses a target success rate, with 75 percent as the example. Wilson, Shenhav, Straccia and Cohen approached the same question from the other end in 2019. For a broad class of stochastic-gradient-descent learners on binary classification tasks, they derive an optimal training error rate of 15.87 percent, so accuracy near 85 percent.

That result is narrower than the headline. It assumes Gaussian noise. The authors report the optimum shifts to 82 percent under Laplacian noise and 75 percent under Cauchy. It is a theorem about gradient descent, not a law about children. Read it as support for a range rather than a constant.

Translated to rating points, the range is easy to state. A problem 300 points below a student gives an 85 percent expected score. A problem 190 points below gives 75 percent. So the target band is roughly 190 to 300 points below the student’s rating. That is the number our selection logic cares about, and it is narrow enough to be useful and wide enough that we do not pretend to know the exact optimum.

Bjork and Bjork supply the reason the band has a floor. Difficulties are desirable because they trigger the encoding and retrieval processes that support learning. Their limit is explicit: if the learner lacks the background knowledge or skills to respond successfully, the difficulty becomes undesirable. They name adaptive schedules tuned to a learner’s past successes as a good instance of getting this right.

They also make a point worth handing to any teacher who reads student feedback. Conditions that make performance improve fastest during practice often fail to support long-term retention. A student reporting that a session felt hard is not evidence that the difficulty was set wrong. It may be evidence of the opposite.

Where it breaks

One number per student is a strong assumption. In Pelanek’s formulation the Elo update shares its functional form with the Rasch model from item response theory, differing in how parameters are estimated. Rasch assumes a single skill. A student can be fluent at factoring and lost at logarithms, and one rating averages that into a lie. Per-topic ratings reduce the damage. They do not remove it, because topics are a human-drawn boundary and skills do not respect it.

Cold start is real. A new problem’s rating is noise for its first attempts, and the fix is a tuning problem with no clean answer. Pelanek states it plainly: a small K converges too slowly, a large K is unstable because it overweights the last few attempts. His recommended starting point is an uncertainty function that shrinks the update as evidence accumulates, U(n) = a / (1 + bn), rather than a fixed K. New problems and new students both move fast, then settle.

The thing Elo does that item response theory does not is track a moving target. IRT models were built for testing, where skill is assumed constant across the test. A student practicing is a student whose skill is changing during the measurement. That is the case Elo was built for, and it is the reason to prefer it here even where an IRT model would fit the data slightly better.

The last limit is the one worth being honest about. A rating records whether an answer was right. It records nothing about why a wrong answer was wrong. A student who drops a negative sign and a student who does not know what a rational equation is both score zero, and both move the same number of points. Elo puts a student in front of the right problem. It does not tell you what to say when they miss it.

That is also why practice mode on Locus is unrated. Rating every attempt turns practice into performance, and a student who is afraid of losing points will choose problems she can already do. The band we spent all this arithmetic locating is the one where she misses roughly one attempt in four, and nobody walks into that band voluntarily while the scoreboard is running.

References