What is a reported 0.88 grading correlation worth? A reanalysis of the BeSTraP short-answer dataset

By Ben Schmidt · Oakmont Lab · pre-registration locked before analysis

The finding in one sentence. A fine-tuned grader reported at Pearson 0.88 with human short-answer grades holds up on every mechanism the released data lets us check: it is a competent grader, less consistent than a human rater (about 0.97) but comparable to human absolute agreement (about 0.87), and the usual clustering-inflation critique does not apply because the questions differ little in difficulty. Its one real limitation is that neither headline correlation can be recomputed from the release, because the model predictions were never included.

We set out to pressure-test a published number, not to rebut it, and it survived. What follows is what the released data does and does not let that number mean, and two reading habits that travel to any automated-grading result. Those two habits, written for practitioners rather than analysts, are the subject of a companion guide.

The source is the publicly released BeSTraP dataset (Khalf et al., Data 2026, 11(3), 57), which reports a fine-tuned GPT-2 grader reaching Pearson 0.8813 with human grades on 651 short-answer responses across 7 exam questions, alongside a human inter-rater reliability of ICC(3,k) = 0.979. The authors released the dataset openly, which is what makes this reanalysis possible. All numbers below are computed from those released files. Intervals are 94% posterior credible intervals from Bayesian models with weakly informative priors.

A correlation with human grades cannot rise higher than the humans agree with each other. The human scores are themselves noisy, so there is a hard ceiling on how well any grader can correlate with them. Where 0.88 falls against that ceiling is what the number tells you.

The human labels are reliable: our variance-components model recovers the reported ICC(3,k) = 0.979 as a posterior of 0.979 [0.976, 0.982], an exact reproduction. Against the like-for-like human benchmark the grader’s standing depends on the frame. In the consistency frame the grader (0.88) is clearly below a fresh human (0.97 [0.965, 0.974]). In the absolute-agreement frame it is comparable (0.88 versus 0.87 [0.46, 0.97]), though that benchmark is weakly identified. So it is less consistent than a human rater, roughly matches human absolute agreement, and carries wide uncertainty either way.

Nothing here contradicts 0.88. The posterior probability that 0.88 exceeds the strict consistency ceiling is 0.00; in the absolute frame it is 0.12, not negligible but not diagnostic on its own.

When responses are nested in a handful of questions, a pooled correlation can be inflated by differences between the questions rather than skill within them. That critique is often correct. Here it does not apply, and only measuring shows which case you are in.

The release contains no model predictions to hold out, so the question of how the grader would fare on new questions cannot be answered by cross-validation. Under a within-question-skill model, the identity r = ICC_q + s(1 - ICC_q) places the implied skill at s = 0.87, and new-question performance at 0.85 across all 28 leave-question-out splits, 0.83 by posterior prediction. Two cautions are essential. First, that small loss is close to an algebraic consequence of the model: with between-question variance the only channel for loss, and that variance small, little can be lost. Second, those intervals carry sampling uncertainty over 7 questions but no model-misspecification uncertainty. If a real grader behaves like this model, its generalization loss is small. The release cannot confirm that it does.

The released files contain human grades, raw answers, and model feedback-quality evaluations, but not the fine-tuned model predictions behind the 0.88, nor the answer-grading scores behind the separately reported 0.6166. The only reported statistic that uses released quantities, the human ICC, we recover exactly. This is a release-completeness gap common across dataset papers, and it is worth stating plainly as a field-level lesson: a dataset paper that reports a headline correlation should ship the predictions behind it, so the headline can be recomputed rather than only bounded.

• Reading 0.88 on this dataset against the human ceiling rather than against an error-free standard: the agreement of a competent grader somewhat less consistent than the human raters, not a near-perfect one.
• The general practice of reporting an automated grader’s agreement against the inter-human ceiling, and of measuring between-cluster variance before invoking a clustering-inflation critique.

• Any claim that the reported 0.88 is wrong or inflated. On the mechanisms we can check, it is not contradicted.
• The generalization estimate as an empirical measurement. It is model-conditional; the release contains no predictions to cross-validate.
• Any ranking of this grader against the zero-shot graders the source also reports. That comparison is outside our scope.
• Treating the human reliability as precisely 0.979 for a single rater. The well-estimated figure is the 2-rater average in its consistency form.

All inference is Bayesian with weakly informative priors; intervals are 94% posterior credible intervals. The between-question structure comes from a one-way multilevel model on the human average; the ceiling from a two-way variance-components model; the generalization from an enumerated leave-question-out analysis inside a within-question-skill model. The analysis is deterministic up to Monte Carlo error: the same vendored files, the same code, and the same seed reproduce every value here.

The reproducer, the pre-registered plan (locked before analysis), the tidy data, the chart series above, and a runnable notebook are archived as a sealed reproducer repository with a permanent DOI, 10.5281/zenodo.21677950. The pre-registration was committed before the numbers were computed; the human-ceiling analysis was registered before it was run; one generalization model was replaced, before reporting, after it was found internally inconsistent, and the substitution is confirmed analytically so it does not select an outcome.

Khalf et al. (2026). BeSTraP short-answer grading dataset. Data 11(3), 57. https://doi.org/10.3390/data11030057

The reported figures reanalysed here (fine-tuned GPT-2 Pearson 0.8813, human ICC(3,k) 0.979) are theirs; the reanalysis, the credible intervals, and the reading method are ours.

Cite this reanalysis. Schmidt, B. (2026). What is a reported 0.88 grading correlation worth? A reanalysis of the BeSTraP short-answer dataset. Oakmont Lab. https://doi.org/10.5281/zenodo.21677950

Oakmont Lab is Ben Schmidt’s independent research imprint. Ben is the founder of HeyLoopy; the Lab is independent of it, and no finding is commissioned. Corrections and data requests are welcome.