BEYOND BINARY REWARDS: TRAINING LMS TO REASON ABOUT THEIR UNCERTAINTY

Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty.

Research question
The paper asks whether reasoning models can be trained to be both accurate and well calibrated, instead of becoming overconfident after reinforcement learning. More specifically, it studies whether standard binary correctness rewards are the reason RL-trained reasoning models tend to guess too confidently and whether adding calibration-aware rewards can fix that.

Methodology
The authors propose RLCR (Reinforcement Learning with Calibration Rewards), which trains a model to produce not only an answer but also a numerical confidence estimate after its reasoning chain. The reward combines ordinary correctness with a Brier score term, and the paper provides both a theoretical argument that this objective encourages calibrated confidence and empirical tests on factual QA and math tasks, including out-of-domain evaluations and comparisons to standard RLVR and post-hoc confidence models.

Findings
The main result is that RLCR substantially improves calibration without reducing accuracy, while ordinary RL with binary rewards tends to worsen calibration. On the examples highlighted in the paper, RLCR reduces expected calibration error from 0.37 to 0.03 on HotpotQA and from 0.26 to 0.10 on a collection of math datasets, and it also improves out-of-domain reliability and helps test-time confidence-weighted scaling methods.

Limitations
A main limitation is that the setup focuses on models that output a single answer and a single scalar confidence score, rather than a full probability distribution over many possible answers. The paper also studies calibration mainly in factual QA and math-style reasoning settings, so broader generalization to more open-ended agentic tasks, interactive settings, or richer uncertainty representations still remains to be tested.

Why it’s important
This paper matters because it shows that better reasoning is not just about getting more answers right, but about knowing when the model might be wrong. It is especially important for safety and reliability because it offers a simple training modification that improves both trustworthiness and downstream usefulness, rather than trading calibration away for raw task performance.