Prabhav Singh

🚨 We introduce the problem of persistent calibration for trustworthy, continually learning models and find substantial headroom even for confidence estimators that appear well-calibrated at a single point in time. As a model gains more knowledge, we would like its confidence to track that change in knowledge, without requiring repeated re-calibration. To test how existing methods do, we evaluate persistent calibration on checkpoints of open models, asking whether confidence estimators trained on earlier checkpoints can generalize to later ones, whose knowledge has changed. To measure the dependence of confidence on knowledge, we evaluate calibration on knowledge contrast sets: subsets containing questions that one checkpoint answers correctly and another checkpoint answers incorrectly, reflecting a change in knowledge. Key findings: 1️⃣ Both inference-time and fine-tuning methods fall short on contrast-set calibration, even when they are well-calibrated on the full set. We quantify the headroom with oracle methods trained on future checkpoints – we call this an oracle because persistent calibration requires learning to estimate confidence in a way that generalizes to future checkpoints, whereas the oracle trains directly on them. 2️⃣ Multi-checkpoint training helps in both the non-oracle and oracle settings, suggesting an avenue for identifying confidence features that remain robust across changing knowledge. 3️⃣ Our analysis shows that two confidence adapters trained on different checkpoints can perform similarly on one checkpoint, but extremely differently on another checkpoint. This suggests that there is a vast space of possible confidence functions that are well-calibrated on a given checkpoint, out of which only some rely on meta-knowledge features that would generalize to other checkpoints, presenting a challenge for persistent calibration.