DoubtBench

DoubtBench is a benchmark that evaluates AI decision models not only on accuracy but also on how well their uncertainty estimates align with human disagreement. It leverages multiple human annotator ratings to test whether models appropriately hesitate on ambiguous cases and supports scoring models on metrics like calibration, disagreement correlation, and accuracy.