Rare Failure Eval

A Python-based reproducible experimentation tool that simulates agent evaluation scenarios, analyzing rare failures, selection errors, and regret metrics under various risk settings and budget constraints. It provides a framework for understanding the tradeoffs between selection accuracy and utility loss in agent evaluation, useful for developers and researchers testing algorithms and policies.