OmnisBench

OmnisBench is an open-source tool designed to benchmark and verify large language model (LLM) routing efficiency. It allows users to assess how close routing policies approach the ideal quality-per-dollar frontier on fresh, uncontaminated datasets. The benchmark includes live model cost data and can re-grade results offline to ensure reproducibility and transparency. It supports executing model-generated code under controlled conditions with sandboxing recommendations for security.