BeelineBench

BeelineBench is a configurable benchmark for evaluating decision models' efficiency in solving multi-step problems such as puzzles, planning tasks, and word games. It scores models based on how directly they reach solutions compared to shortest paths, supporting integration with models that use Jev protocol or OpenAI's Decisions API. It includes multiple problem domains and baseline heuristics for comprehensive evaluation.