InferPilot

InferPilot is an open-source tool designed for diagnosing and optimizing Large Language Model (LLM) inference workloads using vLLM. It analyzes live metrics to determine actual bottlenecks, recommend configuration changes like fp8 KV cache, and abstains when insufficient evidence exists. It aids operators in making informed decisions on performance improvements without running full benchmarks or needing GPUs locally.