galahad-kv

galahad-kv is a public package that saves the key-value state of large language models in encrypted local NVMe storage to enable reuse of computed states for blocks of around 16,000 tokens, allowing for a 50-million-token context window without recomputation. This approach significantly speeds up processing, reduces GPU energy usage, and maintains flat memory usage over large token streams. It is compatible with vLLM and models like Gemma 4 12B and 31B.