vramkit

vramkit is an open-source project providing two VRAM compression techniques for large language model inference: a lossless weight codec that achieves -33% size compared to 4-bit quantization (beating standard compressors like zstd), and a drop-in PyTorch module for KV-cache compression using per-channel quantization to reduce memory by 2× with near-lossless effect on model perplexity. It runs on ROCm/CUDA/MPS/CPU and is developed and benchmarked on consumer hardware like MacBooks.