FastKVComp: Checking Whether High-Throughput KV Compression Pays Off

This episode examines FastKVComp, a system that speeds up KV-cache compression for disaggregated LLM serving, where the cache built during prefill must cross a 100 to 400 gigabit link to the decode GPU. It lays out the break-even rule that compression pays only if encode plus decode time is less than the wire time saved. The compression pipeline is a randomized Hadamard transform, low-bit quantization and a lossless codec. Only quantization loses information, and the paper reports no accuracy numbers. The speedups come from fusing the transform and quantization into one tiled kernel that avoids a full-size intermediate tensor, and from replacing nvCOMP ANS with the LC codec. LC compresses less but runs faster. The discussion questions the headline claims of a 5.9x faster kernel, over 500 GB/s throughput and 2.04x better job completion time. It notes that the throughput unit is undefined and appears to be raw input bytes, and that the baseline comes from the same research group. Listeners get a practical way to check whether a compression speedup holds up on fast networks. Sources: 1. Beyond Compression Ratio: High-Throughput KV Compression for Disaggregated LLM Serving — Junhong Wu, Zedong Liu, Xinyang Ma, Yuhan Chen, Guangming Tan, Dingwen Tao, 2026 http://arxiv.org/abs/2611.04382 2. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, Junchen Jiang, 2024 https://scholar.google.com/scholar?q=CacheGen%3A+KV+Cache+Compression+and+Streaming+for+Fast+Large+Language+Model+Serving 3. Efficient Lossless Compression of Scientific Floating-Point Data on CPUs and GPUs (LC framework line of work) — Noushin Azami, Alex Fallin, Martin Burtscher (Texas State University), 2025 https://scholar.google.com/scholar?q=Efficient+Lossless+Compression+of+Scientific+Floating-Point+Data+on+CPUs+and+GPUs+%28LC+framework+line+of+work%29 4. DFloat11: Lossless Compression of LLMs for Efficient GPU Inference (and ZipNN: Lossless Compression for AI Models) — DFloat11: Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, Vipin Chaudhary, Yang Sui, Xia Hu, Anshumali Shrivastava. ZipNN: Moshik Hershcovitch et al. (IBM), 2024–2025 https://scholar.google.com/scholar?q=DFloat11%3A+Lossless+Compression+of+LLMs+for+Efficient+GPU+Inference+%28and+ZipNN%3A+Lossless+Compression+for+AI+Models%29 5. nvCOMP / GPU lossless codecs (Bitcomp, GDeflate, ANS) and the cuSZ/FZ-GPU line from the same group as KVServe — NVIDIA nvCOMP team; cuSZ/FZ-GPU: Dingwen Tao and collaborators, 2020–2024 https://scholar.google.com/scholar?q=nvCOMP+%2F+GPU+lossless+codecs+%28Bitcomp%2C+GDeflate%2C+ANS%29+and+the+cuSZ%2FFZ-GPU+line+from+the+same+group+as+KVServe 6. SplitZip: Ultra fast lossless KV compression for disaggregated LLM serving — Guo, Y., Joshi, S., 2026 https://scholar.google.com/scholar?q=SplitZip%3A+Ultra+fast+lossless+KV+compression+for+disaggregated+LLM+serving 7. KVServe: Service-aware KV cache compression for communication-efficient disaggregated LLM serving — Liu, Z., Ma, X., Luo, D., et al., 2026 https://scholar.google.com/scholar?q=KVServe%3A+Service-aware+KV+cache+compression+for+communication-efficient+disaggregated+LLM+serving 8. Efficient lossless compression of scientific floating-point data on CPUs and GPUs (LC / FZ-style pipelines) — Azami, N., Fallin, A., Burtscher, M., 2025 https://scholar.google.com/scholar?q=Efficient+lossless+compression+of+scientific+floating-point+data+on+CPUs+and+GPUs+%28LC+%2F+FZ-style+pipelines%29 9. NVIDIA nvCOMP (ANS, Bitcomp, GDeflate, LZ4) documentation and benchmarks — NVIDIA, 2023-2026 https://scholar.google.com/scholar?q=NVIDIA+nvCOMP+%28ANS%2C+Bitcomp%2C+GDeflate%2C+LZ4%29+documentation+and+benchmarks 10. Lossless compression of BF16 weights: DFloat11 and ZipNN — Zhang et al. (DFloat11); Hershcovich et al. (ZipNN), 2025 https://scholar.google.com/scholar?q=Lossless+compression+of+BF16+weights%3A+DFloat11+and+ZipNN 11. Mooncake: Trading more storage for less computation — Qin, R., et al., 2025 https://scholar.google.com/scholar?q=Mooncake%3A+Trading+more+storage+for+less+computation 12. KVQuant and KIVI — Hooper et al. 2024; Liu et al. 2024, 2024 https://scholar.google.com/scholar?q=KVQuant+and+KIVI Interactive Visualization: FastKVComp: Checking Whether High-Throughput KV Compression Pays Off