IdeaWell

Project ideas from Hacker News discussions.

Speculative KV coding: losslessly compressing KV cache by up to ~4×

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

1.Predictive delta compression enables larger effective context

Storing only the differences (deltas) between the real and predicted KV cache lets the model reuse a tiny buffer instead of keeping the full cache, making it possible to fit huge contexts (e.g., 200‑250 k tokens) on a 24 GB GPU alongside a 27 B model.

"Tiny deterministic model predicts the K/V cache, prediction is compared with reality, delta is stored in vram. … predicts the values again, applies the delta, and you have the full correct value while just storing the delta" — porridgeraisin
"The tradeoff gets better the bigger your primary model … The KV cache can consume a lot of expensive VRAM" — wongarsu

2. Practical limits and skepticism about compression benefits

Although the idea sounds attractive, recomputing the entire KV cache for each step is still quadratic in context length and often outweighs the VRAM savings, making the approach worthwhile only for very large models or high‑throughput serving.

"You can use the original model to compress the kv cache and get ∞x compression, since the prediction is perfect. The cost is time, and I don't see how this could be worth it." — 0-_-0
"Even recomputing a 'draft' of the KV cache is still quadratic in context length… that's even worse." — zozbot234

3. Viewing speculation as a first‑class model primitive

Discussion shifts from a one‑off hack to treating speculative prediction as a fundamental inference primitive that can be recursed on, distilled, or aligned architecturally for even greater efficiency.

"If “speculative” approach works so well in different contexts why not make it first class and use everywhere, possibly recursively?" — mirekrusin
"Speculation is only worth it if you can profit from it. Not every context allows this or has a similar idea of what can be speculated." — saagarjha

🚀 Project Ideas

[DeltaKV Compressor]

Summary

  • [An open‑source library that predicts and stores KV cache deltas, allowing 128k‑plus contexts to run on 24 GB GPUs by keeping only compressed deltas.]
  • [Core value: reduces VRAM usage by up to 10× while preserving full‑precision decoding.]

Details

Key Value
Target Audience LLM developers, inference engineers, researchers needing long context on limited GPU memory
Core Feature Real‑time delta prediction and sparse cache reconstruction API
Tech Stack Rust core, PyTorch or ONNX predictor, CUDA kernels, optional WebGPU front‑end
Difficulty Medium
Monetization Revenue-ready: SaaS Cloud API tiering $0.02 per GB‑hour of delta storage

Notes

  • [HN commenters praised the idea of “compressing KV cache with deltas” and asked for a practical tool to try it.]
  • [Could spark discussion on speculative caching and open a marketplace for cache‑optimized models.]

[SpeculativeCache Optimizer]

Summary

  • [A managed cloud inference optimizer that automatically selects and serves the smallest predictor model needed for delta‑based KV cache compression, automatically scaling with primary model size.]
  • [Core value: lets any LLM provider get massive context windows without manual model engineering.]

Details

Key Value
Target Audience Cloud ML platform operators, SaaS founders
Core Feature Adaptive predictor selection & on‑the‑fly delta compression endpoint
Tech Stack FastAPI backend, TorchServe predictor models, Redis cache, GPU‑accelerated delta encoder
Difficulty High
Monetization Revenue-ready: Pay‑per‑token‑served pricing $0.001 per 1k context‑tokens processed

Notes

  • [HN users mentioned “this could make a lot of sense for serving a 1T model with 16 concurrent requests,” indicating clear market need.]
  • [Offers a discussion hook around dynamic speculation primitives and possible integration with existing inference stacks.]

[RecursiveCache Engine]

Summary

  • [A command‑line tool that implements recursive speculative caching, allowing developers to generate and apply delta “building blocks” for any text‑generation model, enabling reuse of early token contexts across long sessions.]
  • [Core value: turns large historic chats into cheap reusable KV blocks, cutting VRAM per request dramatically.]

Details

Key Value
Target Audience Power users, chatbot developers, LLM hobbyists
Core Feature Recursive delta generation and replay API
Tech Stack Python, JAX/Transformers, NumPy cache format, optional Docker
Difficulty Low
Monetization Hobby

Notes- [One HN comment likened the concept to “LRU‑eviction is just a speculative model,” showing community interest in data‑as‑function primitives.]

  • [Could lead to debates about live distillation and whether speculative primitives should become first‑class language features.]

Read Later