IdeaWell

Project ideas from Hacker News discussions.

We could save petabytes of cache storage with Zstandard and Pingora

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1 – Compression as a cache tier (hot vs. cold content)
The discussion repeatedly treats compression as a “cheaper but slower” storage layer, debating whether to apply it to popular (hot) assets or to rarely‑used (cold) data.
- “We initially considered limiting transcoding to popular content” – MayeulC
- “I would compress it all, and then selectively recompress at higher compression levels depending on the link, read frequency…” – genxy
- “In any cache hierarchy you want to put colder content in cheaper but slower storage. Here, compression is the cheaper but slower form of storage.” – articulatepang

Theme 2 – Handling range requests when data is compressed
Many commenters worry that compressing cached objects breaks efficient byte‑range serving, and they explore Zstd’s frame‑based or seekable formats as solutions.
- “I’m confused by how this affects range requests… The article claims ‘range requests remain unchanged’, but I don’t see how that’s possible if the cache no longer stores the uncompressed data.” – CodesInChaos
- “I assume the entire resource needs to be decompressed first, then indexed into, served, and discarded. Well, actually, you could just decompress up to the end of the range.” – pkulak
- “Not with zstd, you could still support range requests… glance over the spec and the capabilities.” – genxy
- “zstd internally splits data into frames, and frames can indicate the decompressed data size… it will not need to decompress the resource.” – kccqzy
- “Zstd has a seekable format for frames, similar to pigz --independent works.” – gopalv

Theme 3 – CPU vs. I/O/network trade‑offs and real‑world savings
Participants emphasize that Zstd’s low‑cost compression yields bandwidth and storage benefits without hurting performance, especially given skewed popularity distributions.
- “When I worked on a large CDN the content popularity distribution was heavily skewed… Spending extra processing time on 50% of your content would be wasted effort as its never read again.” – donavanm
- “Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk.” – genxy
- “Huge assets took less hard drive space, took less time to download, and took less time to decompress. The differences were not small, and resulted in appreciable improvements both in infra cost and dev productivity.” – repsilat

🚀 Project Ideas

Generating project ideas…

ZstdSeekCache – Transparent HTTP reverse proxy with seekable Zstd compression

Summary

  • A reverse proxy that caches origin responses using Zstd’s seekable frame format, enabling efficient range requests while reducing storage and bandwidth.
  • Core value proposition: faster cache fills, lower edge storage costs, and full HTTP range support without decompressing entire objects.

Details

Key Value
Target Audience CDN operators, media streaming platforms, large-scale web services
Core Feature On-the-fly Zstd compression with seekable frames + range-request handling
Tech Stack Go (net/http), zstd C bindings via cgo, seekable-format contrib module
Difficulty Medium
Monetization Revenue-ready: Usage-based pricing ($0.01/GB compressed)

Notes

  • HN commenter CodesInChaos complained about range requests after compression; this solves it directly: “Which would have terrible performance for range requests starting late in a large file.”
  • Enables the insight from genxy that Zstd can still support range requests with minimal effort, opening discussion on deploying seekable Zstd at scale.

AdaptiveZstd – Automated compression tiering based on access frequency

Summary

  • Analyses access logs to classify objects as hot/warm/cold and applies appropriate Zstd levels (3‑5 for hot, 12‑19 for cold) to balance CPU vs storage trade‑offs.
  • Core value proposition: reduces unnecessary CPU spend on rarely‑accessed data while maximizing compression savings for cold assets.

Details

Key Value
Target Audience Storage engineers, backup systems, object‑store admins (S3, GCS, etc.)
Core Feature Dynamic recompression pipeline with access‑frequency scoring
Tech Stack Python, Prometheus‑style metrics, zstd CLI, Redis for scoring
Difficulty Low
Monetization Hobby

Notes

  • Mirrors donavanm’s observation: “Spending extra processing time on 50% of your content would be wasted effort as its never read again.”
  • Would spark discussion on optimal throttling policies and could be extended to tiered storage (SSD vs HDD) as suggested by articulatepang.

DeltaZstd – Delta‑encode + Zstd compression suite for package registries & container images

Summary

  • Computes rolling‑hash chunks, delta‑encodes similar versions, then compresses with Zstd; delivers >90% size reduction for npm, PyPI, and Docker layers.
  • Core value proposition: massive storage and bandwidth savings for CI/CD pipelines and public registries with minimal runtime overhead.

Details

Key Value
Target Audience DevOps teams, registry maintainers, CI/CD platform providers
Core Feature Content‑defined chunking → delta encoding → Zstd compression
Tech Stack Rust (fast CDC), zstd-rs, optional FUSE layer for transparent access
Difficulty High
Monetization Revenue-ready: Enterprise license per node ($2k/yr) + optional hosted SaaS

Notes

  • Directly addresses r3trohack3r’s success: “I applied a similar approach to compress the npm registry by over 90% on disk … delta encode them first and then compress them.”
  • Could leverage Zstandard’s property noted by genxy: recompressing an already compressed stream short‑circuits, making pipeline safe and CPU‑light.

Read Later