IdeaWell

Project ideas from Hacker News discussions.

How Uber Protects Against Retry Storms

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1 – Bound retry amplification (budgets, token buckets, “no‑retry” signals)
Many commenters argued that the key to preventing retry storms is to limit how much extra load retries can add, either locally or via propagated hints.
- “Your service that retries should have some retry budget… only allowing 20% more requests per second as retries, total.” – CBLT
- “Whenever a service retries but has to give up, the error it sends to its callers should never be retried… prevents the multiplicative factor of retry on top of retry.” – CBLT (echoed by jeffbee and otterley)
- “Just limiting your retry budget to 1% of normal rates using a client‑local token bucket … will eliminate the possibility of long‑lived retry storms.” – jeffbee

Theme 2 – Exponential backoff/jitter is a starting point, not a complete solution
Participants agreed that basic backoff helps but must be combined with smarter signals (circuit breakers, load shedding, error budgets, status headers).
- “Exponential back off and jitter are the first things to work on, and good if you don’t have a better signal… also, a simple signal status server or queue system helps… ” – sroussey
- “Just throwing exponential backoffs at the retry problem is not a magic solution.” – anonymars
- “Combine that with exp backoff in the caller and you got yourself a pretty robust starting point” – maxchisto (referring to load shedding)

Theme 3 – Nuance matters: avoid one‑size‑fits‑all policies; weigh developer vs. user experience
Several users warned against blanket rules (e.g., “never retry” or “always retry”) and stressed context‑aware handling.
- “This is trading a good developer experience for a bad user experience… Lack of considering nuance for your situation is just intellectual laziness.” – applfanboysbgon
- “The described situation is an awful user experience… Simply adding a retry and calling it a day sounds like the easy developer experience at the expense of the user experience.” – anonymars
- “You should absolutely consider the impact and what will happen when they go wrong, but the end takeaway to just never engage with them … is … lazy and bad advice.” – applfanboysbgon (responding to the “never retry” stance)

🚀 Project Ideas

Retry Budget Manager Library

Summary

  • A lightweight client‑side library that enforces a per‑service retry budget using a token bucket, preventing retry storms while keeping simple exponential backoff and jitter.
  • Core value: gives developers a “middle path” between retry‑everywhere and never‑retry, automatically bounding extra load from retries.

Details

Key Value
Target Audience Backend engineers building microservices (Java, Go, Node.js, Python)
Core Feature Token‑bucket based retry limiter with configurable budget, per‑request retry count, jitter, and optional circuit‑breaker fallback
Tech Stack Language‑specific adapters (e.g., Go middleware, Java interceptor, Python decorator); optional sidecar for metrics export
Difficulty Medium
Monetization Hobby

Notes

  • HN users praised the idea of a “retry budget” (CBLT: “Your service that retries should have some retry budget… This was critical as it bounds the additional load from retries”).
  • Provides a concrete, drop‑in solution that can be discussed in retry‑storm threads and adopted immediately in existing services.

Retry‑Aware Error Propagation Middleware

Summary

  • Middleware (Envoy filter, NGINX module, or language‑specific wrapper) that tracks retry attempts and injects a “no‑retry” signal (custom header or gRPC error detail) when a service has exhausted its retries.
  • Core value: stops the multiplicative retry effect upstream, addressing the pain point of cascading retry storms.

Details

Key Value
Target Audience Platform/infra teams managing service meshes or API gateways
Core Feature Automatically appends X-No-Retry: true (or google.rpc.RetryInfo with zero backoff) to error responses after a configurable retry threshold
Tech Stack Envoy Lua filter, NGINX unit module, or Go/Java middleware; uses Prometheus for retry‑count metrics
Difficulty Medium
Monetization Hobby

Notes

  • Commenters liked propagating “no‑retry” hints (CBLT: “Whenever a service retries but has to give up, the error it sends to its callers should never be retried… prevents the multiplicative factor”).
  • Offers a practical way to implement that advice without changing every service’s business logic, sparking discussion on best‑practice headers.

Adaptive Retry Policy Service

Summary

  • A centralized, low‑latency service that monitors global error rates, latency, and load, then serves dynamic retry policies (backoff base, max retries, token‑bucket size) to clients via a simple HTTP/gRPC endpoint.
  • Core value: replaces static, hard‑coded retry configs with adaptive, data‑driven guidance, reducing the need for manual tuning and preventing storms during degradation.

Details

Key Value
Target Audience SREs and DevOps teams operating large‑scale distributed systems
Core Feature Policy API (GET /retry-policy?service=X) returning JSON with backoff parameters; clients adjust their retry logic in real time
Tech Stack Python/FastAPI API, Redis for sliding‑window error counters, Prometheus scraping, optional Envoy sidecar for client‑side fetch
Difficulty High
Monetization Revenue-ready: tiered SaaS plan (free tier for low QPS, paid for higher throughput and SLA)

Notes

  • Users asked for a “simple signal status server or queue system” to keep global state (sroussey: “If you have a central error rate server you can skip your retry based on the error rate”).
  • Provides exactly that, offering a concrete tool that would generate lively HN discussion on adaptive retry strategies and operational best practices.

Read Later