3 Prevalent Themes
| # | Theme | Supporting Quote |
|---|---|---|
| 1 | Prompt‑cache limits routing flexibility – The router can’t freely move a conversation between models because caches are “sticky” per session. Switching models forces the next message to be sent to the same model until the cache can be rebuilt. | “Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.” — thehamkercat |
| 2 | Cache‑hit economics of multi‑model switching – Keeping caches warm across models yields only a small hit‑rate drop but allows cheaper generations to use the cached context, providing a net cost benefit. | “typically a lot or most of the cost in “agentic” API usage is cached read + generation. Cached read costs scale with turn count, which multi‑model switching doesn't increase, and of course generation gets cheaper if you do some of it with a cheaper model.” — hedgehog |
| 3 | Model‑size vs. efficiency tension – Pushing ever‑larger models eventually runs into information‑theoretic ceilings; many argue that “small models sound really nice but at some point you're just fighting the laws of information theory.” | “Small models sound really nice but at some point you're just fighting the laws of information theory.” — jrflo |
These three motifs capture the core concerns emerging from the Hacker News thread: the practical constraints of prompt caching in router designs, the modest yet meaningful efficiency gains from keeping caches warm when swapping models, and the broader debate over whether ever‑larger LLMs are sustainable or if smaller, more efficient models represent the viable future.