IdeaWell

Project ideas from Hacker News discussions.

Janas: Run LLMs larger than RAM by streaming experts from SSD

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

1. Token throughput matters differently for generation vs. input (especially for agentic work)
- “60 toks/sec may be ok for pure chat or token generation, but not for token preprocessing nor any sort of agentic work.” — kolinko
- “Most of the tokens consumed by agents is reading stuff – docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens.” — kolinko
- “My self hosted LLM runs at 60‑ish tok/s and does great at agentic tasks. I think that’s about the threshold where it stops feeling annoyingly slow.” — teaearlgraycold

2. Hardware‑storage innovations (NVMe, high‑bandwidth flash, GPU‑NVMe coupling) are seen as key to lifting LLM performance
- “I wonder when we will start seeing small commercial accelerators integrated with NVMes; could potentially use far more bandwidth than typical NVMe drive can push thru the four lanes.” — InsidiousWombat
- “When High Bandwidth Flash comes out, most likely: …” — ajb (link)
- “Seems like a useful innovation on the desktop level, but on the server level, isn't it already pretty normal to see machines with 24x high‑performance NVMe drives … that can then be used in software RAID0 to achieve pretty massive read bandwidth?” — derefr

3. Repeated reinvention of existing tools (e.g., llama.cpp) and fixation on model versions draws criticism
- “Every few weeks, someone stumbles on what llama.cpp has had for years again, and reimplements it. I'm not sure why.” — jdb9001
- “Because with AI everybody can reinvent the wheel with ease.” — croes
- “Why are the models being used so old (in llm years anyway). Why no Qwen 3.8?” — kadoban
- “Qwen 3.6 & 3.8 have the exact same model architecture. In theory, it should just be a drop‑in replacement.” — x‑complexity (link)

🚀 Project Ideas

FlashCache: NVMe‑Backed KV Cache Accelerator for Local LLMs

Summary

  • Speeds up context prefill and token preprocessing for agentic workloads by storing intermediate key/value tensors on high‑speed NVMe storage.
  • Core value proposition: reduces effective latency from thousands of tokens/second to near‑generation speed, making local LLMs viable for complex agents.

Details

Key Value
Target Audience Developers running local LLMs for agents, AI researchers, hobbyists with GPUs but limited VRAM
Core Feature Memory‑mapped NVMe cache for transformer KV tensors with automatic reload/reuse across inference steps
Tech Stack Python, PyTorch/Triton, CUDA, io_uring/SPDK for NVMe, optional Rust bindings
Difficulty Medium‑High
Monetization Revenue-ready: subscription per GPU‑hour or enterprise license

Notes

  • HN commenters complained that input token processing is the bottleneck for agentic work (kolinko: “Most of the tokens consumed by agents is reading stuff…”) – FlashCache directly addresses this.
  • Enables faster iteration on agent pipelines and could be paired with existing llama.cpp or vLLM backends for immediate practical utility.

ModelBench Hub: Community‑Driven LLM Benchmark Dashboard

Summary

  • Centralized web platform where users submit benchmark results (latency, throughput, memory) for LLMs like Qwen 3.6/3.8, Llama, etc., and view side‑by‑side comparisons.
  • Core value proposition: gives developers reliable, reproducible data to pick the right model for their hardware and use case without guessing.

Details

Key Value
Target Audience ML engineers, DevOps, AI product managers choosing LLMs for production or local experiments
Core Feature Automated benchmark suite (Docker‑based workers) that uploads results; interactive charts and model‑version diff view
Tech Stack React/TypeScript frontend, Node.js/Express backend, PostgreSQL, Docker, Prometheus/Grafana for worker metrics
Difficulty Medium
Monetization Revenue-ready: freemium – public benchmarks free, private team workspaces $15/mo per seat

Notes

  • The thread highlighted confusion over model versions (e.g., Qwen 3.6 vs 3.8) and a desire for a guide to compare them (KerrAvon: “What's needed is a guide to all of them…”). ModelBench Hub fulfills that need.
  • Provides a concrete discussion point for HN: users can post their own benchmark PRs, sparking conversation about hardware‑software tradeoffs.

RepoModelTag: GitHub Action to Auto‑Tag LLM Usage & Token Cost

Summary

  • Scans repository code for LLM imports/API calls, estimates token usage per commit, and adds a visible badge (model used, estimated cost) to the README.
  • Core value proposition: lets maintainers and consumers instantly see the AI footprint of a project, encouraging transparency and informed dependency choices.

Details

Key Value
Target Audience Open‑source maintainers, developers integrating LLMs into libraries or tools
Core Feature Static analysis (Python/Rust) to detect model strings, compute cost using pricing tables, update repo metadata and generate markdown badge
Tech Stack Python (or Rust) core, GitHub Actions wrapper, GitHub API, optional caching with Redis
Difficulty Low‑Medium
Monetization Hobby (open‑source) – can later offer paid private‑repo scanning as a side product

Notes

  • Zelcon’s comment about wanting a tag listing the model used to write a repo (“GitHub repos should have a new tag listing the model used to write it along with the token cost”) is directly satisfied.
  • Encourages practical utility: developers can quickly assess whether a dependency’s LLM usage aligns with their budget or latency requirements, fostering informed discussion on HN.

Read Later