IdeaWell

Project ideas from Hacker News discussions.

Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1 – Performance bottlenecks & optimization insights
Discussion focuses on the model’s token‑per‑second rates, the impact of drive count on bandwidth, and the costly prefill stage that repeatedly reads experts from disk.

“The useful bits: one drive gives ≈52% of four, two ≈73%, three ≈90%; and prefill is slow because of ~9 TB of reads for a 1.4 TB model — a scheduling bug with a planned fix.” – Argonautlabs
“prefill currently reads each layer's experts once per 64‑row pass (~9 TB of reads for a 1.4 TB model).” – Argonautlabs

Theme 2 – Practical usefulness (unattended batch vs. interactive use)
Many commenters debate whether ~1 token/s is useful, concluding it suits scheduled, unattended workloads (e.g., overnight report generation) but not real‑time chat.

“Not useful for chat, agreed — and I wouldn't pretend otherwise. It's useful for the other kind of work: scheduled, unattended jobs where nobody is waiting on the cursor.” – Argonautlabs
“It's not useful for actual work, but the fact it can be run at all shows that we're evolving towards enabling powerful LLMs to run locally.” – glimshe

Theme 3 – Appreciation for the engineering feat & exploratory spirit
A strong thread values the achievement as a proof‑of‑concept, celebrating the “look at this crazy thing I did” mindset that drives innovation.

“The entire history of innovation is filled with people doing something just to see they can get it to work… then people continue to iterate on that until it works better.” – roadside_picnic
“We do these things because they are interesting to us, for the fun of exploration… to make the world better, or any plethora of reasons that involve intellectual curiosity.” – sixothree
“I hate seeing comments like this on HN. We used to upvote ‘look at this crazy thing I did’ work.” – LatencyKills

🚀 Project Ideas

Generating project ideas…

Expert-Prefill Scheduler for MoE (EPFS)

Summary

  • Reduces prefill I/O amplification in Mixture‑of‑Experts models by scheduling each expert read once per layer, turning a 6.2× read amplification into near‑1×.
  • Core value proposition: cuts time‑to‑first‑token from minutes to seconds on SSD‑streamed LLMs, making local MoE inference usable for interactive workloads.

Details

Key Value
Target Audience ML engineers and researchers running large MoE models on commodity hardware with SSD/NVMe expert storage
Core Feature Expert‑major prefill scheduler that issues one read per expert per layer and dispatches kernels over all routed tokens
Tech Stack Rust (low‑level async I/O), Python bindings, optional CUDA kernels for expert execution
Difficulty High
Monetization Revenue-ready: Enterprise licensing / support contracts

Notes

  • HN commenters asked “whether anyone has done expert‑major prefill scheduling on an MoE” (Argonautlabs) and noted the prefill bottleneck as the biggest open problem.
  • Provides a concrete solution to the scheduling fix discussed, enabling token‑per‑second speeds suitable for classifier‑style workloads and interactive use.

SSD Expert Streaming Monitor (SESMonitor)

Summary

  • Generalizes the ARGODRIVE instrumentation suite: per‑device read monitoring, barrier trace, and config assertion harness for any SSD‑backed expert streaming system.
  • Core value proposition: gives engineers real‑time visibility into read latency, load balancing, and experiment validity, turning guesswork into data‑driven optimization.

Details

Key Value
Target Audience Researchers, performance engineers, and hobbyists benchmarking MoE or other expert‑streamed models on NVMe/Thunderbolt
Core Feature Live metrics dashboard (read throughput per device, in‑flight counts, barrier completion times) plus automated validation that a benchmark actually exercised the variable under test
Tech Stack Rust for low‑level tracing (using io_uring/Linux perf or macOS equivalents), React/Vue for dashboard, optional Python API
Difficulty Medium
Monetization Hobby

Notes

  • Commenters praised the “instruments are in a second repo, ARGODRIVE” and expressed interest in reproducing the drive‑ladder on other hardware (Argonautlabs).
  • A reusable monitoring tool would let the community verify optimizations across diverse storage configurations, sparking discussion and further improvements.

Local LLM Job Runner (LLMJob)

Summary

  • A self‑hosted service for scheduling and executing unattended LLM tasks (e.g., report generation, data extraction) on private hardware, with result delivery via email, Telegram, or webhook.
  • Core value proposition: enables reliable, overnight LLM work without exposing data, turning slow token‑per‑second speeds into useful batch productivity.

Details

Key Value
Target Audience Analysts, small businesses, and power users who need private LLM automation for recurring jobs
Core Feature Job queue, prompt templating, integration with any Llama.cpp / MoE inference backend, configurable result export and notification channels
Tech Stack Python (FastAPI), Redis for queue, Docker for deployment, optional WebSocket UI
Difficulty Medium
Monetization Revenue-ready: Subscription tier (hosted version) or paid support for self‑hosted enterprises

Notes

  • Users highlighted the usefulness of the setup for “scheduled, unattended jobs where nobody is waiting on the cursor” (Argonautlabs) and for generating reports delivered to Telegram.
  • Provides a polished, reusable wrapper around the demonstrated workflow, likely to earn enthusiasm from HN’s productivity‑focused audience.

Read Later