IdeaWell

Project ideas from Hacker News discussions.

Running Kimi K3 on a M1 Max

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

3 Core Themes from the Discussion

1. Speed is a critical limitation
Users repeatedly point out how slow even the best local runs are, making them feel more like curiosities than useful tools.

"0.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?" – Azantys

"16 tokens / s is not nothing." – winstonp

"Would it even finish over a few weeks?" – whatsThisBtn4

2. Email‑style, asynchronous interfaces are preferred
Many suggest that a chat‑centric view is the wrong mental model; treating the model as a “pen‑pal” that replies via email fits the realistic expectations of slow inference.

"I like seeing the latest and greatest model crammed into new systems to see how it fares. To deal with the speed, one person on reddit suggested using it in an email interface rather than a chat interface." – tito

"Kimi Pen Pal. Bring back lettets and postcards. Do OCR, and use one of those 3D printer‑like pen plotters write the model output as a letter." – magicalhippo

3. Local AI on consumer hardware is intriguing despite slowness
There’s strong interest in pushing frontier models onto laptops and Macs, even if it means dealing with massive models, steep storage demands, and painfully low token rates.

"It reminds me of when Willow Garage chose to name their bot the TurtleBot, because if they named it anything else, people would think it was fast and capable. But when they called it Turtle Bot, people just kind of liked it and were satisfied with what it did." – tito

"SSD streaming on an M5 Max 128GB … Soon decent speed across two Mac Studios with 512GB of RAM." – antirez

"Would it even finish over a few weeks? For AI agents this would take a year." – whatsThisBtn4

These three themes capture the community’s focus: speed constraints, preference for email‑style interaction, and the appeal of running large models locally despite current performance limits.

🚀 Project Ideas

Kimi Pen Pal

Summary

  • A hobby service that takes local LLM output and prints it as handwritten letters mailed to the user’s inbox, turning ultra‑slow inference into a tangible “pen‑pal” experience.
  • Turns minutes‑per‑token delays into a nostalgic, analog communication format that HN users find charming and nostalgic.

Details

Key Value
Target Audience LLM enthusiasts with local Kimi/K3 models, hobbyist coders, retro‑tech fans
Core Feature Generates model replies, sends them to a small plotter printer, folds them into envelopes, and mails them back with OCR‑generated return letter
Tech Stack Raspberry Pi + thermal printer driver, Python script for email parsing, CNC‑style plotter, USPS API for mailing, SQLite DB
Difficulty Medium
Monetization Revenue-ready: Subscription $4.99/mo for up to 10 letters/month

Notes

  • HN commenters explicitly praised the email/letter idea (e.g., “hugopuybareau: Love the email idea”, “magicalhippo: Kimi Pen Pal… Do OCR…”).
  • Provides a practical utility for users who otherwise wait days for a response, turning latency into a collectible artifact.
  • Potential for community challenges (e.g., “slowest reply wins”) to drive engagement.

AsyncLLM Batch Engine

Summary

  • A lightweight SaaS that accepts large inference requests, queues them, and runs them on idle local Mac Studios or M1‑Max rigs overnight, delivering results via email or file drop.
  • Solves the “waiting a whole day for 1k tokens” frustration by batching and leveraging spare compute cycles.

Details

Key Value
Target Audience Researchers, developers, hobbyists with powerful local hardware who run non‑interactive workloads
Core Feature Submit a prompt/file via API, engine schedules inference on pooled idle Macs, returns output when completed (typically overnight)
Tech Stack FastAPI backend, PostgreSQL for job queue, Docker containers on user‑provided machines (via SSH), secure file storage
Difficulty High
Monetization Revenue-ready: Pay‑as‑you‑go $0.02 per 1k tokens processed, plus a $9.99/mo premium for priority queue

Notes

  • Directly addresses HN concerns about “0.01 tk/s is unusable” by allowing users to offload compute to idle machines and receive results later.
  • Aligns with Tito’s “asynchronous LLM systems” comment and the desire for “slow, complete responses”.
  • Could be marketed as “LLM‑as‑Judge for overnight batch scoring” – a use case HN users highlighted.

Local LLM Streaming Suite

Summary

  • An open‑source CLI/desktop tool that streams model weights from high‑capacity NVMe RAID or Thunderbolt‑attached SSD arrays, reducing load‑time bottlenecks and enabling faster token‑per‑second rates on consumer hardware.
  • Makes running 100‑GB+ models on a single laptop feasible for hobbyists.

Details

Key Value
Target Audience Home‑lab tinkerers, M1‑Max/M2‑Ultra owners, developers experimenting with frontier local LLMs
Core Feature Dynamically streams weights from a configurable RAID‑0 of NVMe drives or external Thunderbolt SSD enclosures to the inference engine, caches KV‑cache in RAM, and reports real‑time throughput
Tech Stack Rust CLI, libfabric for RDMA over Thunderbolt, RocksDB for index, OpenCL/Kernel for weight loading, Docker for optional remote streaming service
Difficulty High
Monetization Hobby: Free, with optional paid “Premium Mirror Service” $5/mo for hosted high‑speed mirrors

Notes

  • Solves antirez’s “streaming from online host is extremely slow” and the broader SSD lifespan concerns raised in HN threads.
  • Echoes discussions about “PCIe bus transfers” and “NVMe RAID” as ways to achieve near‑RAM speeds.
  • Provides a concrete technical path for users wanting to push the envelope of local inference speed, resonating with the community’s engineering curiosity.

Read Later