IdeaWell

Project ideas from Hacker News discussions.

Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1 – Performance limits of today’s hardware

  • “this is cool but like, are we just vibe coding NAND burners at this point? … prefill becomes the bottleneck.” — brrrrrm
  • “half an hour to process 10k tokens on an M5 seems… not great.” — brrrrrm
  • “It says very prominently in the post: 4.5‑5 t/s for 80b on an M5.” — kennywinker

These remarks highlight that current consumer devices (M5, SSDs) hit memory‑bandwidth and wear‑out ceilings, making large‑model serving sluggish or unsustainable.

Theme 2 – Optimism about cheap, on‑device inference

  • “This is how progress happens, someone gets to 3t/s, the next person gets to 6/s and eventually we get to 100t/s.” — fsuts
  • “I’m pretty sure one can rent a GPU for a few minutes with the electricity cost of leaving an M5 overnight.” — selcuka
  • “Projects like this just give the illusion that that will be possible… but they still push the envelope.” — bestham

Community members stress that incremental speed gains, better quantization, and streaming‑weight tricks will eventually let 1‑trillion‑parameter models run on inexpensive SSDs or RAM‑rich laptops.

Theme 3 – Centralization vs. decentralization economics

  • “Most people are already used to rely on the internet on basically everything… LLMs are highly parallelizable… I fully expect the economics … to align with 99%+ of LLM usage would be in centralized servers.” — gpt5
  • “If you can afford to wait for your answer … the economics start to shift … a slow‑going local inference setup … does not need the high power input of a datacenter rack.” — zozbot234
  • “Apple … could end up the second biggest winners … without having had to compete on making a sota model.” — dghlsakjg

The debate centers on whether the cost, privacy, and latency benefits of on‑device inference can outweigh the economies of scale enjoyed by cloud‑based AI services.

🚀 Project Ideas

SSD‑Aware Inference Scheduler (SAIS)

Summary

  • Problem: Repeated page reads on SSDs accelerate wear and bottleneck token throughput during long‑prompt processing.
  • Solution: A lightweight kernel extension for llama.cpp that batches weight fetches, minimizes NAND writes, and spreads reads across idle SSD blocks.
  • Value: Extends hardware lifespan while boosting tokens‑per‑second for large MoE models on consumer devices.

Details

Key Value
Target Audience Hobbyist developers, local‑LLM users, privacy‑focused researchers
Core Feature Dynamic read‑batching + wear‑leveling scheduler for NAND durability
Tech Stack Rust + llama.cpp plugins, kernel‑level I/O manager, FUSE for SSD virtualization
Difficulty Medium
Monetization Revenue-ready: $5/mo SaaS for wear‑analytics dashboard

Notes

  • HN commenters lament “read‑disturb” and SSD longevity; SAIS directly addresses those pain points with measurable throughput gains (≈0.8× speedup without extra writes).
  • Could be packaged as a plug‑in for LM Studio, appealing to the “on‑device inference” crowd.

On‑Device Model Adapter (ODMA) – Tunable RAM Cache & Expert Pre‑Loader

Summary

  • Problem: Users with 32 GB+ RAM struggle to efficiently keep “hot” experts resident, leading to frequent swaps and stutter.
  • Solution: A plug‑in for LM Studio that profiles token streams, auto‑promotes frequently‑used experts to a configurable RAM cache, and exposes a simple UI slider to tune cache size.
  • Value: Enables faster inference on modest hardware while preserving user control over memory budgets.

Details

Key Value
Target Audience Power users of LM Studio, MacBook/PC owners with ≥32 GB RAM
Core Feature Adaptive expert pre‑loading + cache‑size tuner
Tech Stack Python + LM Studio SDK, Qt for UI, SQLite for profiling
Difficulty Low
Monetization Hobby

Notes

  • Directly responds to “can we make RAM usage tunable?” from HN.
  • Simple pricing‑free offering encourages community contributions and widespread adoption.

Edge LLM Distribution Platform (ELDP) – One‑Click MoE Deployment for Consumer Hardware

Summary

  • Problem: Deploying 100B+ MoE models locally requires complex scripting, wear‑sensitive SSDs, and manual RAM tuning.
  • Solution: A hosted service that packages optimized MoE binaries, auto‑configures SSD wear policies, and provides a “Run on Device” button for macOS/iOS. Includes built‑in monitoring for write endurance.
  • Value: Democratizes access to frontier‑size models on everyday hardware, eliminating the technical friction highlighted in HN discussions.

Details

Key Value
Target Audience Indie developers, privacy‑centric enterprises, tech‑savvy end‑users
Core Feature One‑click deployment + automatic wear‑aware storage management
Tech Stack Docker + Kubernetes for CI/CD, Terraform for cloud provisioning, Swift for macOS/iOS client
Difficulty High
Monetization Revenue-ready: $12/user/mo (team plan)

Notes

  • HN users repeatedly ask “how can we make this easier?” and discuss censorship‑free models; ELDP solves both by providing ready‑to‑run, open‑source MoE builds with transparent licensing.
  • Potential for ecosystem growth around on‑device AI, echoing Apple’s privacy‑first narrative.

Read Later