IdeaWell

Project ideas from Hacker News discussions.

M5 Ultra Mac Studio Review

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1: Cost and Value Considerations
- andreandre: “5 years of (200/month) tokens at that price, meanwhile an rtx 5090 pc is about half that… hmm”
- simonw: “Yeah, anyone who thinks local AI is going to save them money is likely to be disappointed, at least if they want to run models that are even remotely capable.”

Theme 2: Performance and Speed Benchmarks
- simonw (speed table):
Qwen3.8 27B tokens/sec generation speed Prompt size 8K 64K 128K 256K RTX 5090 PC 59 51 44 n/a M5 Ultra 48 39 32 24
- beastman82: “The 5090 is running qwen 3.8 at 200+ tps!! That's 1-2 orders of magnitude faster.”

Theme 3: Hardware Architecture and Model Suitability
- tcdent: “A dense model (up to the amount of memory available) actually does make the most sense on unified memory architectures. But when you hit the limit of what you can hold in memory, you reach the limitation of the platform.”
- nacs: “People don't buy Sparks and M5 Ultras to run a 27B model - you buy it to run an MoE model like Qwen Next which this M5 excelled at.”

Theme 4: Ecosystem, Software, and Usability
- throwaway27448: “A) the macos value add is enormous if you have any investment in the ecosystem, B) for me at least a GPU is completely useless for anything but being a token generator.”
- kokonokko1337: “Yes Apple has some of the best hardware out there, albeit overpriced. But the software is such a hindrance and I can't take anyone that states otherwise seriously.”

🚀 Project Ideas

Local AI Cost & ROI Calculator

Summary

  • A web-based calculator that compares the total cost of ownership (hardware, electricity, depreciation) of running LLMs locally (Mac Studio, RTX PC, etc.) against cloud API token pricing for various usage patterns.
  • Core value proposition: Helps users make data-driven decisions about whether investing in local AI hardware is economical based on their specific model, token volume, and local power rates.

Details

Key Value
Target Audience AI hobbyists, developers, small teams considering local LLM deployment
Core Feature Input model size, expected daily tokens, hardware choice (Mac Studio M5 Ultra, RTX 5090, etc.), electricity cost; outputs break-even time, monthly cost, and sensitivity analysis
Tech Stack React frontend, Python/FastAPI backend, hosted on Vercel or similar; uses real‑time hardware pricing APIs (e.g., eBay, Newegg) and cloud pricing data
Difficulty Medium
Monetization Hobby (could later add premium plans for team sharing)

Notes

  • HN commenters debated whether a $15k Mac Studio is worth it versus “12 years of OpenAI Pro” (nowittyusername) and token costs (snarfy, andreandre). This tool would give concrete numbers to settle those arguments.
  • Enables discussion on HW vs cloud trade‑offs and could be shared in threads like this one to ground opinions in personal usage stats.

Unified Inference Offload Engine

Summary

  • A library/server that automatically splits LLM layers between GPU VRAM and system RAM/CPU, using unified memory techniques on Apple Silicon and paging‑style offload on discrete GPUs, enabling models larger than VRAM to run smoothly.
  • Core value proposition: Lets users run MoE models like Qwen 3.8 Next (125B/6B active) on a single RTX 5090 or Mac Studio without manual model swapping or complex CPU‑offload scripts.

Details

Key Value
Target Audience Local AI enthusiasts, researchers, small labs wanting to run large MoE models on commodity hardware
Core Feature Transparent offloading: profiles model layers, stores less‑used weights in RAM/Virtual memory, streams them on‑demand with async prefetch; supports NVFP4, MXFP4 quantizations and MTP
Tech Stack Written in Rust with bindings for Metal (Apple), CUDA, and Vulkan; uses llama.cpp or vLLM as inference core; configurable via TOML
Difficulty High
Monetization Hobby (open‑source; optional paid support/consulting)

Notes

  • Commenters noted the 5090’s 32 GB VRAM limits running Qwen 3.8 Next (peri-cl, nacs) and praised CPU offload tricks (peri-cl: “CPU offload to DDR5 6000 RAM”). This engine would automate that workflow.
  • Could spark discussion on optimal offload strategies and become a go‑to tool for running the newest open‑weight MoE models locally.

LLM Inference Benchmark Suite

Summary

  • An open‑source CLI/GUI suite that runs standardized LLM inference workloads (prefill, decode, batch sizes) across hardware platforms, measuring tokens/sec, latency, power draw (via IPMI/SMC), and memory bandwidth utilization.
  • Core value proposition: Provides reproducible, apples‑to‑apples performance numbers so users can compare Mac Studio M5 Ultra, RTX 5090, AMD GPUs, etc., under identical quantizations and optimizations (MLX, llama.cpp, TensorRT‑LLM).

Details

Key Value
Target Audience Hardware reviewers, AI developers, procurement teams evaluating local AI hardware
Core Feature Runs prompts of varying lengths (8K‑256K tokens) on models like Qwen 3.8, GLM 5.3‑flash, DeepSeek‑V4 flash; logs results to CSV and optionally uploads to a public leaderboard
Tech Stack Python core, uses MLX, llama.cpp, TensorRT‑LLM backends; power measurement via Intel RAPL, Apple’s powermetrics, or NVML; frontend in Tauri/Rust for GUI
Difficulty Medium
Monetization Hobby (donations via Open Collective)

Notes

  • The thread is full of benchmark numbers (simonw’s table, peri‑cl’s GLM 5.3‑flash numbers, gpugreg’s NVFP4 claims). A unified benchmark would let commenters verify claims like “200 tps with ninfer using NVFP4” (gpugreg) and settle disputes about real‑world throughput.
  • Enables practical utility: reviewers can point to a single source of truth when discussing cost‑per‑token or performance‑per‑watt.

Thunderbolt Distributed Inference Orchestrator

Summary

  • A service that simplifies scaling LLM inference across multiple Mac Studio (or Thunderbolt‑enabled PC) nodes over Thunderbolt 4/5, handling model sharding, KV‑cache synchronization, and load‑balancing via a simple API.
  • Core value proposition: Turns a cluster of Mac Studios into a coherent, high‑memory AI backend without needing Infiniband, complex networking, or manual DevOps—addressing the desire to run >1T‑parameter models locally.

Details

Key Value
Target Audience Power users, small research teams, AI enthusiasts who own multiple Mac Studios or Thunderbolt‑capable workstations
Core Feature Automatic model partitioning (tensor‑parallel or pipeline‑parallel), dynamic KV‑cache transfer over TB5, health‑checking, and a REST/gRPC endpoint that mirrors a single‑instance LLM API
Tech Stack Built with Swift for macOS services, uses Apple’s Thunderbolt networking framework; RPC via gRPC‑Swift; model execution via MLX or llama.cpp compiled for Apple Silicon
Difficulty High
Monetization

Read Later