IdeaWell

Project ideas from Hacker News discussions.

Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

1. Local hardware feasibility & performance
Users repeatedly discuss whether the model can run on their own GPUs, Macs, or Strix Halo hardware and what token‑per‑second speeds they expect.

  • “As a 5090 owner and local model enthusiast, I was hoping it would be 35B A3B so I could run it myself =()” – hasteg
  • “You can run the 27B released last week… the 3.6 version runs great on my 5090.” – cpburns2009
  • “Strongly recommend [ninfer]… which can pull ~180 TPS on 5090 with 3.8, and 500 (!) with 3.6 35B‑A3B.” – Philpax
  • “50‑70 tk/s is what I get on my m5 max on a 5‑6bit Qwen 3.8 27B?” – smcleod
  • “With MTP? I get 25‑30 TPS on a strix halo. 50+ on a M5 max should very doable.” – syntaxing

2. Model quality & comparison to competitors
Many commenters compare the new Qwen release to Anthropic’s Claude/Opus, Sonnet, DeepSeek, etc., praising its potential to match or exceed those models.

  • “Alibaba is giving sleepless nights to the tech giants.” – honestranked
  • “Qwen3.8 27B is supposed to rival Sonnet/Opus 4.6.” – cpburns2009
  • “I’d expect this to mostly replace Claude for my workflows.” – hedora
  • “Qwen3.8/Qwen3.6 … ‘approximates’ Opus 4.X class models well enough especially for coding/linux problems.” – hugmynutus
  • “Very curious to see how this compares to Deepseek v4 Flash. I have to assume they wouldn't be releasing this if it was worse.” – Catloafdev

3. Provider reliability & self‑hosted routing
Several users express frustration with OpenRouter’s capacity/flakiness and advocate for self‑hosted litellm or custom routers to route between local and cloud models.

  • “OpenRouter … many Qwen models have almost no capacity or is so flaky you literally have to just litter your code with a blacklist/whitelist of providers.” – ddtaylor
  • “Openrouter was pretty great before prompt caching became common. Now it is extremely expensive for most individual workflows …” – irthomasthomas
  • “The best solution … is to self host litellm or a different router and use model aliases.” – geek_at
  • “I built a router that lets you route between local and cloud models. Link in my profile.” – try-working

🚀 Project Ideas

Generating project ideas…

MoE-Offload Inference Engine

Summary

  • Enables running large mixture-of-experts (MoE) models like Qwen3.8 125B-A6B on consumer GPUs by offloading inactive experts to system RAM or CPU, reducing VRAM pressure.
  • Core value proposition: near‑full model quality with usable token speeds on hardware as modest as an RTX 3090 or 64 GB Mac, without manual model sharding.

Details

Key Value
Target Audience Local AI enthusiasts, researchers, and small teams wanting to run MoE models on limited VRAM
Core Feature Automatic GPU/CPU expert partitioning with quantized expert loading and unified HuggingFace/llama.cpp API
Tech Stack Rust (core), Python bindings, CUDA, Metal, ggml for quantized experts, optional Triton kernels
Difficulty High
Monetization Revenue-ready: SaaS hosting of pre‑offloaded model containers ($9/mo per model) or Hobby
#### Notes
- HN users complained about needing a 3090/128 GB RAM to run Qwen MoE locally; this engine would let them do it on far less VRAM (quote: “If all it takes for a competitive model to run locally at good speeds is a used 3090 and some DDR4, then we might be in for the year of local AI.”)
- Provides a concrete tool for the FreeToken‑style work mentioned, lowering the barrier for local MoE experimentation and discussion.

Local-First Model Alias Router

Summary

  • A lightweight, self‑hosted API gateway that lets you define model aliases (e.g., “coding”, “agent”) and transparently routes requests to local inference backends (llama.cpp, vLLM, Ollama) or falls back to cloud providers, with health‑checking and caching.
  • Core value proposition: eliminates the need to maintain brittle provider blacklists/whitelists and gives OpenRouter‑like flexibility without flaky third‑party routing.

Details

Key Value
Target Audience Developers building agentic workflows or coding assistants who rely on multiple models and want reliable local/cloud failover
Core Feature Alias‑based routing, dynamic backend selection, latency‑based failover, OpenAI‑compatible endpoint, simple config UI
Tech Stack Go (or Rust) for HTTP server, Prometheus metrics, Redis for caching, Docker‑compose deployment
Difficulty Medium
Monetization Hobby (open‑source) – optional paid support/hosting ($5/mo)
#### Notes
- Commenters expressed frustration with OpenRouter’s flaky providers and the need to self‑host litellm or custom routers (quote: “The best solution to this for me is to self host litellm or a different router and use model aliases…”) – this project gives a batteries‑included alternative.
- Enables practical utility: users can switch between a local Qwen model and a cloud model without changing code, fostering experimentation and reducing reliance on unstable third‑party services.

Speculative Decoding Booster for Consumer GPUs

Summary

  • Implements speculative decoding (draft‑model + target‑model) using a tiny, fast draft model (e.g., TinyLlama or a quantized Qwen‑1.5B) to generate candidate tokens that are verified by the large Qwen MoE target, boosting token generation speed on RTX 5090, Strix Halo, or Mac M5.
  • Core value proposition: 2‑3× higher tokens‑per‑second for large models without sacrificing quality, making local use of Qwen3.8 125B‑A6B feasible for interactive workflows.

Details

Key Value
Target Audience Local LLM power users, gamers/researchers with high‑end GPUs or Apple Silicon seeking faster interactive generation
Core Feature Draft‑target pipeline with automatic model loading, KV‑cache reuse, support for llama.cpp, vLLM, and TensorRT‑LLM backends
Tech Stack Python (or C++), CUDA kernels, Metal Performance Shaders, GGML quantizers, optional Triton for draft model
Difficulty Medium
Monetization Revenue-ready: Premium plugin for popular front‑ends ($10 one‑time) or Hobby
#### Notes
- Many HN participants noted low token/s on 5090 and Macs, wishing for speculative decoding or MTP to close the gap with Claude (quote: “I was hoping it would be 35B A3B so I could run it myself… 50‑70 tok/s might be something…”) – this directly addresses that pain.
- Provides a tangible performance uplift that can be benchmarked and discussed, encouraging community adoption and further optimizations.

Read Later