IdeaWell

Project ideas from Hacker News discussions.

Token-maxxing is dead, OpenRouter's token usage plateaus

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1 – Doubts about OpenRouter’s data representativeness

“It only tracks companies that have decided to use OpenRouter to help them switch models – which biases towards companies that like open weights. Anyone using OpenAI or Anthropic directly is much less likely to be represented.” – simonw

Theme 2 – Perceived plateau or slowdown in usage/revenue growth

“Yeah, after scrutinizing the graph more closely, I'm actually surprised that people are even taking this headline seriously. There's been far larger ‘plateaus’ in the past.” – throwaway23597

Theme 3 – Shift toward alternatives, direct APIs, and cost‑efficiency

“I've had a hell of a time with more complex model setups in open router and have gone to just using the underlying APIs directly.” – noosphr

🚀 Project Ideas

Unified LLM Usage Dashboard & Cost Optimizer

Summary

  • Aggregates token usage and spend from OpenRouter, direct provider APIs, and self‑hosted models into a single real‑time view, highlighting cost drivers like prompt overhead and model‑specific pricing.
  • Provides actionable recommendations to switch to cheaper, equally capable models or trim wasteful tokens, reducing LLM bills by 20‑40% for typical workloads.

Details

Key Value
Target Audience AI engineers, product teams, and startups using multiple LLM services
Core Feature Unified telemetry, cost attribution, and automated model‑switch suggestions
Tech Stack TypeScript/React frontend, Python FastAPI backend, PostgreSQL + TimescaleDB, Prometheus‑style metrics, OAuth integration with provider APIs
Difficulty Medium
Monetization Revenue-ready: Subscription tiers based on monthly monitored tokens (Free <1M tokens, Pro $19/mo up to 10M, Enterprise custom)

Notes

  • HN users lamented OpenRouter’s biased stats and difficulty tracking real spend (“total token spent is lot lower than before… we are at %20 of our total peak spent”). This tool gives the transparent, cross‑provider view they asked for.
  • Enables data‑driven discussions about model ROI and can surface hidden waste like Hermes’ massive system prompts, directly addressing concerns raised by commenters.

Prompt & System Prompt Optimizer

Summary

  • Analyzes LLM prompts (including verbose system prompts such as those from Hermes) and returns a semantically equivalent, token‑minified version using techniques like redundancy removal, synonym substitution, and dynamic truncation.
  • Cuts prompt length by up to 50% without degrading output quality, lowering both latency and cost per request.

Details

Key Value
Target Audience Developers building agentic workflows, RAG pipelines, or chatbots who rely on lengthy system prompts
Core Feature Prompt compression API/SDK that outputs optimized prompt and estimates token savings
Tech Stack Rust core for fast string transformations, Python bindings, optional HTTP microservice, Docker deployable, unit‑tested with pytest
Difficulty Low
Monetization Revenue-ready: Pay‑per‑optimization at $0.0001 per 1k tokens processed (free tier 100k tokens/mo)

Notes

  • Commenters pointed out Hermes’ “massive overhead … 20‑30k tokens” per request; this optimizer directly shrinks that waste, appealing to those frustrated by token‑maxxing costs.
  • Provides a concrete, measurable improvement that can be benchmarked and shared in HN threads, sparking discussion on prompt engineering best practices.

Self‑Hosted Smart LLM Gateway with Adaptive Routing

Summary

  • Lightweight proxy that sits between application code and multiple LLM backends (OpenRouter, direct APIs, local models) and routes each request to the cheapest model likely to satisfy the task, falling back to higher‑quality models only when needed.
  • Includes token caching, request‑level cost analytics, and automatic failover, cutting unnecessary spending while preserving output quality.

Details

Key Value
Target Audience Startups and indie hackers looking to lower LLM expenses without sacrificing reliability
Core Feature Dynamic model selection based on task classification, cost‑quality scoring, and fallback logic
Tech Stack Go (Gin) gateway, Redis for caching, lightweight ML classifier (fastText or ONNX) for task tagging, Prometheus metrics, Configurable via YAML
Difficulty Medium
Monetization Revenue-ready: Open‑source core with optional hosted SaaS tier ($9/mo per gateway instance) offering UI, alerts, and SLA support

Notes

  • Users noted shifting usage to alternatives like OpenCode and frustration with “extra costs that no one wants to see”; a self‑hosted gateway gives them control and visibility.
  • Enables experimentation with newer, cheaper open models while keeping a safety net—exactly the flexibility discussants were craving when they considered moving off OpenRouter.

Read Later