IdeaWell

Project ideas from Hacker News discussions.

Fable 5 – Median thinking declined in August

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Prevalent Themes in Anthropic Model Performance Discussion

  1. Observed Performance Degradation Over Time
    Users consistently report that models like Fable/Opus start strong but noticeably worsen after weeks of use, even at fixed reasoning levels.

    "the drop in performance from, say, week 1 to week 8 is often massive. Whereas in the beginning, it seemed like a capable research assistant, by the end of week 8 or so it starts acting like a puppy dog eager to make its 'master' happy for a few treats." – mlmonkey
    "As of last 2 weeks or so both models are nearly on par with DeepSeek4.1 now... They're still better, but that difference is not as pronounced as before" – cromka

  2. Suspicions of Intentional Nerfing
    Many suspect companies deliberately reduce model performance post-launch (via quantization, inference tweaks, or resource allocation) to cut costs or drive upgrades, despite public denials.

    "What is actually stopping these model companies from running a model at full capacity on release then once its name rings out, start serving users quantized garbage?" – chrsw
    "Yep, that's what they've been doing for a long while now. Also the amount of tokens you get per sub varies drastically from month to month. Needs to be regulated." – rfgplk
    "Could there be a benefit to releasing a new model, slowly dumbing it down over a couple months, then releasing a new model that’s marginally if at all better than the original..." – talon8635

  3. Skepticism About Measurement & Perception
    Critics argue perceived degradation may stem from cognitive biases, non-determinism, or changing workloads—not actual model changes—making user reports unreliable without rigorous benchmarking.

    "people are still not used to non deterministic tools like this, and human perception is absolutely horrible at evaluating trends like this no matter how smart, clever, and experienced you are." – JMKH42
    "If you thought this was a repeated test of the same problems showing fluctuating performance, it's not... Your daily coding work should not be producing a proportional number of thinking tokens on every invocation..." – Aurornis
    "The implication is that humans are unreliable and shouldn't be trusted... humans have certain shorthands when they complain on reddit, but their diagnoses are accurate for the specific context?" – sigbottle

🚀 Project Ideas

ModelWatch

Summary

  • Continuously probes your subscribed LLM API with a fixed set of deterministic prompts to detect silent performance degradation or quantization.
  • Provides real-time alerts and historical trends so you can verify you’re getting the model you paid for.

Details

Key Value
Target Audience Developers and teams paying for Claude, GPT, or other hosted LLM subscriptions
Core Feature Automated daily benchmark suite (latency, token usage, correctness on held‑out tasks) with deviation detection and alerting
Tech Stack Python backend, FastAPI, Prometheus/Grafana for metrics, optional SQLite; uses openai/anthropic SDKs
Difficulty Medium
Monetization Revenue-ready: SaaS tiered pricing ($10/mo per monitored model)

Notes

  • HN users complain about “massive peaks and troughs in performance week to week” and want proof; ModelWatch gives objective data. (see comments like “I’ve anecdotally noticed massive peaks and troughs”)
  • Enables discussion: users can share anonymized degradation curves, fueling transparency pressure on providers.

HarnessDiff

Summary

  • Runs the same prompt through both the vendor’s hosted API and a locally‑run open‑weight model (via Ollama/HuggingFace) and compares token‑level outputs to surface hidden changes in the service.
  • Highlights when the hosted model diverges from its open counterpart, indicating possible throttling or quantization.

Details

Key Value
Target Audience Power users who self‑host models and also subscribe to hosted plans (e.g., Claude Max, GPT‑Pro)
Core Feature Side‑by‑side execution and diff of outputs (token counts, reasoning traces, final answer) with visual diff UI
Tech Stack React frontend, Node.js orchestrator, Docker containers for local models, API clients for vendors
Difficulty Medium
Monetization Hobby (open‑source) – could later add paid team collaboration features

Notes

  • Commenters suspect “they are serving quantized garbage” and want to see the difference; HarnessDiff makes that observable. (see “what is actually stopping these model companies from running a model at full capacity…”)
  • Provides concrete evidence for debates about model degradation, useful for HN threads and potential regulator scrutiny.

ModelFingerprint

Summary

  • Sends a suite of probing prompts designed to elicit model‑specific behavior (e.g., unique token distributions, special‑token responses) and derives a probabilistic fingerprint that can be compared against known fingerprints for advertised model versions.
  • Alerts when the fingerprint drifts, suggesting a silent swap or quantization.

Details

Key Value
Target Audience Enterprises and developers needing SLA‑grade assurance that the model version hasn’t changed
Core Feature Fingerprint generation API + dashboard showing version match score over time
Tech Stack Python (scikit-learn for clustering), FastAPI, Redis cache for storing reference fingerprints, optional WASM for lightweight probing
Difficulty High (requires research on probing robustness)
Monetization Revenue-ready: Enterprise licensing ($500/mo) + pay‑per‑probe API

Notes

  • HN discussion highlights lack of transparency: “they sell a black box with magic inside… no idea if it even is the model you’re paying for.” ModelFingerprint gives a way to peek inside.
  • Could spark discussion on what constitutes a reliable fingerprint and push vendors to expose version hashes.

ReasoningTransparency

Summary

  • Visual dashboard that tracks and plots thinking‑token usage, effort levels (xhigh, high, medium), and latency per request over time, making it easy to spot systematic throttling of reasoning capacity.
  • Allows users to correlate drops in thinking tokens with degraded performance on coding tasks.

Details

Key Value
Target Audience Heavy users of Claude Code, Codex, or similar agentic IDEs who rely on reasoning tokens
Core Feature Real‑time ingestion of token usage metadata (via API or local logs) and charting of thinking‑token trends with anomaly detection
Tech Stack TypeScript + React, WebSocket for live data, Node.js backend, TimescaleDB for time‑series
Difficulty Medium
Monetization Revenue-ready: Freemium (free basic charts, $8/mo for advanced alerts & export)

Notes

  • Users report “thinking tokens… dropped over weeks” and suspect “they are limiting the amount of gpu time normal users get to spend on the ‘thinking’ portion.” This tool makes that visible. (see comment: “I think they give a kind of intelligence boost also for new accounts…”)
  • Enables community sharing of token‑usage graphs, fueling debate about whether providers adjust reasoning budgets.

BenchmarkLedger

Summary

  • Decentralized, tamper‑evident log where users can submit benchmark results (prompts, scores, timestamps, model version) for any hosted LLM; submissions are signed and optionally verified via zero‑knowledge proofs to prevent gaming.
  • Provides a public, auditable history of model performance that anyone can query.

Details

Key Value
Target Audience Researchers, power users, and watchdog groups who want an open record of LLM quality over time
Core Feature Submit‑and‑verify benchmark entries; UI to view trend lines per model; integration with CI pipelines
Tech Stack Go backend, PostgreSQL + pgcrypto for signatures, optional zk-SNARK library (circom) for proof, React frontend
Difficulty High
Monetization Hobby (open‑source) – could accept donations or sponsorships; optional paid private‑team tier

Notes

  • HN users call for “a site that regularly benchmarks cloud hosted models with private benchmarks” and express frustration about inability to verify claims (see saejox comment). BenchmarkLedger satisfies that need.
  • Encourages discussion: transparent data can be cited in debates, and the ledger itself becomes a reference for potential regulation or lawsuits.

Read Later