IdeaWell

Project ideas from Hacker News discussions.

Benchmarking Opus 5 on SlopCodeBench

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

1. Opus 5’s Slop & Modest Improvement

"reviews are overly pedantic, and that leads to feedback loops where each fix generates more feedback" — patwolf
"opus 5 is not much of an improvement. The only time i ever felt a wow factor was opus 4, 4.6 and fable pre trump admin lobotimizing" — Johnny_Bonk
"I almost prefer Opus 4.8. Opus 5.0 has the overly scholastic tone of Fable but without the intelligence." — roncesvalles

2. Need for Robust Long‑Term Code‑Quality Benchmarks

"maintainable is probably some high‑dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is" — dhorthy
"I'm not ready to blame Opus 5 for being stupid. Perhaps we have a prompt buried somewhere that's essentially asking it to be pedantic, and it's just obeying the prompt." — latentsea
"Really hoping that all of the attention you're bringing to the longitudinal sloppification of codebases makes it back to the labs..." — akurilin

3. Practical Harness & Cost Strategies

"the next thing on my radar is to try with a more realistic “factory‑shaped” harness where you have feedback from linters and other models after each coding episode" — dhorthy
"At least for Claude Code, putting “run /simplify at the end” in an “implement the plan” skill helps a little." — igregoryca
"Medium or low supposedly prevents Opus 5 from overthinking" — ValentineC

🚀 Project Ideas

Generating project ideas…

SlopBench Insight Dashboard

Summary

  • A web-based dashboard that ingests SlopBench JSON results, visualizes failure modes, and surfaces maintainability signals across model versions.
  • Provides a shared, interactive space for teams to compare Opus 4.x, Opus 5, and Fable on standardized code‑quality metrics.

Details

Key Value
Target Audience Engineering managers, model evaluation engineers, and research teams
Core Feature Interactive visual reports of benchmark outcomes, drift detection, and model‑specific slop patterns
Tech Stack React + TypeScript, Node.js (Express), PostgreSQL, D3.js, Docker
Difficulty Medium
Monetization Revenue-ready: Subscription (team tier $29/mo, enterprise $199/mo)

Notes

  • HN users repeatedly mention “slop”, “over‑thinking”, and “feedback loops” – a visual tool directly addresses those pain points.
  • Could spark discussion by exposing hidden difficulty patterns in benchmark ordering and prompting better harness design.

Agentic Refactor Loop Orchestrator

Summary

  • A CLI/SDK that wraps language models in a controlled refactor‑turn cycle, automatically invoking linters, static analyzers, and simplification prompts between coding passes.
  • Guarantees that each iteration ends with a “refactor turn” to curb slop accumulation and maintain code clarity.

Details

Key Value
Target Audience Power users of Claude Code, Opus, and Fable who build long‑running agent pipelines
Core Feature Multi‑step workflow engine with mandatory refactor validation hooks and token‑budget enforcement
Tech Stack Python 3.11, FastAPI, Pydantic, GitPython, Kokoro TTS (optional), Docker Compose
Difficulty High
Monetization Revenue-ready: Usage‑based GPU billing (e.g., $0.02 per refactor turn)

Notes

  • Directly responds to comments like “models are overly pedantic” and “need a harness for refactors”; the tool enforces disciplined iteration.
  • Will be of high interest to the community looking for practical ways to mitigate slop in automated engineering pipelines.

StateSpace Code Visualizer

Summary

  • A SaaS that parses a codebase into bounded state‑machine modules and generates a visual graph of system state transitions, making it easier for LLMs to “see” the environment.
  • Bridges the gap between raw code and model‑usable state representations, improving benchmark reliability.

Details

Key Value
Target Audience Teams building production‑grade agentic systems and researchers measuring maintainability
Core Feature Automatic extraction of state diagrams (e.g., via abstract syntax trees and dependency graphs) and export to JSON/GraphViz for model consumption
Tech Stack Go, PostgreSQL, D3.js front‑end, GraphQL API, Kubernetes (for scaling)
Difficulty Medium
Monetization Revenue-ready: Tiered pricing (Starter $15/mo, Pro $75/mo, Enterprise custom)

Notes

  • Frequently cited in HN threads about “state space” and “making state visible” – this product turns that abstraction into a concrete tool.
  • Offers immediate discussion potential by enabling standardized measurement of maintainability signals across models.

Read Later