IdeaWell

Project ideas from Hacker News discussions.

Measuring the sloppiness of code

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

1. AI‑generated code tends to be low‑quality “slop” that accumulates over time
- “bad coding decisions accumulate over time and for the strict solve rate, where all tests have to be passed at all checkpoints, even state of the art models achieve 0% pass rate” – tphyahoo2
- “agentic coding uses 2‑10× as many lines of code to accomplish the same task” – bunderbunder

2. Coding is not “solved” by AI; human oversight, design, and software engineering remain essential
- “Coding is not solved, correctness is not a feature, it is the bare minimum…. current models are pretty bad at this.” – danbruc
- “Best explanation I’ve heard is that ‘coding is solved but software engineering hasn’t’.” – elcritch
- “Companies that produce novel things are still hiring, often to manage agents and review output.” – guywithahat

3. Business and management pressures often favor speed over code quality, influencing the prevalence of slop
- “The pressures from management and the company environment are not always a bad thing… taking a long time to ship great code can mean that the company goes out of business.” – hax0ron3
- “Blaming workers is always an excuse by poor management.” – shimman

4. There is a strong need for better metrics and benchmarks to evaluate AI‑generated code quality
- “I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast.” – fosterfriends
- Describing a benchmark: “From all these different values, we can create a heuristic ‘architecture score’ to benchmark against.” – technoplato

🚀 Project Ideas

SlopBench Pro

Summary

  • An extensible benchmark suite that measures LLM-generated code quality across iterative rounds with context resets, capturing LOC, cyclomatic complexity, churn, token usage, and architectural cohesion to quantify “slop.”
  • Core value proposition: gives AI labs and agent developers a reproducible, automated metric to optimize models for maintainable, not just correct, code.

Details

Key Value
Target Audience LLM researchers, AI agent builders, evaluation teams
Core Feature Multi‑round harness that resets model context, runs a task, collects slop metrics, and aggregates a SlopScore over iterations
Tech Stack Python, Docker, pytest, optional LLM APIs (OpenAI, Anthropic), JSON schema for results
Difficulty Medium
Monetization Revenue-ready: tiered SaaS ($49/month for basic, $199/month for enterprise with private benchmarks)

Notes

  • HN commenters stressed the need for “solid benchmarks measuring the quality of generated code” (fosterfriends) and noted that “bad coding decisions accumulate over time” (tphyahoo2); this directly addresses those calls.
  • Provides a concrete platform for discussion on what constitutes slop and can be integrated into model training loops to close the reward‑function gap.

ArchScore

Summary

  • Static analysis tool that computes an architectural quality score for codebases produced by LLMs, focusing on coupling, cohesion, modularity, and cross‑platform duplication.
  • Core value proposition: makes hidden architectural debt visible so teams can steer agents toward clean, maintainable designs before technical debt accrues.

Details

Key Value
Target Audience Engineering leads, platform teams using AI coding assistants
Core Feature Dependency‑graph analysis, cyclic complexity, churn detection, and an aggregated “ArchScore” reported via CLI or GitHub Action
Tech Stack Rust (for speed), Tree-sitter parsers, SQLite for metrics, optional WASM for browser preview
Difficulty Medium-High
Monetization Hobby (open‑source core) with optional hosted dashboard Revenue-ready: $15/seat/month

Notes

  • Users like dherman asked for “ways to measure architectural properties, like separation of concerns, clear architectural layering, well‑defined interfaces”; ArchScore delivers exactly that.
  • Enables practical utility: teams can set ArchScore thresholds in CI to block PRs that degrade architecture, sparking discussion on sustainable AI‑generated codebases.

SlopLinter

Summary

  • Configurable linter that flags patterns indicative of LLM slop (excessive LOC, defensive code, unnecessary abstractions, duplicated logic, high token‑per‑function usage) and offers quick‑fix suggestions.
  • Core value proposition: integrates into editors and CI to instantly catch slop before it spreads, reducing cognitive debt and cleanup effort.

Details

Key Value
Target Audience Developers using Copilot, Cursor, Devin, or similar AI pair‑programmers
Core Feature Rule‑based detection (AST/Treesitter) with customizable thresholds, auto‑fixes, and inline explanations of why a construct is sloppy
Tech Stack Python (AST), optional Tree‑sitter for multi‑language, pre‑commit hook, VS Code extension template
Difficulty Low-Medium
Monetization Hobby (MIT‑licensed) – hosted team version Revenue-ready: $9/user/month for dashboard & trend analytics

Notes

  • Commenters noted that “LLMs generate too much code, code that is doing more than was asked” (einrealist) and “defensive code… because of DB columns I just forgot should be NOT NULL” (pdimitar); SlopLinter catches these exact patterns.
  • Low barrier to adoption encourages community contributions of new slop rules, fostering ongoing discussion on what constitutes slop in the LLM era.

AgentReview

Summary

  • AI‑powered code review agent that evaluates pull requests generated by LLMs (or humans) for slop, maintainability, and architectural soundness, leaving actionable review comments.
  • Core value proposition: augments human reviewers with a consistent, metric‑driven reviewer that reduces the overload of reviewing AI‑generated code while improving overall code health.

Details

Key Value
Target Audience Teams using autonomous coding agents (e.g., Devin, Astra, Fable) or AI‑assisted PRs
Core Feature LLM‑as‑judge using a rubric of LOC, cyclomatic complexity, churn, duplication, and token efficiency; posts structured review comments via GitHub API
Tech Stack Python, LangChain or LiteLLM for model orchestration, Prompt templating, GitHub Actions integration
Difficulty Medium
Monetization Revenue-ready: $0.02 per PR reviewed or $99/month for unlimited private‑repo reviews

Notes

  • HN discussion highlighted the need for “agentic code review” (justinmarsan) and “LLM as judge in harness evals” (anilgulecha); AgentReview fulfills both by providing an automated reviewer that can be tuned to project‑specific quality standards.
  • Generates tangible discussion points: teams can compare human vs. AI review efficacy, and iterate on the rubric to align with organizational definitions of “good code.”

Read Later