IdeaWell

Project ideas from Hacker News discussions.

Cognition launches new SWE-2 model, Rivaling Fable 5.1 and GPT-Astra

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

1. Benchmarking skepticism / “benchmaxing”
Many commenters doubt the advertised scores, pointing out the steep drop from older benchmarks to the newer Terminal‑Bench 4 as evidence of over‑fitting.
- “Postalcoder: … this is the definition of bench maxxing.”
- “enraged_camel: Yeah, this echoes my thoughts. I will be very surprised if a model with 2.8T parameters reaches the intelligence and capabilities of 10T parameter models. RL can take things far, but not that far.”
- “nullbio: Closed weights AND benchmaxxed. Somehow this company raised 2bil at a 48bil valuation. Pure insanity.”

2. Model origins, distillation, and openness concerns
A recurring thread is that SWE‑2 appears to be a post‑trained/distilled version of the Chinese Kimi K3 model, raising questions about novelty, weight availability, and whether it’s just a re‑package.
- “Tsarp: SWE-2 is post-trained from Kimi K3.”
- “airstrafer: I guess I don't. Does post‑training from another (larger) model not fall under the umbrella of distillation? I’d imagine it leads to the same spiky‑ness issues…?”
- “FergusArgyll: Distilling you don't have the actual model weights of the teacher… Fine tuning you have the actual model weights of the original model…”
- “xlbuttplug2: I presume post training is significantly easier than the distillation/training the top Chinese labs are doing…”

3. Platform lock‑in and usability concerns (Devin‑centric access)
Several users criticize the requirement to use Cognition’s own CLI/Devin ecosystem, saying it creates friction and limits easy evaluation alongside other models.
- “scronkfinkle: Please correct me if I'm wrong, but this appears to require Devin to use? I'm disappointed to see I need to use a bespoke platform to interact with this agent…”
- “wren6991: Your own CLI? Not even a /v1/chat/completions API? Is your business model based on pretending LLMs are not an interchangeable commodity already?”
- “randomblock1: I just gave it a try and it doesn't appear to be free, it used up some of my on demand usage…”
- “anthonypasq: they are an agent company not a model provider, is this that difficult to comprehend?”

🚀 Project Ideas

OpenRouter-Compatible Devin Model Gateway

Summary

  • Provides a standard OpenAI-compatible /v1/chat/completions endpoint for Devin’s proprietary models (SWE‑2, SWE‑1.7, etc.) so users can evaluate them with existing tools (OpenRouter, curl, SDKs) without installing Devin’s CLI or desktop app.
  • Core value: eliminates vendor lock‑in and friction for model comparison, letting developers run benchmarks or integrate Devin models into their own workflows using familiar APIs.

Details

Key Value
Target Audience AI researchers, developers, and evaluators who want to test Devin models alongside open‑weight models using standard LLM APIs.
Core Feature Reverse‑engineered API gateway that authenticates with a Devin account, forwards chat completions to Devin’s backend, and returns OpenAI‑format responses (including token usage, logprobs if available).
Tech Stack Node.js/Express (or FastAPI), Docker, Redis for rate‑limit caching, OAuth2 for Devin auth, OpenAPI spec generation.
Difficulty Medium
Monetization Revenue-ready: subscription tier ($5/mo for unlimited gateway calls) or pay‑per‑request ($0.0005 per 1k tokens).

Notes

  • HN commenters complained about needing to use Devin’s bespoke CLI (scronkfinkle, wren6991) and the inability to evaluate models via OpenRouter; this gateway directly addresses that friction.
  • Enables side‑by‑side benchmarking (e.g., Terminal Bench 4, HumanEval) without leaving familiar evaluation harnesses, encouraging community‑driven reproducibility.
  • Potential for discussion: how reliable is a reverse‑engineered gateway? Could be open‑sourced to let the community improve auth handling and rate‑limit transparency.

BenchMarkX – Reproducible Coding Benchmark Suite

Summary

  • A container‑based benchmark framework that generates synthetic, verifiable coding tasks (e.g., fix‑bug, implement‑feature, refactor) with deterministic scoring based on unit‑test pass rates and resource usage.
  • Core value: reduces “benchmaxxing” by providing tasks that are not publicly available in training data and can be regenerated with different seeds, making overfitting obvious and generalization measurable.

Details

Key Value
Target Audience Model developers, AI labs, and benchmark curators who need trustworthy evaluation of coding agents.
Core Feature CLI tool (bmx generate --seed 123 --count 100) that creates isolated Docker containers, runs agent solutions, and aggregates pass/fail, latency, and token‑cost metrics; includes a public task registry for community contributions.
Tech Stack Python, Docker SDK, Jinja2 templates for task generation, Pytest for verification, GitHub Actions for CI, optional Web UI built with React.
Difficulty Medium
Monetization Hobby (open source); optional hosted service for private task generation at $10/mo.

Notes

  • Commenters lamented that existing benchmarks (Terminal Bench 2 vs 4) are easily gamed and that synthetic, verifiable problems would be better (dpweb, gunalx). BenchMarkX offers exactly that.
  • By exposing seed‑based task generation, labs can show whether a model’s performance drops on unseen variants, directly addressing “benchmaxxing” accusations.
  • Encourages discussion: the project could become a community standard akin to BigCode’s evaluation harness, with leaderboards that penalize overfitting to specific seeds.

AgentHarness – Universal Adapter for Coding Agents

Summary

  • A thin adapter layer (CLI plugin + VS Code/JetBrains extension) that lets any coding agent (Devin, Cursor, Codex, custom LLM wrappers) be invoked through a uniform interface (agent run --prompt "…"), returning structured diffs or patch files.
  • Core value: eliminates workflow friction when switching between agents, letting users keep their preferred IDE/CLI while experimenting with new models like SWE‑2 without learning bespoke toolchains.

Details

Key Value
Target Audience Developers who already have established coding workflows (e.g., custom scripts, pre‑commit hooks) and want to test multiple agents without changing their tooling.
Core Feature Proxy that translates a common agent spec (input: prompt + context files; output: unified diff or JSON plan) into each agent’s native API (Devin CLI, Cursor API, Codex endpoint), handling auth, sandboxing, and result normalization.
Tech Stack Rust (for low‑overhead CLI) + TypeScript extension shims; uses tree-sitter for language‑aware diff generation; config via YAML.
Difficulty High
Monetization Revenue-ready: per‑seat licensing for teams ($8/user/mo) with free tier for individual open‑source use.

Notes

  • Several users expressed frustration at being forced to use Devin’s CLI or desktop app (scronkfinkle, wren6991, esafak) and wanted to evaluate models via their own harnesses. AgentHarness directly solves that lock‑in.
  • By providing a standard output format (e.g., unified diff), it enables automated comparison pipelines and easy integration with existing code review tools.
  • Could spark discussion on what the “universal agent contract” should look like, inviting contributions from the agent‑builder community.

BenchMax Detector – Analytics Dashboard for Benchmark Overfitting

Summary

  • A web‑based analytics service that ingests a model’s scores across multiple public coding benchmarks (Terminal Bench, HumanEval, MBPP, FrontierCode, etc.) and visualizes performance spikes, variance, and correlation to detect likely benchmaxxing.
  • Core value: gives model providers and consumers a quick, quantitative signal of over‑optimization on narrow benchmarks, encouraging more robust evaluation practices.

Details

Key Value
Target Audience AI labs, benchmark curators, and purchasers of AI coding models who want to assess generalization before adoption.
Core Feature Upload CSV/JSON of benchmark results; the dashboard computes z‑score spread, calculates a “BenchMax Index” (high score on one benchmark, low on others), and highlights outlier benchmarks with drill‑down to task‑level examples.
Tech Stack Backend: Go (Gin) for API; Frontend: Svelte + Chart.js; Data store: PostgreSQL; Deployment: Docker‑Compose or Fly.io.
Difficulty Low
Monetization Hobby (free public instance); optional private instance for teams at $20/mo.

Notes

  • The discussion repeatedly pointed out huge gaps between TB2.1 (90%+) and TB4 (≈27%) as evidence of benchmaxxing (postalcoder, thereitgoes456). A detector would make such spikes obvious at a glance.
  • Enables labs to self‑audit before releasing model claims, and lets buyers ask for a “BenchMax Report” alongside model cards.
  • Low effort to build but high potential impact; could become a reference point in future model releases, fostering healthier benchmark culture.

Read Later