IdeaWell

Project ideas from Hacker News discussions.

Show HN: Gemini 4 Argon plays StarCraft

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1 – Memorization / reliance on training data
- “How do they avoid the LLMs simply working with pre‑training data? Which apparently they do, since the human written bots win.” – jstummbillig
- “Yeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best open‑source Starcraft bots and can reproduce them.” – chis

Theme 2 – Interpreting win‑rate performance and cross‑benchmark consistency
- “It's beyond insane that Astra achieves above 50%.” – qsort
- “They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.” – cayenne
- “The relative rankings of this (StarSkirmish Bench v0.2) benchmark … matches the order of his GoBench rankings exactly, even though they are different problems at surface level.” – GodelNumbering

Theme 3 – Practical implications, limits, and future directions
- “I noticed Pluto is running on a simple CPU? If so, that's very nice for future RTS games. You just ship (or use a server‑client arch) a bunch of models of various difficulty levels to let players practice.” – deburo
- “What if we let the agents play directly instead of writing bots?” – brap
- “Even the most powerful frontier models on SOTA hardware are not fast enough for starcraft, which generally involves multiple informed decisions per second at peak moments.” – devmor

🚀 Project Ideas

StarSkirmish Hub: Extensible LLM Game Bot Benchmark Platform

Summary

  • Provides a unified, configurable benchmark harness for running LLMs to write bots across multiple RTS games (StarCraft:BW, SC2, etc.) with adjustable time limits, tool access, and persistent agent state between matches.
  • Core value proposition: Enables fair, repeatable measurement of LLM agentic abilities (adaptability, strategy, learning speed) while reducing noise from memorization and harness variability.

Details

Key Value
Target Audience AI researchers, LLM evaluators, game AI developers
Core Feature Modular benchmark harness supporting multiple games, persistent agent memory, multi-harness testing, automated win‑rate‑vs‑time tracking, and code‑quality metrics
Tech Stack Python, Docker, BWAPI/SC2API, Redis for state, FastAPI + React UI, Prometheus/Grafana for monitoring
Difficulty Medium
Monetization Revenue-ready: Subscription tiers for private benchmarks and premium analytics

Notes

  • HN users expressed interest in timing‑to‑skill: M_w_ said “It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.” – this platform directly measures that.
  • Commenters also noted harness differences affect results (aleksiy123); the hub lets you swap harnesses and compare performance side‑by‑side, which HN discussants would love to see.

PlutoPractice: Adaptive Opponent Training Arena for LLMs

Summary

  • A service where LLMs can train against specific high‑level bots (like Pluto) via limited‑time self‑play or RL, with adjustable difficulty levels and instant feedback loops.
  • Core value proposition: Focuses LLM training on adversarial targets to improve win‑rates against top‑tier bots, addressing the frustration that LLMs struggle against A/S‑tier opponents.

Details

Key Value
Target Audience LLM developers, AI game researchers
Core Feature Arena for submitting LLM‑generated bot code, playing matches against Pluto and other bots, skill‑tier opponent models, optional RL fine‑tuning with bounded compute
Tech Stack Python, RLlib (or custom RL loop), BWAPI, Kubernetes for scaling, WebSocket for live match streaming, PostgreSQL for logs & leaderboards
Difficulty High
Monetization Revenue-ready: Pay‑per‑match or subscription for private training runs

Notes

  • cayenne mentioned “I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)” – PlutoPractice builds exactly that capability.
  • wredcoll noted the usefulness of “a bunch of models of various difficulty levels to let players practice”; this service provides those models on demand, which HN commenters would find valuable for iterative LLM bot improvement.

BotCodeLinter: LLM‑Generated Game Bot Code Quality Analyzer

Summary

  • Static analysis tool that evaluates LLMs’ generated bot code for maintainability, modularity, and best‑practice adherence, providing metrics and refactoring suggestions.
  • Core value proposition: Helps reduce the monolithic, spaghetti‑style code often produced by LLMs, turning bot outputs into clean, reusable components.

Details

Key Value
Target Audience Developers evaluating LLM codegen, AI safety researchers
Core Feature Linting of bot code (function count, coupling, cyclomatic complexity, duplicate files, config‑file usage) with GitHub integration, badge, and actionable reports
Tech Stack Python (ast, radon, pylint), GitHub Actions, Docker, React dashboard for reports
Difficulty Low
Monetization Hobby (open‑source) – can be extended to a SaaS for private repos if desired

Notes

  • serf observed “LLMs have been trending away from those paradigms into small self‑encapsulated programs… because those are less abusive to ctx count and produce better results”; BotCodeLinter encourages exactly that trend.
  • cayenne earlier warned about “huge unmaintainable monoliths” from early LLM bot work; this linter would give HN commenters a concrete way to avoid that pitfall and discuss code‑quality improvements in LLM‑generated game bots.

Read Later