Three dominant themes
| Theme | Summary | Supporting quotation |
|---|---|---|
| 1. Public‑set scores are misleading | Many commenters stress that a 99 % result on the public ARC‑AGI‑3 set does not imply real generalisation. | “It doesn't necessarily mean anything to reach 99% score on Arc‑AGI‑3. All of the public set is known in advance, so it's possible to hardcode rules that make this easy for the models.” – modeless |
| 2. The harness sidesteps the test’s intent | Several users argue that the presented method essentially “cheats” by building a game simulator and planning instead of letting the model infer rules organically. | “You go to get your chess ELO. You don't know chess at all… you pull out your laptop and write a chess engine. Then when you go to get ranked, you just copy the moves from the software. Now you're a grand master.” – Tadpole9181 |
| 3. Mechanistic discovery as a genuine breakthrough | Despite the criticism, a number of participants see value in the approach of converting game observations into an executable world model, labeling it a meaningful step toward agency‑level AI. | “State grounding turns raw observations into objects, variables, and relations that can be tracked. Mechanism discovery finds how that state changes under an action and writes the rule as an executable program.” – ClassAndBurn |
These three threads dominate the discussion: skepticism about benchmark results, concerns that the evaluation is being gamed, and tentative praise for the underlying mechanistic‑discovery technique.