1. Whether the model is truly an LLM
Commenters debated if a transformer trained from scratch on non‑language data qualifies as a “large language model.”
- evilmathkid: “Its not an LLM if there's no pretraining.”
- dakolli: “Nowhere does he say he built an llm. Hes using a transformer, not an llm.”
- embedding‑shape: “Please describe what in your mind a 'LLM' is exactly… To me this sounds like 'He's not building a calculator, he's just building a program that can do addition…'”
2. Benchmarking concerns – training on eval set and overfitting
Many questioned whether using the ARC evaluation puzzles during training constituted cheating or benchmaxxing.
- jrflo: “The point of ARC is essentially an 'IQ Test' for AI systems… training on eval set… can indirectly cause 'overfitting' by tailoring your model to do well on the eval set.”
- embedding‑shape: “But benchmaxxing is what we generally try to avoid… We used to call it 'overfitting', now you're saying this person does it intentionally?”
- porridgeraisin: Clarified that training on the questions (not the answers) is a legitimate transductive‑learning technique, not label leakage.
3. Sample efficiency and low computational cost
The author stressed achieving strong results with minimal compute (≈ $0.67), highlighting sample efficiency as a key goal.
- evilmathkid: “Sample Efficiency is one of the most important unsolved problems today… We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible.”
- bbor: “Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost…”
- evilmathkid (later): “44% @ 67 cents was a good stopping point for me… more compute would require a lot of effort…”
4. Demonstrating that LLMs are not necessary for certain tasks
Several participants saw the work as a counter‑point to the prevailing LLM‑centric hype, showing that specialized ML can succeed without large language pretraining.
- bkaae: “I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture.”
- evilmathkid: “One of the points was that extremely complex problems can be tackled without LLMs.”
- K0balt: “using the legos of LLMs to build highly specialized machine learning systems, which is a very pragmatic approach.”