IdeaWell

Project ideas from Hacker News discussions.

I trained a small transformer in 1.5hrs and it beats many LLMs

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1 – Is the model really an LLM?
Many commenters debate whether a transformer trained only on ARC puzzle sequences counts as a “large language model.”
- “Nowhere does he say he built an llm. Hes using a transformer, not an llm.” – dakolli
- “Its not an LLM if there's no pretraining… I did not train it on any language data. Only ARC puzzles.” – evilmathkid
- “A LLM should at the very least be a language model, i.e. be able to take human-readable text as input or produce it as output.” – yorwba
- “Isn't it a LLM he's building though? … if it quacks like a duck…” – embedding-shape

Theme 2 – Benchmark‑specific optimization (benchmaxxing) for the ARC prize
The work is seen as deliberately tuning a model to excel on a particular benchmark rather than pursuing general reasoning.
- “The whole point of his model is to optimize for a very specific benchmark.” – f311a
- “Why not? There is $700k reward for the next iteration of this benchmark… I would not call this overfitting, it's finetuning for specific task where you have a benchmark.” – f311a
- “He is showing computers can solve these issues, without using an LLM architecture… a very pragmatic approach.” – K0balt
- “There are plenty of applications where a machine learning system needs to optimize for a very limited data set… It’s interesting, because he is using the legos of LLMs to build highly specialized machine learning systems.” – K0balt

Theme 3 – Training on eval/test inputs (without labels) and whether it counts as cheating
Discussion centers on using the ARC evaluation set’s inputs during training, which some view as permissible meta‑learning and others as leakage.
- “Training on the eval puzzles is cheating / “training on test”… No this is false. “Training on test” specifically means training on the labels of test data. The labels were not trained on.” – porridgeraisin
- “The point of ARC is essentially an 'IQ Test' for AI systems… The other tension is the fact that this score is on the public eval set… you can use performance against the eval set to modify your hyperparameters… What you really want to see is the private test set performance.” – jrflo
- “You can just think of next-word training Q_test as a 'retrieval' process… This practice often used in continual learning… is not yet useful in general real world ML tasks.” – porridgeraisin
- “I'm currently 10th in the world on the private set on Kaggle. And iirc, at one point I was 4th.” – evilmathkid (showing the method works on private test as well).

🚀 Project Ideas

ArcTrainer Lite

Summary

  • A lightweight CLI tool to train small autoregressive transformers from scratch on ARC‑style puzzles with minimal compute and cost tracking.
  • Enables researchers to reproduce sample‑efficient results like the 67‑cent experiment without needing full LLM pipelines.

Details

Key Value
Target Audience ML researchers, hobbyists, Kaggle competitors
Core Feature Train/eval loop with automatic USD cost estimation, supports ARC‑AGI‑1/2 datasets
Tech Stack Python, PyTorch, Lightning, HuggingFace Tokenizers, optional CUDA
Difficulty Medium
Monetization Hobby

Notes

  • HN commenters praised the idea of “training from scratch only during the 12 hours allowed on Kaggle” and noted the value of avoiding pretraining (“what you describe would be cheating. My approach is the opposite…”) – this tool codifies that approach.
  • Provides a reproducible, low‑cost way to explore sample efficiency, sparking discussion on whether LLMs are necessary for narrow benchmarks.

ARC‑Integrity Benchmark Hub

Summary

  • A web platform that hosts ARC‑AGI (and similar meta‑learning) benchmarks with enforced train/test separation to prevent leakage and enable fair comparison of sample‑efficient models.
  • Offers sandboxed execution, automatic cost reporting, and leaderboards that highlight models trained from scratch.

Details

Key Value
Target Audience AI researchers, competition organizers, educators
Core Feature Secure evaluation containers that forbid access to evaluation labels, with optional test‑time training mode
Tech Stack React frontend, FastAPI backend, Docker/gVisor sandboxes, PostgreSQL, S3 for artifacts
Difficulty High
Monetization Revenue-ready: per‑private‑benchmark run ($5) or enterprise SaaS plan

Notes

  • Commenters warned that “training on the eval puzzles is cheating” and discussed the need for “private test set performance” – this hub directly addresses those concerns by locking down the eval set.
  • Enables practical utility: researchers can submit models and see unbiased scores, fostering trust in benchmark results and encouraging novel, low‑compute approaches.

SpotTrain Burst

Summary

  • An on‑demand service that spins up cheap GPU spot instances pre‑configured for training tiny transformers on ARC‑like tasks, with automatic shutdown and cost reporting.
  • Lets users replicate low‑budget experiments (e.g., <$1) without managing infrastructure.

Details

Key Value
Target Audience Students, independent ML experimenters, budget‑conscious researchers
Core Feature One‑click launch of a Jupyter notebook with GPU, pre‑installed PyTorch, ARC dataset, and cost‑tracker that stops after a set budget
Tech Stack Wrapper around RunPod/vast.ai APIs, Node.js/Express for UI, Python notebooks, AWS Lambda for orchestration
Difficulty Medium
Monetization Revenue-ready: take 15 % of GPU spot cost or flat $0.10 per session

Notes

  • HN users highlighted the impressiveness of achieving “44 % @ 67 cents” and the desire to “train from scratch only during the 12 hours allowed”; SpotTrain makes that cheap, repeatable experiment trivial.
  • Encourages discussion on sample‑efficiency trade‑offs and lowers the barrier for reproducing cutting‑edge results, potentially spawning new leaderboard entrants.

Read Later