Project ideas from Hacker News discussions.

I trained a small transformer in 1.5hrs and it beats many LLMs

📝 Discussion Summary (Click to expand)

1. Whether the model is truly an LLM
Commenters debated if a transformer trained from scratch on non‑language data qualifies as a “large language model.”
- evilmathkid: “Its not an LLM if there's no pretraining.”
- dakolli: “Nowhere does he say he built an llm. Hes using a transformer, not an llm.”
- embedding‑shape: “Please describe what in your mind a 'LLM' is exactly… To me this sounds like 'He's not building a calculator, he's just building a program that can do addition…'”

2. Benchmarking concerns – training on eval set and overfitting
Many questioned whether using the ARC evaluation puzzles during training constituted cheating or benchmaxxing.
- jrflo: “The point of ARC is essentially an 'IQ Test' for AI systems… training on eval set… can indirectly cause 'overfitting' by tailoring your model to do well on the eval set.”
- embedding‑shape: “But benchmaxxing is what we generally try to avoid… We used to call it 'overfitting', now you're saying this person does it intentionally?”
- porridgeraisin: Clarified that training on the questions (not the answers) is a legitimate transductive‑learning technique, not label leakage.

3. Sample efficiency and low computational cost
The author stressed achieving strong results with minimal compute (≈ $0.67), highlighting sample efficiency as a key goal.
- evilmathkid: “Sample Efficiency is one of the most important unsolved problems today… We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible.”
- bbor: “Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost…”
- evilmathkid (later): “44% @ 67 cents was a good stopping point for me… more compute would require a lot of effort…”

4. Demonstrating that LLMs are not necessary for certain tasks
Several participants saw the work as a counter‑point to the prevailing LLM‑centric hype, showing that specialized ML can succeed without large language pretraining.
- bkaae: “I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture.”
- evilmathkid: “One of the points was that extremely complex problems can be tackled without LLMs.”
- K0balt: “using the legos of LLMs to build highly specialized machine learning systems, which is a very pragmatic approach.”


🚀 Project Ideas

ArcTrainer Lite

Summary

  • A lightweight CLI tool to train small autoregressive transformers from scratch on ARC‑style puzzles with minimal compute and cost tracking.
  • Enables researchers to reproduce sample‑efficient results like the 67‑cent experiment without needing full LLM pipelines.

Details

Key Value
Target Audience ML researchers, hobbyists, Kaggle competitors
Core Feature Train/eval loop with automatic USD cost estimation, supports ARC‑AGI‑1/2 datasets
Tech Stack Python, PyTorch, Lightning, HuggingFace Tokenizers, optional CUDA
Difficulty Medium
Monetization Hobby

Notes

  • HN commenters praised the idea of “training from scratch only during the 12 hours allowed on Kaggle” and noted the value of avoiding pretraining (“what you describe would be cheating. My approach is the opposite…”) – this tool codifies that approach.
  • Provides a reproducible, low‑cost way to explore sample efficiency, sparking discussion on whether LLMs are necessary for narrow benchmarks.

ARC‑Integrity Benchmark Hub

Summary

  • A web platform that hosts ARC‑AGI (and similar meta‑learning) benchmarks with enforced train/test separation to prevent leakage and enable fair comparison of sample‑efficient models.
  • Offers sandboxed execution, automatic cost reporting, and leaderboards that highlight models trained from scratch.

Details

Key Value
Target Audience AI researchers, competition organizers, educators
Core Feature Secure evaluation containers that forbid access to evaluation labels, with optional test‑time training mode
Tech Stack React frontend, FastAPI backend, Docker/gVisor sandboxes, PostgreSQL, S3 for artifacts
Difficulty High
Monetization Revenue-ready: per‑private‑benchmark run ($5) or enterprise SaaS plan

Notes

  • Commenters warned that “training on the eval puzzles is cheating” and discussed the need for “private test set performance” – this hub directly addresses those concerns by locking down the eval set.
  • Enables practical utility: researchers can submit models and see unbiased scores, fostering trust in benchmark results and encouraging novel, low‑compute approaches.

SpotTrain Burst

Summary

  • An on‑demand service that spins up cheap GPU spot instances pre‑configured for training tiny transformers on ARC‑like tasks, with automatic shutdown and cost reporting.
  • Lets users replicate low‑budget experiments (e.g., <$1) without managing infrastructure.

Details

Key Value
Target Audience Students, independent ML experimenters, budget‑conscious researchers
Core Feature One‑click launch of a Jupyter notebook with GPU, pre‑installed PyTorch, ARC dataset, and cost‑tracker that stops after a set budget
Tech Stack Wrapper around RunPod/vast.ai APIs, Node.js/Express for UI, Python notebooks, AWS Lambda for orchestration
Difficulty Medium
Monetization Revenue-ready: take 15 % of GPU spot cost or flat $0.10 per session

Notes

  • HN users highlighted the impressiveness of achieving “44 % @ 67 cents” and the desire to “train from scratch only during the 12 hours allowed”; SpotTrain makes that cheap, repeatable experiment trivial.
  • Encourages discussion on sample‑efficiency trade‑offs and lowers the barrier for reproducing cutting‑edge results, potentially spawning new leaderboard entrants.

Read Later