Theme 1 – Hardware limits on mobile LLMs
- “Apple historically is extremely stingy when it comes to RAM and they never bothered giving iPads and iPhones more ram until fairly recently… Your covid era 10th Gen iPads only have 4GB of RAM for the base models.” – HawtAds
- “The Android ecosystem is much more liberal when it comes to RAM… but the main issue with Android is that their mid/low end … are very inefficient when it comes to single core compute performance.” – HawtAds
- “At the end of the day, the current gen of ‘pocket scale’ LLMs are still far from being able to be deployed at scale on mobile.” – HawtAds
Theme 2 – Practical mobile ML relies on tiny, transfer‑learned models
- “Current gen production mobile ML still heavily rely on traditional transfer learning like Apple's Core.ML (tiny pre trained models, with perhaps the final layer inference head swapped out).” – HawtAds
- “I would say right now 100M‑300M params is about the maximum you can do if you want any reasonable level of performance on your 'average' non flagship mobile. I would keep to sub 100M for power efficiency sake.” – HawtAds
- Example: “FUTO keyboard's transformer model for their keystroke prediction/autocorrect which operates in near real‑time.” – HawtAds
Theme 3 – Benchmark gains don’t always translate to real‑world usefulness
- “Qwen 3.5 9B scores 2‑3x higher than 4o (depending on the 4o version), on the benchmarks.” – andai
- “Whether it's actually better for the kind of things people actually use it for… the benchmarks don't really tell you that.” – andai
- “In my experience even small local models are now surprisingly good at programming and using a computer (bash), i.e. completing agentic tasks, but fall apart quickly in conversation (especially knowledge and understanding).” – andai