How to build a diffusion language model

Learning and grasping the ELBO through structured naming – Users found that giving names to the underlying mathematical structures made the derivation “quite simple” and “a very fun and elucidating exercise.”

“Once you give names to the larger mathematical structures and understand them a bit better it becomes quite simple.” – quirino
“Last week I went over the derivation of the ELBO for a couple hours and it was a very fun and elucidating exercise.” – quirino

Challenges with probability notation and terminology – Many noted confusion over variable “types” and wished papers had explicitly named concepts like importance sampling.

“The probability notation can be pretty confusing too. Sometimes it's hard to understand the ‘types’ of some variables. But I'm inexperienced.” – quirino
“I wish some of the blogs/papers I'd read had named ‘Importance Sampling’.” – quirino

Supplemental resources and AI assistance to fill gaps – Recommendations to look at tools like diffusiongemma and the usefulness of ChatGPT for clarifying unintuitive parts.

“good stuff, no mention of confidence tho, recommend having a look at diffusiongemma and others.” – electroglyph
“ChatGPT was surprisingly helpful. If you put in the work to truly understand the where the gaps are in your mental model… it can do an amazing model filling in the gaps.” – quirino