Top Themes of the Discussion
| # | Theme | Key Takeaway & Supporting Quote |
|---|---|---|
| 1 | LLM‑augmented categorization using embeddings | Many users rely on embedding pools to fetch a shortlist of candidate categories, then feed that list to an LLM for final classification. “Using a Nano model, a tad worse than shipping a vocabulary to a larger OpenAI model… no classification is perfect.” — softwaredoug |
| 2 | Cost & latency concerns | Deployments stress the need for fast, cheap inference; a 2‑second LLM call was called “pretty slow” and “a huge impact”. “Isn't search engines quicker than calling a LLM? … 2s LLM call is pretty slow.” — Colegno |
| 3 | Critiques & alternative lightweight approaches | Several commenters argue that simple embedding similarity or HyDE‑style hallucinated “prototype” docs often outperform LLMs and are cheaper/faster. “Just embed the query, compare it to the embedding of the categories, then ship only categories that are close to it in the prompt to a smaller model?” — tantalor |
These three themes capture the most common viewpoints: effective yet expensive LLM pipelines, the pressing need for speed and cost efficiency, and the push toward simpler embedding‑based or hybrid solutions.