IdeaWell

Project ideas from Hacker News discussions.

Breaking the 1.58-bit Barrier for Ternary LLMs

Original Article

Hacker News Discussion

📝 Discussion Summary (Click to expand)

Theme 1 – Ternary quantization offers real compression and hardware‑friendly speed gains
- “Woah. Good science.” – Kevcmk
- “This is the only time '1.58 bit' phrase makes more sense than '1 trit'” – NooneAtAll3
- “Very interesting, I was just exploring this to hopefully fit one of the latest quantized models in 16GB of VRAM.” – plqbfbv
- “They get down from 1.58 to 1.48 bits per weight by exploiting the fact that actual weights in practice are 0 51% of the time.” – infogulch
- “If ternary LLMs work out and are baked into hardware as custom silicon I bet they'll be shockingly efficient.” – infogulch
- “Sounds like a perfect fit for ASIC‑optimized models … achieving record power efficiency for on‑device inference.” – yalok
- “Pushing past log₂(3) for real. This could drastically shrink LLMs for embedded systems, making them truly portable.” – Marchant_hq
- “It also means you can read them faster, more parameters per second during an inference which tends to be memory bandwidth limited on most systems. Thus faster inferences.” – pieter3d

Theme 2 – Trade‑offs vs. other quantization schemes (vector QT, PTQ, QAT)
- “Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.” – om8
- “If you want sub‑2‑bit LLM, get one that’s already trained in higher precision, and compress it with something like YAQA/QTIP with finetuning or PV‑tuning + AQLM/HIGGS.” – om8
- “PTQ and vector quantization aren’t used for this because part of the point of ternary LLMs is to make them faster. In a ternary LLM every weight is an add, subtract, or no‑op so it is fast on CPU.” – janalsncm
- “If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.” – janalsncm
- “That’s why you need to use efficient GEMM kernels like FLUTE for inference. They are ~as good as what you can do with ternary quantization.” – om8
- “We can barely quantize to dynamic fp4 with small block size – still not completely lossless on all benchmarks.” – kadushka
- “The average information content of transformer LLMs is about 3‑4 bits per parameter, but 4‑bit block‑wise quantization does not guarantee preserving 4 bits of useful information per parameter.” – kadushka
- “QAT helps, but usually at the cost of learning efficiency.” – kadushka
- “You are conflating post‑training quantization and low‑bit training.” – danielmarkbruce
- “Bitnet also keeps high‑precision latent weights during training; the optimizer updates those, while the forward‑pass weights are quantized to ternary values.” – danielmarkbruce

Theme 3 – Practical deployment: bandwidth, inference speed, model scaling, and accuracy‑recovery tricks
- “Sending it across the wire over and over … is actually the main bottleneck.” – mitxela (referring to janalsncm’s wire‑saving comment)
- “Another complementary option is, if the model is fast enough, we should be able to push up correctness by self‑consistency voting at close to T=1. Smart/fast Zero‑shot classifiers like the recent Jev could help with aggregation across answers …” – Vetch
- “I'm not convinced by this argument – if such a method improves accuracy of a degraded quantized model, then it could in theory also help non‑degraded full‑precision model. And if so, then we are back to square one …” – kadushka
- “We do know one thing – increasing the size of the model usually makes it more robust to quantization. If going from 8 bits to 2 bits speeds things up by a factor of, say, 4×, then if we double the size of the model, we might still end up with an overall speedup.” – kadushka
- “Only a presence bitmap? If we're contemplating packing schemes I'm tempted to write a paper that uses arithmetic coding to squeeze out a few more centi‑bits.” – wgd
- “Per [0], a model needs only ~30% more weights to be at comparable quality, if quantization‑aware training is done …” – yalok
- “We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to 51.5% of all weights. Motivated by this finding, we introduce BITCOS, a simple distribution‑adaptive layout.” – c7b

These three themes capture the dominant conversation: the promise of ternary quantization for compact, fast hardware‑friendly models; the skepticism and comparison with alternative quantization techniques; and the real‑world engineering concerns around bandwidth, inference speed, model scaling, and methods to recover accuracy.

🚀 Project Ideas

BitCos Packer: Adaptive Ternary Weight Compression Library

Summary

  • Automatically analyzes ternary LLM weight distributions and packs them using the BITCOS layout (zeros ≈ 51 %), reducing storage to ~1.58 bits/weight and enabling faster, memory‑bandwidth‑friendly inference.
  • Core value proposition: drop‑in replacement for existing weight files that cuts RAM/VRAM usage by ~40 % without retraining, while preserving inference speed via SIMD‑friendly decoding.

Details

Key Value
Target Audience ML engineers/researchers deploying LLMs on edge devices, cloud inference services seeking lower memory footprint
Core Feature Distribution‑aware bit‑packing/unpacking of ternary weights with metadata for zero‑copy decompression
Tech Stack Python (NumPy), optional Rust extension for high‑speed bit‑wise ops, support for PyTorch/TensorFlow via custom loader
Difficulty Medium
Monetization Hobby

Notes

  • HN users highlighted the BITCOS finding: “zeros account for up to 51.5% of all weights” and wondered why such packing isn’t default (c7b). This tool directly addresses that gap.
  • Enables discussion on distribution‑adaptive coding vs. static quantisation, and can be benchmarked against existing quantization‑aware training pipelines.

Self‑Consistency Voting API for Quantized LLMs

Summary

  • Provides a lightweight inference wrapper that runs a quantized LLM multiple times at temperature ≈ 1 and aggregates answers via majority voting (self‑consistency) to recover accuracy lost to low‑bit quantisation.
  • Core value proposition: improves output quality of sub‑2‑bit LLMs without additional training, trading a small latency increase for higher correctness—useful when model speed is already high (as noted by janalsncm).

Details

Key Value
Target Audience Developers using ternary or ultra‑low‑bit LLMs on latency‑constrained hardware (e.g., edge, mobile)
Core Feature Configurable‑round voting service with caching of intermediate logits to amortize cost
Tech Stack FastAPI, HuggingFace Transformers (with bitsandbytes or custom ternary kernels), Redis for cache, optional GPU acceleration via Triton
Difficulty Medium
Monetization Revenue-ready: pay‑per‑API‑call (e.g., $0.0005 per 1k token‑votes)

Notes

  • Commenters noted self‑consistency voting can push correctness upward for fast models (janalsncm) and that zero‑shot classifiers like Jev could help aggregate answers. This product turns that insight into a usable service.
  • Opens discussion on trade‑offs between inference steps vs. model size, and invites benchmarking on popular benchmarks (MMLU, GSM8K) for various ternary models.

Ternary‑to‑Hardware Codegen: ASIC/FPGA Kernel Generator for BITCOS‑Packed LLMs

Summary

  • Takes a BITCOS‑packed ternary LLM and emits optimized compute kernels (C intrinsics for SIMD, Verilog for FPGA/ASIC) that exploit the add/subtract/no‑op nature of ternary weights, enabling ultra‑low‑power inference.
  • Core value proposition: bridges the gap between software quantisation research and hardware realization, letting designers generate custom silicon‑ready datapaths with minimal manual effort.

Details

Key Value
Target Audience Embedded system designers, ASIC/FPGA engineers, research groups building custom AI accelerators
Core Feature Automatic generation of ternary GEMM kernels and control logic from packed weight files, with configurable bit‑width and parallelism
Tech Stack Python (sympy/numpy), LLVM‑MLIR for C kernel generation, Jinja2 templates for Verilog, optional integration with Vitis/Vivado
Difficulty High
Monetization Hobby (open‑source) – potential future licensing for commercial IP cores

Notes

  • HN discussion emphasized the speed advantage of ternary LLMs on CPU (“every weight is an add, subtract, or no-op”) (janalsncm) and the excitement about ASIC‑optimized models (yalok). This tool makes that vision practical.
  • Generates concrete artifacts that can be benchmarked against existing quantization‑aware training pipelines and invites collaboration on open‑source hardware projects like OpenROAD or VexRiscv.

Read Later