Quantization explained
Quantization stores a model's weights with fewer bits than they were trained with. This shrinks memory use at a small cost in output quality. A model distributed in FP16 uses 16 bits (2 bytes) per parameter, while common quantized formats use around 4 to 8 bits. As a rough guide, a 4-bit quantized model needs a little over half a gigabyte per billion parameters. That means a 70B model drops from about 140 GB in FP16 to roughly 40 GB. This is why the MINISFORUM UM890 Pro, with 96 GB of RAM, can fit 70B quantized models, and why the 32 GB Apple Mac mini (M6) handles quantized models of around 30B. Smaller files also generate tokens faster, because less data has to be read from memory for each token. Lower bit counts, such as 3-bit, save more space but degrade quality more noticeably.