What are model parameters?
Parameters are the learned numbers, also called weights, that make up a language model. Model size is quoted in billions of parameters and written with a B, such as 7B, 30B or 70B. More parameters generally mean a more capable model, but they also mean more memory and slower responses. Memory needs scale directly with parameter count and precision: about 2 GB per billion parameters in FP16, and a little over 0.5 GB per billion at 4-bit quantization. On this site's hardware, the 32 GB Apple Mac mini (M6) fits quantized models of roughly 30B. Machines with 96 GB, such as the MINISFORUM UM890 Pro, fit 70B quantized models, and the 128 GB AMD Ryzen AI Halo can load 200B models. Some models use a mixture-of-experts design, in which only part of the parameters is active for each token. This speeds up generation.