LocalLLM Rigs

What does 70B mean?

70B is shorthand for a model with roughly 70 billion parameters, the learned numbers that make up a language model. Size sets how much memory you need. At full 16-bit precision a 70B model needs about 140 GB just for its weights. Quantized to around 4 bits, it shrinks to roughly 35–40 GB, plus extra room for context. That is why the Apple Mac mini (M6), capped at 32 GB, cannot run 70B models, while the MINISFORUM UM890 Pro (96 GB) and the MINISFORUM AI X1 Pro-370 (128 GB) can load quantized versions. Fitting the model is only half the story. Every generated token reads the whole model from memory, so on a machine with 89.6 GB/s of bandwidth, a 4-bit 70B model tops out at about 2 tok/s in theory.