LocalLLM Rigs

What is FP16?

FP16 is a 16-bit floating-point number format, also called half precision. Many open models are published with their weights in FP16 or in the similar 16-bit BF16 format. That makes it the usual reference size before quantization. Each parameter takes 2 bytes, so memory needs are easy to estimate: a 7B model is about 14 GB, a 30B model about 60 GB and a 70B model about 140 GB, before adding memory for context. That last figure exceeds 128 GB, the highest memory ceiling in this site's catalog, offered by machines such as the Apple Mac Studio (M5 Max). In practice, most readers running models on mini PCs and Macs use quantized versions instead. FP16 is mainly kept for small models or for quality comparisons.