What is memory headroom?
Headroom is the spare memory left after a model's weights are loaded. A local LLM needs more memory than the size of its model file. The KV cache, which holds your conversation context, grows with prompt length, and the operating system and other apps need memory too. If headroom runs out, the model may fail to load, the context may have to be shortened, or the system may slow down sharply. For example, a 70B model quantized to 4 bits takes roughly 40 GB. On a 96 GB MINISFORUM UM890 Pro, that leaves around 50 GB of headroom, but the same model would not fit at all on the 32 GB Apple Mac mini (M6). Headroom matters most on fixed-memory machines such as Apple's Mac mini and Mac Studio. Their memory cannot be upgraded later, so you have to choose capacity at purchase.