LocalLLM Rigs

What is llama.cpp?

llama.cpp is an open-source inference engine written in C/C++ for running large language models on consumer hardware. It loads models in the GGUF file format and supports quantization from about 2 to 8 bits per weight, which is what lets a 70B model fit in about 40 GB. It has several GPU backends, including Metal for Apple Silicon, CUDA for NVIDIA, HIP/ROCm for AMD, and Vulkan for many other GPUs, and it can also fall back to the CPU. That means it runs on every machine in this catalog. For example, it can use Vulkan on the Radeon 780M in the MINISFORUM UM870 Slim. Many friendly apps, including Ollama and LM Studio, build on llama.cpp. Benchmarks quoted in tokens per second often come from its bundled llama-bench tool.