What are tokens per second?
Tokens per second (tok/s) measures how quickly a model generates output. To produce each token, the hardware must read essentially all of the model's weights from memory. A useful ceiling is therefore memory bandwidth divided by model size. For a 4-bit 70B model of about 40 GB, that gives roughly 15 tok/s on the Apple Mac Studio (M5 Max)'s 614 GB/s, about 6 tok/s on the AMD Ryzen AI Halo's 256 GB/s, and about 2 tok/s on the 89.6 GB/s MINISFORUM mini PCs. Real results land below these ceilings. Smaller models run proportionally faster. Many people find speeds of around 5–10 tok/s or more comfortable for reading along. Note that prompt processing speed, meaning how quickly input is read, is reported separately and depends more on compute.