LocalLLM Rigs

What is tok/s?

tok/s means tokens per second, the standard speed measure for LLM output. A token is a chunk of text, often part of a word. In English, a token averages roughly three-quarters of a word. During generation the hardware reads essentially the whole model from memory for every token. A rough theoretical ceiling is therefore memory bandwidth divided by model size. For example, a 4-bit 70B model of about 40 GB caps out near 15 tok/s on the Apple Mac Studio (M5 Max) at 614 GB/s, and near 2 tok/s on a mini PC with 89.6 GB/s. Real results land below these ceilings. Prompt processing, which is reading your input before replying, is measured separately and depends more on compute than on bandwidth.