What is a token?
A token is the basic unit of text a language model processes. Models don't read letters or whole words. A tokenizer splits text into pieces that may be a whole word, part of a word, a punctuation mark or a space. In English, a token averages about 4 characters, or roughly 0.75 words, so 1,000 tokens is about 750 words. Exact counts depend on the model's tokenizer and on the language, and code or non-English text often uses more tokens. Tokens matter because nearly every local-LLM figure is measured in them. Speed is quoted in tokens per second, and a model's context window, meaning how much text it can consider at once, is a token count. When a review on this site reports generation speed for a machine, it is counting these units.