What is prompt processing?
Prompt processing, also called prefill, is the first phase of a model's response. The model reads every token of your prompt and builds the context before the first output word appears. Its speed is measured in tokens per second and depends mostly on compute power, such as GPU cores or TOPS. The token-generation phase that follows depends mostly on memory bandwidth instead. Prompt processing is usually much faster per token than generation. With long inputs, such as pasted documents or long chats, it can still cause a noticeable wait before the reply starts. This is where accelerators help. The Apple Mac mini (M6) lists GPU Neural Accelerators that speed up prompt processing. Our catalog lists compute as, for example, 20 GPU/NPU cores on the Mac mini (M5 Pro) or 50 TOPS on the AMD Ryzen AI Halo. The related metric to watch is time to first token.