Tokens (LLM)
Tokens are the units a language model reads and generates — usually words, word-pieces, or punctuation, with leading spaces attached — and are the basis for both context limits and billing.
Models do not process text as words or characters but as tokens. In English, a token averages about four characters, or roughly three tokens for every four words, though code, punctuation, and other languages tokenize very differently.
Tokens matter for two practical reasons: context windows are measured in tokens, so they cap how much a model can consider at once, and API pricing is per token, usually with output priced higher than input. Estimating token usage is the first step in estimating cost.
Learn this stuff by email
New definitions, tools, and explainers when I publish them. No spam, unsubscribe anytime.