Tokenizer visualizer
Paste any text to see it broken into approximate tokens — the units a language model actually reads and is billed for — with each token shown as a colored chip.
≈ 18 tokens
Approximate visualization. Real tokenizers apply model-specific merges this heuristic can’t reproduce, so exact counts differ — use each model’s own tokenizer or its API usage metadata for precise numbers. A middot (·) marks a leading space.
How it works
Language models do not read words; they read tokens. Before anything else, text is pre-tokenized on word, number, and punctuation boundaries, with leading spaces attached to the following word — which is why a space is part of the token, shown here with a middot.
Real tokenizers then apply model-specific merge rules (byte-pair encoding) learned from training data, which this visualizer cannot reproduce without each model's vocabulary. To approximate, long word-pieces are split into short subchunks, so the shape and rough count are representative even though the exact boundaries differ from any given model.
The value is intuition: you can see why whitespace, punctuation, code, and non-English text tokenize very differently from plain English prose, and why token count rarely equals word count. Everything runs in your browser.
Assumptions and limitations
- This is an approximation, not a model's real tokenizer. Exact tokens and counts require the specific model's byte-pair encoding, which differs across vendors and versions.
- The heuristic is tuned for English prose. Code, URLs, CJK languages, and unusual punctuation will diverge more, usually toward overcounting long words.
- It visualizes plain text; it does not add the chat formatting, role markers, or system prompt overhead that real API requests carry.
Frequently asked questions
What is a token in an LLM?
A token is the atomic unit a language model reads and generates — often a word, a word-piece, or a punctuation mark, with leading spaces attached. Models process and are billed by tokens, not words or characters, which is why token count is the number that matters for cost and context limits.
How many tokens is a word?
For ordinary English, roughly 1.3 tokens per word or about four characters per token. Short common words are usually a single token; rare or long words split into several. This visualizer shows that splitting directly.
Why is this tokenizer approximate and not exact?
Exact tokenization depends on a model's byte-pair-encoding vocabulary — the specific merges it learned during training. Without that vocabulary the exact boundaries can't be reproduced, so this tool approximates the shape and count. For precise numbers, use the model's own tokenizer or its API usage metadata.
Why does whitespace show as a middot?
Leading spaces are part of the following token in most modern tokenizers, so ' token' and 'token' are different tokens. The middot (·) marks those spaces so you can see them rather than lose them visually.
Is the text I paste stored anywhere?
No. Tokenization runs entirely in your browser with client-side JavaScript. Nothing you paste is transmitted, logged, or stored.
More free tools like this, by email
I publish new calculators and explainers regularly. Get them when they land — no spam, unsubscribe anytime.