Are the token counts exact?

For o200k_base and cl100k_base, yes: the published byte-pair encoding runs in your browser, so token IDs, counts and byte offsets match the encoder. Claude, Gemini and generic options are calibrated approximations of the count only and are always labelled Approximation.

Why is one emoji several tokens?

Byte-pair encodings work on UTF-8 bytes. A character outside the common vocabulary is split into byte-level tokens, each holding part of the character. The explorer groups those tokens into one span and shows each token ID with its bytes in hex.

Is my text sent anywhere?

No. Tokenization runs locally in a background worker, and the text is never stored, logged or uploaded. Only display preferences such as the chosen tokenizer are remembered in your browser.

AI Tokenizer Explorer

See how text becomes tokens, with IDs, bytes and offsets

Input

Updates live up to 50,000 characters. Longer text, up to 2 MB, runs when you select Tokenize.

Options

Tokenizer

Used by GPT-4o, GPT-4.1, GPT-5, o-series

Compare with

Display

Show whitespace

Color

Output

Tokens appear here once there is text to tokenize.

Paste text or load an example to see its tokens.

Counts cover the raw text only. Chat message formatting and special tokens that an API adds are not included. Split this kind of text into chunks with RAG Chunker · Estimate API cost

LLM Cost & Context PlannerEstimate prompt, agent, and production LLM costs before you run them RAG ChunkerSplit documents into retrieval chunks by tokens, sentences or Markdown structure and inspect overlap JSON ↔ TOON ConverterConvert JSON to TOON and back for compact LLM prompts Word CounterCount words, characters, sentences and paragraphs instantly

What the explorer shows

AI Tokenizer Explorer splits your text with the same byte-pair encodings that OpenAI models use (o200k_base and cl100k_base) and draws every token as a span. Select a token to see its ID, decoded text, raw bytes in hex, its UTF-8 byte range and the characters it covers. Stats show the token count, characters, UTF-8 bytes and characters per token.

Exact and approximate counts

o200k_base and cl100k_base are exact: the published vocabulary runs in your browser, so token IDs and counts match the encoder. Claude, Gemini and the generic option are approximations from a calibrated local heuristic. They give an estimated count only, are always labelled Approximation, and never show token boundaries, because their tokenizers are not available locally. Models that are not listed are unsupported.

Bytes, emoji and multilingual text

Tokens are byte sequences, not characters. An emoji or a character from a non-Latin script can be split across several tokens; each of those tokens holds only part of the character and decodes to the replacement character on its own. The explorer groups such tokens into one span, lists all their IDs and keeps the exact byte offsets.

Comparing tokenizers

Pick a second tokenizer under Compare with to see both totals, the difference and the percentage change. The comparison is useful for spotting how much a model change affects prompt size; a comparison that involves an approximation is only indicative.

Privacy and limits

Tokenization runs locally in a background worker and your text is never stored or sent anywhere. Text up to 50,000 characters updates live; longer text up to 2 MB runs when you select Tokenize. The view renders the first 5,000 tokens while counts and copied lists include every token. Counts cover raw text only, without the message formatting or special tokens an API adds.

AI Tokenizer Explorer

Input

Options

Output

Related Tools