What a token actually is.

A token is the smallest unit of text a language model processes. It is not a character and not a word. It sits somewhere in between. For English text, one token is roughly four characters or about three-quarters of a word. For code, numbers, and non-Latin scripts, the ratios shift significantly.

The reason tokens exist at all is computational. Models cannot handle raw characters efficiently because the English vocabulary is too long for practical processing. Models cannot handle full words either because the vocabulary would need to contain millions of entries to cover every conjugation, plural, and compound. Tokens are the compromise. A fixed vocabulary of typically 50,000 to 200,000 subword units covers all real-world text with minimal storage.

The process of turning text into tokens is called tokenization. It happens before the model sees your input and after the model generates its output. When the model decides "dog" is the next token, it emits a number that the tokenizer converts back into the string "dog" before you see it. Every step of generation is a token decision.

How tokenizers split text.

All three major tokenizers use some form of Byte Pair Encoding (BPE). The algorithm starts with individual characters and iteratively merges the most common pairs. After millions of merges on training data, common sequences like "the", "ing", and "tion" become single tokens. Rare sequences remain as multiple tokens.

This creates some intuitive behavior and some counterintuitive behavior. Common words get one token each. Rare words split into multiple tokens. Numbers are almost always expensive because long digit sequences are rare in training data. Non-English scripts can be extremely expensive because the tokenizers were trained primarily on English.

For an illustrative example: the English sentence "The tokenizer splits this sentence" is eight tokens with most tokenizers, even though it has six words. The word "tokenizer" splits into "token" plus "izer" because "tokenizer" itself was not common enough in training data to warrant a single token. "Sentence" splits into "sen" plus "tence" for the same reason.

Claude tokenization.

Claude uses a proprietary BPE-based tokenizer. Anthropic has not published the full vocabulary but has released the official count_tokens API endpoint for exact counts. For estimation, the rule of thumb is 3.5 characters per token for English prose, 3 for code, and 2 for numeric content.

Claude tokenization tends to be slightly more granular than GPT tokenization, which means Claude will generally report more tokens for the same input. The practical impact is small for short prompts but can matter at scale. A million characters of English text produces roughly 285,000 Claude tokens versus 250,000 GPT tokens.

For exact counts in production code, use the Anthropic SDK's messages.countTokens method. It is free and does not consume your API rate limit. Call it before calling the actual generation endpoint if you need precise cost forecasting or strict context window enforcement.

GPT tokenization.

OpenAI uses tiktoken, an open-source BPE tokenizer with several encoding variants. GPT-3.5 and GPT-4 use cl100k_base with a 100,000 token vocabulary. GPT-4o and newer models use o200k_base with 200,000 tokens, handling non-Latin scripts and emoji far more efficiently than the older encoding.

The rule of thumb for GPT is 4 characters per token for English prose. The OpenAI tokenizer demo page shows the exact breakdown for any input text. For JavaScript and TypeScript code, js-tiktoken provides the official tokenizer in a format that runs in browsers.

Chat completions add a small per-message overhead. Each message in the messages array consumes an extra 3 to 4 tokens for role metadata, message boundaries, and formatting. Over long conversations this adds up. A 50-turn conversation might have 150 to 200 tokens of pure structural overhead on top of the content tokens.

Gemini tokenization.

Google uses SentencePiece, a different subword tokenization algorithm that handles multilingual text more efficiently than BPE. For English, the rule of thumb is 4.2 characters per token, slightly better than GPT and noticeably better than Claude.

Gemini exposes a dedicated countTokens endpoint both in the Gemini API and Vertex AI. Like Anthropic's endpoint, it is free and does not count against quotas. For long document analysis, call countTokens before the generation call to confirm the prompt fits within the context window, especially for multi-megabyte inputs.

Where Gemini tokenization really shines is multilingual content. CJK languages (Chinese, Japanese, Korean), Arabic, and Cyrillic scripts tokenize far more efficiently with SentencePiece than with tiktoken's default encodings. A document in Japanese might use 40 percent fewer tokens with Gemini than with GPT or Claude.

The three levers for cutting tokens.

Reducing token usage is the single highest-ROI optimization for any production AI application. Three strategies stack for combined savings of up to 95 percent on production workloads, and they work across all three major providers.

Lever 1: Prompt caching.

Anthropic prompt caching reduces cached input tokens to 10 percent of the standard rate. OpenAI offers similar discounts on cached inputs. Google's context caching works the same way. The rule: if the same prefix appears in multiple API calls (a long system prompt, a large document, a set of tool definitions), mark it as cacheable and pay once.

The best candidates for caching are long system prompts over 1,000 tokens, fixed document context that multiple queries reference, tool schemas that do not change between calls, and few-shot example sets. Typical savings: 70 to 90 percent on applications with structured prompts.

Lever 2: Batch processing.

If your workload tolerates up to 24 hours of latency, batch processing provides a flat 50 percent discount on both input and output tokens. Anthropic and Google offer similar batch discounts. The use case is offline jobs: classification, summarization of archives, data enrichment, ETL-style operations, and evaluation runs.

The decision is operational: can the result wait until tomorrow? If yes, batch. If the user is waiting on a screen, do not batch. For asynchronous workloads, batch cuts the bill in half with essentially zero code changes.

Lever 3: Tiered routing.

Not every task needs a flagship model. Simple classification, extraction, and formatting tasks run fine on Haiku, GPT-Nano, or Flash-Lite at one tenth the cost. Hard reasoning tasks need Opus, GPT Pro, or Gemini Pro. The pattern is called tiered routing: an orchestrator decides which model to call based on task complexity.

A typical split is 70 percent of calls to the cheapest model, 20 percent to the mid-tier, and 10 percent to the flagship. Done well, this cuts 60 to 80 percent of cost with minimal quality loss. The tradeoff is engineering effort to build the router, which matters less as traffic scales.

Stacked together, the three levers can take a $60 workload down to $8.50, an 86 percent reduction. The math is multiplicative: half-off from batch, times tenth-off from caching, times tiered routing on top.