Tokenization Explained

Models don't see words

When you send text to an LLM, it doesn't read words or letters — it reads tokens. A token is a common chunk of characters: sometimes a whole word, sometimes a piece of one. "tokenization" might be split into token + ization; a common word like "the" is one token. As a rule of thumb, 1 token ≈ 4 characters of English, or about ¾ of a word. Everything the model does — reading your prompt, generating a reply — happens in these units.

Why tokens matter to you

This isn't trivia; tokens shape cost, limits, and behavior:

• Cost and rate limits are measured in tokens, not words. A wordy prompt literally costs more.
• Context windows are token budgets — a "128k context" means 128,000 tokens of prompt plus response must fit together.
• Weird behavior with numbers, code, or rare words often traces back to how they were tokenized — a number like 12345 may split into several tokens in odd ways, which is part of why models fumble arithmetic.

Count tokens yourself

You can see exactly how text tokenizes with a library like tiktoken:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print(len(enc.encode("AnybodyCanAI makes AI approachable")))

Run it on your own prompts and you'll quickly build intuition for what's "expensive." Try a paragraph of English versus the same length of code or another language — the token counts vary a lot.

Practical implications

• Trim fat prompts. Fewer tokens means lower cost and more room for the actual content.
• Budget your context. When stuffing documents into a prompt (as in RAG), count tokens so you don't silently overflow the window.
• Different languages tokenize differently — non-English text often uses more tokens per word, which affects both cost and how much fits.

The takeaway

Tokens are the true unit of LLMs — what they read, what they generate, and what you're billed for. Understanding them turns vague worries about "cost" and "limits" into something you can measure and control.

Try it now: Run the tiktoken snippet on a prompt you use often, then on a trimmed version. Seeing the token count drop is the fastest way to internalize why concise prompts save money.