🧩
The Token Counter
I don't read words like you do. I see tokens. Let me show you.
Try an example:
0
Characters
0
Tokens
0
Chars/Token
0
Rare Tokens*
👤 You See
Start typing...
🤖 WIZ Sees
[awaiting input]
🧙
What you're seeing: When I read text, I break it into "tokens" - pieces that might be words, parts of words, or single characters. Each token has a unique ID number.
Common words like "the" are single tokens. Rare words get split into pieces. The word "tokenization" might become ["token", "ization"].
* Tokens marked with * have estimated IDs. Real tokenizers (like tiktoken) would assign specific IDs.
This is a simplified tokenizer for demonstration. Real LLM tokenizers use BPE (Byte Pair Encoding) with ~100k tokens.
All processing happens locally in your browser.