What Are Tokens in LLMs? Complete Guide to Tokenization
Tokens in LLMs 1. What is a Token? A token is the basic unit of text that a Large Language Model (LLM) processes. An LLM does not directly process complete words or sentences. It first breaks the input text into smaller pieces called tokens . A token can be: A complete word Part of a word A punctuation mark A symbol Sometimes a space combined with a word Example ChatGPT is amazing! The tokenizer might split it into: ["Chat", "GPT", " is", " amazing", "!"] The exact tokens depend on the tokenizer used by the model. 2. Why Do LLMs Use Tokens? Neural networks work with numbers , not raw text. Therefore, text goes through several steps: Text ↓ Tokenizer ↓ Tokens ↓ Token IDs ↓ Embeddings ↓ Transformer ↓ Next Token Prediction For example: "I love AI" may...
