Tokenization splits text into subword units (tokens) that map to integer IDs; models work on tokens, not characters or words.
BPE starts from characters (or bytes) and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary reaches a target size, so common words become single tokens and rare words split into pieces.
Rule of thumb: 1,000 English words is roughly 750 tokens; code, numbers and non-English text often use more tokens per word.
Tokenization affects cost, context usage and quality across languages. Alternatives include WordPiece, SentencePiece and byte-level tokenizers.
Loading…