Perplexity is the exponential of the average negative log-likelihood per token; intuitively, how many equally likely next tokens the model is choosing between, so lower is better.
It is useful for comparing language models on the same data and tokenizer and for monitoring pretraining.
It does not measure helpfulness, factual accuracy, safety or instruction following, and it is not comparable across different tokenizers.
Loading…