Scaling laws show that loss falls predictably as model size, data and compute grow.
The Chinchilla result suggested a compute-optimal balance of roughly 20 training tokens per parameter, so many earlier models were too large and under-trained.
Recent models are deliberately trained on far more tokens than that: it costs more at training time but gives a smaller, cheaper-to-serve model with the same quality, which matters when inference dominates cost.
Loading…