An LLM is a transformer neural network trained on huge amounts of text to predict the next token given the tokens so far.
Generation is autoregressive: the model outputs a probability distribution over the vocabulary, one token is chosen (greedy or sampled), appended to the input, and the process repeats until an end-of-sequence token or a length limit.
Capabilities such as summarization, translation and reasoning emerge from this objective at scale, then are shaped by instruction tuning and preference alignment.
LLMs predict plausible text, not verified truth, which is the root of hallucinations.
Loading…