Pretraining: next-token prediction on a massive corpus; the expensive stage that gives general language and world knowledge.
Fine-tuning: further training on a smaller task or instruction dataset, which changes the model's weights.
In-context learning: the model adapts to a task from instructions and examples placed in the prompt, with no weight updates.
Start with prompting, add retrieval for knowledge, and fine-tune only when you need consistent style or behavior, lower cost, or higher quality than prompting reaches.
Loading…