Fine-tune when prompting and retrieval are not enough: you need consistent style or output format, domain-specific behavior, lower latency and cost through a smaller model, or you have lots of good examples.
Do not fine-tune to add fast-changing facts; use retrieval for that.
Full fine-tuning updates all weights: highest capacity but heavy GPU memory, large checkpoints, and a higher risk of forgetting.
Parameter-efficient methods (LoRA, adapters, prefix tuning) update a small number of parameters, with far lower cost and swappable task-specific adapters.
Loading…