Stage 1, supervised fine-tuning (SFT): train on high-quality instruction and response examples so the model follows instructions.
Stage 2, reward model: humans rank multiple model responses for the same prompt; a model is trained to predict those preferences as a score.
Stage 3, reinforcement learning (PPO): optimize the language model to maximize the reward score, with a KL penalty that keeps it close to the SFT model.
Risks: reward hacking (exploiting flaws in the reward model), instability, and the cost of collecting good preference data.
Loading…