Direct Preference Optimization trains directly on preference pairs (prompt, chosen response, rejected response) with a simple classification-style loss that raises the likelihood of the chosen response relative to the rejected one, measured against a frozen reference model; a beta parameter controls how far the model may drift.
It needs no separate reward model and no reinforcement-learning loop, so it is simpler, more stable and cheaper than PPO.
Training data: collect or generate two responses per prompt, label which is better (human or strong-model judge), and filter noisy pairs.
PPO-style methods can still be stronger when you have a good reward signal, such as verifiable rewards for math and code.
Loading…