Combine methods: exact-match or programmatic checks where answers are verifiable, task-specific metrics, LLM-as-judge with an explicit rubric for quality dimensions, and human review of a sample.
Use public benchmarks (MMLU, HumanEval, GSM8K and others) only as rough signals: they can be contaminated by training data and may not match your use case.
Build a custom evaluation set from real user queries, include failure cases and adversarial prompts, and track regressions across model, prompt and retrieval versions.
Pair offline evaluation with online signals such as user ratings, task completion and escalation rates.
Loading…