Collect real or realistic inputs (logs, support tickets, domain experts' questions) covering common cases, edge cases, ambiguous inputs and adversarial prompts.
Define success criteria per case: a reference answer, a checklist or rubric, required sources, or a programmatic assertion.
Start small (50 to 200 high-quality cases), label carefully with domain experts, version the dataset, and grow it with failures found in production.
Run it automatically on every prompt, model or retrieval change and block releases that regress key metrics.
Loading…