Evaluate retrieval and generation separately so you know which stage fails.
Retrieval: recall@k, precision@k, MRR or NDCG against labeled relevant chunks; context precision and context recall.
Generation: faithfulness (is the answer supported by the retrieved context), answer relevance, and correctness against reference answers; tools like RAGAS automate these with LLM judges.
Build a golden set from real questions, add LLM-as-judge with a clear rubric calibrated against human review, and track regressions on every change; add online signals such as thumbs up/down and escalations.
Loading…