Ragas' context_recall metrics use attribution analysis, which requires developers to prepare annotated standard answers (ground truth) in advance. During evaluation, the system compares the standard answer sentence by sentence with the retrieval context, categorizes it as [Attributed] or [Not Attributed], and finally takes the attribution ratio as the score. For example, in the case of Einstein, birth information and Nobel Prize were correctly attributed, while the number of papers and moving information were labeled as Not Attributed.
This approach requires manual annotation costs but accurately assesses the information coverage capability of the retrieval system. Its unique value lies in the ability to identify 'correct but unsourced' generated content, which is particularly important for rigorous scenarios such as academic and medical. In terms of implementation, the metric relies on the fine-grained semantic understanding capabilities of LLM, and the evaluation process includes detailed attribution reasoning instructions.
Used by developers to ensure annotation quality, this metric is particularly suitable for evaluating RAG systems in specialized fields where high recall is required, and is the gold standard for validating the comprehensiveness of searches.
This answer comes from the articleRagas: assessing RAG recall QA accuracy and answer correlationThe
