What is RAGAS in RAG evaluation

 RAGA, in the context of RAG (Retrieval-Augmented Generation) evaluation, specifically refers to RAG Assessment with Augmentation. It's not a separate engine but an  extension of the popular RAGAS (RAG Assessment System) framework.

Here's a breakdown of RAGA and its role in RAG evaluation:

Traditional RAGAS:

RAGAS is an open-source framework designed to evaluate RAG pipelines. It offers various metrics to assess both the retrieval and generation components of a RAG system.

These metrics include:

Retrieval metrics (context_relevancy, context_recall) to measure how well the retrieval component finds relevant information.

Generative metrics (faithfulness, answer_relevancy) to evaluate how well the generation component utilizes retrieved information and produces accurate and relevant answers.

RAGA (RAG Assessment with Augmentation):

RAGA builds upon RAGAS by introducing an additional layer of evaluation focused on the model's ability to leverage retrieved information.

It injects carefully crafted augmentations into the retrieved context before feeding it to the generation stage.

These augmentations can be:

Fact deletions: Removing specific facts from the retrieved context to see if the model can still generate accurate answers.

Fact replacements: Replacing factual elements with incorrect information to assess the model's reliance on retrieved information and its ability to identify inconsistencies.

Noise additions: Adding irrelevant or misleading information to the context to evaluate the model's robustness to noise and its capacity to focus on the essential retrieved elements.

Benefits of Using RAGA:

Deeper Insights: RAGA provides a more comprehensive evaluation by testing the model's dependence on retrieved information and its ability to handle different types of noise or inconsistencies.

Improved Generalizability: By analyzing how the model performs under augmented contexts, RAGA helps identify potential weaknesses and areas for improvement, leading to a more robust and generalizable RAG pipeline.

Who should use RAGA?

While RAGAS offers valuable core evaluation functionalities, RAGA is particularly beneficial for those who want to go beyond basic metrics and gain a deeper understanding of how well their RAG model leverages retrieved information for generation. It's suitable for developers and researchers working on advanced RAG applications where factual correctness and robustness are crucial.

In Conclusion:

RAGA (RAG Assessment with Augmentation) is a valuable extension to the RAGAS framework. By incorporating context augmentation techniques, it provides a more rigorous evaluation of RAG pipelines, helping developers build more reliable and informative RAG systems.

references:

Gemini 

https://docs.ragas.io/en/stable/getstarted/install.html