RAGAS(Retrieval Augmented Generation Assessment)

RAGAS create infra to evalute RAG pipeline because output from RAG is inputted to RAGAS pipeline

RAG Pipeline RAGAS Evaluation
Check detials of each stage on RAG Pipeline Code

llm = OpenAI(model="gpt-4o-mini",)

--------- Retrieval Phase ----
langchain_embeddings = LangchainOpenAIEmbeddings(
    model="text-embedding-3-small",
    api_key=api_key,
    base_url=base_url,
)
embeddings = LangchainEmbeddingsWrapper(langchain_embeddings)

---------Augumentation Phase---------
query_engine = index.as_query_engine(llm=llm, similarity_top_k=top_k)

--------Retrieval Phase-------------
//user_question = "Show firewall policies blocking outbound traffic"
rag_response = query_engine.query(user_question)
RAGAS for testing RAG

// 1. Create LLM Client for ragas
ragas_llm_client = OpenAI(api_key=api_key, base_url=base_url)

// 2. Create embeddings
ragas_langchain_embeddings = LangchainOpenAIEmbeddings(..)

// 3. Wrap LLM client into llm factory
llm = llm_factory("gpt-4o-mini", client=ragas_llm_client)

// 4. Configuration for RAGAS
// max_workers: how many concurrent threads are used for parallel tasks
// max_retries: how many times the system should attempt to call the API
// max_wait: maximum time the system should wait between retries or for an API response before marking the task as failed.
run_config = RunConfig(max_workers=max_workers, max_retries=5, max_wait=120)

// 5. Evaluate RAG pipeline
from ragas import evaluate
ragas_result = evaluate(
  dataset = rag_response,
  metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
  llm=ragas_llm,
  embeddings=ragas_embeddings,
  run_config=ragas_run_config,
  raise_exceptions=False,
)

************ ragas_result *************
1,          //id
What destination IP was blocked when source 10.1.1.5 attempted outbound traffic at 2025-05-01 08:02:11?,			//question
8.8.8.8. The firewall log shows: 2025-05-01 08:02:11 FIREWALL_DENY src=10.1.1.5 dst=8.8.8.8 policy=OUTBOUND_BLOCK.,	//ground_truth
The destination IP that was blocked when source 10.1.1.5 attempted outbound traffic at the specified time was 8.8.8.8.,//answer
easy,								//difficulty
single-event lookup,				//category
1.0,								//faithfulness
,									//answer_relevancy
0.9999999999,						//context_precision
1.0									//context_recall