RAG / Retrieval-Augmented Generation
Generally, RAG is a
LLM(Large Language Model)
which can fetch data from external source(eg:
vector database, SQL Db, Graph DB, Web Search engines) & feed to AI generation
process.
Purpose of RAG? To give more Context to LLM models to predict
better
What is vector?
What is Embedding Model?
RAG Pipeline
RAG pipeline on firewall,vpn logs:
We have log files(eg: VPN, firewall).
RAG pipeline will read log files and provide answers to
Administrator questions.
|
1. The Retrieval Phase: Chunks: Raw documents are broken down into chunks. Chunk = actual log + meta data Embedding: Find embedding of content of every chunk Store Embedding: Store {embedding, actual_log, meta_data} into vector database(pinecone) or list in RAM 2. The Augmentation Phase: Find embedding for user query Find closeness of user's query embedding with stored logs embeddings using Semantic Search Cosine similarity or euclidian distance Take top k closest log files to user's query embeddings 3. The Generation Phase Using embedding, Get actual_log from vector database(stored in Retrieval Phase) Join actual user query, top k files retrieved using embeddings as string, because to LLM embeddings are of no meaning. This is combined prompt This combined prompt (the user's query + Top k matching logs) is fed into the LLM. This forces the model to synthesize an answer based only on the provided external data, which drastically reduces hallucinations |
Evaluating RAG Pipeline
Evaluating a RAG pipeline requires measuring
1. How well Retriever fetches relevant logs (Recall/Precision) and
2. How accurately Generator synthesizes the answer (Faithfulness/Answer
Relevance).
1. Precision
Measures how many returned documents are relevant
Precision = (Relevant documents retrieved)/(Total number of documents retrieved)
if total documents retrieved = 10
Relevant Documents = 5
So Precision = 5/10 = 50%2. Recall
How many relevant documents are returned
Recall = (Relevant documents retrieved)/(Total relevant documents)
Total Relevant documents = 5
Retrieved = 2. Recall = 2/5 = 40%Chunking
Breaking the longer text document into smaller chunks
Reasons of Chunking:
1. Token Limit: Many embedding models have limit on amount of text they
can convert to vectors
2. Improve search relevancy
3. Ensures we are only sending relevant text from document to the LLM.
Problem without Chunking
Let's suppose we have 1000 pdfs(ie books) and we are creating a vector
embedding for each book.
Issue is we are compressing entire book into 1 vector, These vectors
will not represent contents of any chapter/keyword/page precisely. These
are just generic vectors
HENCE we want to take a book and chunk into smaller pages or paragraph
or Sentence. Vector DB will have 1 million paragraphs instead of 1000
books.
What chunk size should be taken?
- Too Big: if chunks are big as 4-5 chapters then it does not
make sense to create chunks.
- Too Small: Chuking at word level. vectors will loose context of
surrounding words/paragraphs/sentences which reduces search relevance
Chunking Techniques
1. Fixed Size
We choose fixed size and we chunk the document at that fixed size
words.
Issue in this approach is, chunk boundary can fall on middle of word,
middle of sentence etc, which again looses the nearness context.
1-250(chunk-1), 250-500(chunk-2), 500-750(chunk-3)2. Overlapping Chunking
Chunk-1(paragraph 1), Chunk-2(some part of paragraph-1 + paragraph-2 + Some part of paragraph-3)
3. Recursive Character Splitting
Splitting text into chunks at specified characters, eg: Newline
characters. This gives variable chunk size, ie we can have very large or
very small chunk sizes depending on newline character location.
4. Based on Type of Document
- HTML Document can be splitted using HTML tag types. h1, h2 all in
seperate chunks
- Code(eg: Python) Seperate functions can be splitted into seperate
chunks
5. Semantic Chunking
In a paragraph there can be 4-5 sentences.
In this type of chunking, we find similarity/nearness/closeness of
meaning of 1 sentence to next sentence in the same paragraph. if they
are similar they will be placed in same chunk
How? embedding is found for every sentence in the paragraph and
if (embedding of sentence-1 - embdedding of sentence-2) is below a
certain threshold they have similar meaning and are put in same
chunk.
Advantages:
1. Variable sized chunks that follow train of thought of author
Disadvantages
1. Can be expensive as we are calculating embeddings of every sentence
in paragraph
2. Repeated embeddings calculations.
6. Language Based Chunking
We provide document to LLM to create chunks. Along with documents additional instructions are provided to LLM to create chunks(eg: Keep concepts together, add breaks when new topic starts)
7. Context Aware Chunking
Add context(eg: summary) to every chunk.
User Query Rewriting
Production RAG systems does not provide direct query asked by user to
the RAG system.
RAG systems are written for interacting with humans in natural language,
but human writeen prompts are BAD search queries
Rather than feeding those prompts directly to the system, retriever can
parse the prompt to identify its intent, EDIT/REWRITE/OPTIMIZE the
prompt for retrieval
Query Parsing Techniques
1. Query Rewriting
LLM rewrites the query before its submitted to the retriever
We will setup a seperate LLM as query rewriter
Example:
User's Prompt: // Example of Messy Prompt
I was out walking my dog, a beautiful black pammarian dog named Ranu, when she raced away from me and yanked on her
leash while I was holding it.
3 days later my shoulder is still numb and my finders are all pins and needles. What's going on?
|--------------------| |-----------|
Messy Prompt -> | Query Rewriter LLM | -> Optimized Prompt -> | Retriever |
|--------------------| |-----------|
Optimized Prompt:
Experienced a sudden, forceful pull on the shoulder, resulting in persistent shoulder numbness and finder numbness
for 3 days. What are potential causes or diagnosis, such as neuropathy or nerve impingement?2. Named Entity Recognization
This is used for recognizing categories of information in the query. Eg:
Places, Characters, Dates, people etc.
This retrieved information can be provided to retrieval to get matching
documents
User's Query:
I read the Great Gatsby by F. Scott Fitzgerald last summer while visting New York.
Are there any similar books set in the 1920s that I might enjoy?
Entities: Person, books, location, dates, actors, characters
Entities
\/
|------------|
Messy Prompt -> | GLINER LLM | -> Labelled Prompt
|------------|
Labelled Prompt:
I read (The Great Gatsbay BOOK) by (F Scott Fitzgerald PERSON) last summer while visiting (New York LOCATION).
Are there any similar books set in the (1920s DATE) that I might enjoy?
3. Hypothetical Document Embeddings (HyDeE)
This generates a hypothetical document from User's prompt and then that hypothetical document is compared against the documents by retriever and which is used to provide response to the user
User's Prompt:
|----------|
User Prompt -> | HyDE LLM | -> Hypothetical Document -> |Embedding LLM| -> [0.1, 0.94, 0.45]
|----------| embdedding of document