Retrieval-Augmented Generation (RAG)

What is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation (RAG) gives a generative AI model outside source text during answer creation. NIST defines retrieval-augmented generation as a GenAI model paired with a separate information retrieval system or knowledge base.

Retrieval finds source text. Augmentation places source text near the user question. Generation writes the answer from model output and supplied context.

RAG does not create a newly trained model for each answer. RAG changes answer context at request time. Stored source text guides the answer more directly than model memory alone.

How RAG creates an answer

RAG creates an answer by searching first and writing second. User questions reach a retrieval layer before reaching the language model. Retrieval software selects source passages, then prompt assembly places selected passages beside the question.

Basic answer flow:

1. User submits one question.
2. Retrieval software processes the question.
3. Index search finds related source chunks.
4. Prompt assembly adds selected chunks.
5. LLM writes from supplied context.
6. Interface may show source labels.

AWS explains RAG as a technique that augments an LLM with external data, such as internal documents. Google Cloud describes RAG as a way to connect models with external knowledge bases.

Example:

Input:
“What is the refund period?”

Retrieval action:
Search policy documents for refund rules.

Returned source:
Refund policy passage with time limit.

Answer rule:
Use only the returned policy passage.

Review limit:
Outdated policy text can still produce an outdated answer.

Which components form a RAG system

RAG systems need source material, retrieval structure, prompt assembly, and answer generation. Each component has one job. Weakness in one component can reduce answer quality.

Component	Plain role
Source corpus	Original documents, webpages, tickets, policies, or records
Chunk	Smaller source unit stored for retrieval
Metadata	Labels such as date, topic, author, source, or permission
Embedding	Numeric representation of text meaning
Vector index	Searchable store that compares embeddings
Retriever	Software that finds matching chunks
Reranker	Sorting step that reorders retrieved chunks
Prompt context	Retrieved text sent beside the user question
LLM	Model that writes the answer
Review record	Saved question, retrieved passages, answer, and reviewer notes

AWS describes vector databases as systems that store and query high-dimensional vectors for RAG applications. AWS also describes retrievers that compare query vectors with stored vectors through similarity measures.

How NLP helps RAG find relevant passages

Natural language processing helps RAG match meaning across different wording. Exact keyword search can miss a useful passage when the user question uses different terms. Embeddings help retrieval compare semantic meaning instead of only matching words.

Core NLP terms for RAG:

• Token: small text unit used during processing.
• Embedding: numeric representation of meaning.
• Semantic search: retrieval based on meaning.
• Similarity measure: method for comparing vectors.
• Reranking: second sorting step after first retrieval.
• Query rewriting: clearer version of a weak user query.
• Query decomposition: splitting one complex question into smaller questions.

RQ-RAG research studies query rewriting, query decomposition, and disambiguation for retrieval-augmented generation. Better query handling can improve retrieval before generation starts.

How chunking prepares source text for RAG

Chunking breaks long source material into smaller retrieval units. Good chunks keep one answerable idea together. Poor chunks split a fact from its limit, date, exception, or condition.

Useful chunk checks:

• Keep one complete idea inside one chunk.
• Preserve parent heading context inside stored text.
• Use overlap when facts sit near boundaries.
• Add source, date, topic, and permission metadata.
• Return enough text for answer review.

Microsoft guidance on chunking states that chunking helps RAG and vector search meet model input limits. Microsoft suggests starting with a 512-token chunk and 25% overlap for fixed-size chunking.

Example problem:

Source sentence 1:
“Refunds are available.”

Source sentence 2:
“Refund requests must arrive within 14 days.”

Bad chunking can separate those sentences. Retrieval may return only the first sentence. Model output may then miss the 14-day condition.

How RAG differs from search, fine-tuning, and prompt stuffing

RAG retrieves passages during answer creation. Search returns links or records. Fine-tuning changes model behavior before use. Prompt stuffing inserts large text blocks without a retrieval layer.

Method	Main action	Reader sees
Search	Finds matching records	Links, snippets, or records
RAG	Retrieves passages and generates text	Answer based on selected context
Fine-tuning	Updates model behavior	Changed response pattern
Prompt stuffing	Adds large text into one prompt	Temporary context without retrieval control

NIST notes that RAG can modify knowledge use without retraining. That difference matters because source records can change faster than a model training cycle. Retrieval quality still controls which records reach the answer.

How to measure RAG answer support rate

RAG answer support rate checks whether reviewed answers have source support for every factual claim. This formula is a working review formula, not an official RAG standard.

ARES evaluates RAG systems through context relevance, answer faithfulness, and answer relevance. Answer support rate mainly checks faithfulness because each factual claim must map to retrieved source text.

Where RAG can fail

RAG can fail even when answer text sounds fluent. Failure can start before the model writes. Source quality, chunk structure, retrieval accuracy, permissions, and prompt rules all affect answer trust.

Main failure points:

• Source gap: needed fact does not exist in the corpus.
• Chunk error: retrieved text misses a condition or exception.
• Retrieval mismatch: semantic search finds related but wrong text.
• Metadata loss: retrieved passage lacks date, topic, or source label.
• Permission error: restricted text reaches the wrong user.
• Prompt weakness: model adds claims outside retrieved passages.
• Review gap: no reviewer checks answer claims against source passages.

RAG reduces some knowledge-access problems. RAG does not remove hallucination risk. Source matching and answer review still decide whether output deserves trust.

How reviewers check a RAG answer

RAG review should inspect retrieval before judging answer quality. Reviewers need the question, retrieved passages, final answer, and claim-by-claim support record.

Review process:

1. Save the user question.
2. Save each retrieved passage.
3. Record source, date, and permission.
4. Highlight every factual answer claim.
5. Match each claim to source text.
6. Mark unsupported claims as failures.
7. Label the failure source.
8. Correct corpus, chunks, retrieval, prompt, or review rules.
9. Retest the same question.

A reviewer should reject fluent text when retrieved passages do not support the facts. RAG quality depends on traceable support, not only readable wording.