Anthropic’s New Approach to RAG

October 2, 2024in Data

Anthropic’s New Approach to RAG: Enhancing Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) has emerged as a promising solution to the limitations of fine-tuning Large Language Models (LLMs). Anthropic’s new contextual RAG approach enhances the precision and reliability of AI-driven systems, especially in domain-specific applications, by addressing key challenges in retrieval and generation.

Thank you for reading this post, don't forget to subscribe!

Understanding LLMs and Their Challenges

Large Language Models are powerful tools capable of general knowledge tasks, such as writing code or answering complex queries. However, their generalist nature often results in underperformance in specialized domains, necessitating fine-tuning or alternative approaches like RAG.

Why not just fine-tune?

Cost: Fine-tuning requires significant investment in cloud GPU resources or proprietary APIs.
Data Sensitivity: Organizations must carefully manage data privacy and attribution.
Complexity: Effective fine-tuning demands high-quality, task-specific data and a significant engineering effort.

RAG: A Practical Alternative

RAG systems address these challenges by connecting LLMs directly to an organization’s knowledge base. Instead of retraining the model, RAG retrieves relevant information dynamically, combining it with the model’s generative capabilities to deliver tailored responses.

How RAG Works

Knowledge Base Creation:
- Document Chunking: Break large documents into smaller sections.
- Embedding Computation: Represent chunks as numerical embeddings that capture their semantic meaning.
- Vector Store: Store these embeddings in a database for efficient retrieval.
Response Generation:
- Query Processing: Compute an embedding for the user’s query.
- Retrieval: Use the query embedding to find the most relevant chunks.
- LLM Integration: Combine retrieved chunks with the query and pass them to the LLM.
- Response Creation: Generate an answer based on the context provided by the retrieved chunks.

Addressing RAG Limitations with Contextual Retrieval

A standard RAG system can struggle when retrieved chunks lack sufficient context to answer a query. For example:

Query: What were the long-term effects of Drug X in the 2023 clinical trial?
Retrieved Chunk: Participants showed significant improvements after treatment.

This chunk fails to clarify whether the improvement was from Drug X, whether it was part of the 2023 trial, or if it reflects long-term effects.

Anthropic’s Contextual Retrieval Solution

To address this, Anthropic’s approach includes:

Contextual Embeddings: Each document chunk is enriched with a succinct context generated by an LLM to situate the chunk within the overall document.
Enhanced Indexing: Both the enriched chunks and their embeddings are stored, and the BM25 keyword-based index is updated to capture term importance (TF-IDF).

Performance Improvements:

Contextual embeddings reduced top-20-chunk retrieval failure rates by 35% (from 5.7% to 3.7%).
Combining contextual embeddings with BM25 indexing reduced failures by 49% (to 2.9%).

Improving Retrieval Accuracy with Hybrid Methods

To enhance RAG systems further, Anthropic integrates traditional keyword-based methods with semantic search:

BM25 Integration: Ideal for exact matches like error codes or product numbers (e.g., “Error XYZ-123”).
Hybrid Search: Combines BM25’s precision with semantic search’s broader understanding for robust results.

Reranking for Higher Accuracy

While retriever models excel at efficiently extracting relevant chunks, their reliance on simple similarity measures (e.g., cosine similarity) can lead to suboptimal results.

Rerankers

Perform cross-attention between user queries and chunks to uncover deeper relationships.
Rerank smaller selections identified by retrievers, improving the quality of final outputs.

Key Benefits:

Enhanced alignment between query intent and retrieved content.
Reduced failure rates in complex retrieval scenarios.

RAG in Action: A Summary

Break down documents into smaller, manageable pieces.
Add contextual embeddings to enrich each chunk.
Use hybrid retrieval methods to combine semantic and keyword-based search.
Employ rerankers to fine-tune the final selection of chunks for optimal accuracy.

Anthropic’s advanced RAG methodology demonstrates how AI can overcome traditional challenges, delivering more precise, context-aware responses while maintaining efficiency and scalability.