•Arnab Chakraborty, Paarth Chothani, Christopher Settles, Adi Raghavendra•13 min read•advanced•
--
•View OriginalOverview
The article discusses the Enhanced Agentic-RAG (EAg-RAG) framework developed by Uber to improve the accuracy and relevance of chatbot responses in the engineering security and privacy domain. It details the challenges faced, the architectural improvements made, and the outcomes achieved in enhancing the performance of the Genie chatbot.
What You'll Learn
1
How to enhance chatbot accuracy through enriched document processing
2
Why integrating LLM-powered agents improves retrieval and answer generation
3
How to automate evaluation of chatbot responses using LLM-as-Judge
Prerequisites & Requirements
- Understanding of LLMs and RAG architectures
- Familiarity with LangChain and LangGraph(optional)
Key Questions Answered
How does the Enhanced Agentic-RAG framework improve chatbot performance?
The Enhanced Agentic-RAG framework improves chatbot performance by integrating LLM-powered agents for pre-retrieval and post-processing steps, which enhances context relevance and accuracy in answer generation. This approach has led to a 27% increase in acceptable answers and a 60% reduction in incorrect advice.
What challenges did Uber face in improving Genie’s response quality?
Uber faced challenges such as high SME involvement leading to slow evaluations and marginal gains in accuracy that plateaued. These issues prompted the adoption of the agentic RAG architecture to facilitate faster iterations and improvements.
What role does the LLM-as-Judge play in the evaluation process?
The LLM-as-Judge automates the evaluation of chatbot responses by scoring them on a 0-5 scale and providing reasoning for its evaluations. This helps in aligning responses with SME quality standards and enhances the reliability of evaluations.
How does the EAg-RAG architecture handle document processing?
The EAg-RAG architecture uses enriched document processing to convert policy documents into structured formats, improving the accuracy of information retrieval and ensuring that the context is preserved during chunking and embedding.
Key Statistics & Figures
Increase in acceptable answers
27%
Achieved through improvements in the EAg-RAG framework.
Reduction in incorrect advice
60%
Resulting from the transition to the agentic RAG approach.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Framework
Langchain
Used for developing the agentic RAG framework.
Framework
Langgraph
Facilitates scalable agentic AI workflows.
Tool
Google Docs
Used for document processing and enrichment.
Key Actionable Insights
1Implementing an agentic RAG approach can significantly enhance the accuracy of chatbot responses.This method allows for better context handling and retrieval accuracy, which is crucial for applications in sensitive domains like security and privacy.
2Utilizing LLMs for document enrichment can improve the quality of data fed into chatbots.By ensuring that the data is well-structured and relevant, organizations can reduce the risk of misinformation and improve user trust in automated systems.
3Automating the evaluation of chatbot responses can streamline the improvement process.This reduces the time spent on manual evaluations and allows for quicker iterations, which is essential in fast-paced environments.
Common Pitfalls
1
Relying solely on traditional RAG architectures can lead to limited improvements in accuracy.
This happens because traditional methods may not adequately address the nuances of domain-specific queries, resulting in irrelevant context being retrieved.
2
Neglecting the importance of document formatting can hinder the effectiveness of data extraction.
When documents are poorly formatted, it complicates the retrieval process and can lead to fragmented information that is difficult for models to interpret.
Related Concepts
Retrieval-augmented Generation (rag)
Large Language Models (llms)
Chatbot Development
Document Processing Techniques