How we used smart RAG and LLMs to resolve merchant matching problems at Ramp in under 10 seconds.
Overview
The article discusses how Ramp utilizes AI to enhance the accuracy of merchant classification for transactions. It highlights the challenges of manual matching and the implementation of an AI agent that significantly reduces the time taken to resolve incorrect classifications.
What You'll Learn
1
How to implement an AI agent for transaction classification
2
Why guardrails are essential for LLM operations
3
How to use user input to improve merchant classification accuracy
Key Questions Answered
How does Ramp's AI agent improve merchant classification?
Ramp's AI agent uses a large language model (LLM) to quickly process user requests for merchant classification changes, reducing resolution time from hours to under 10 seconds. The agent leverages transaction metadata, user-provided information, and contextual data to make informed decisions, significantly enhancing user experience.
What challenges does Ramp face with merchant classification?
Ramp encounters issues with vague card acceptor names and misleading Merchant Category Codes (MCCs) that complicate accurate merchant classification. These challenges can lead to incorrect classifications, frustrating users who rely on accurate expense tracking.
What are the benefits of using an AI agent for merchant classification?
The AI agent improves customer satisfaction by resolving incorrect merchant information requests almost instantly, allowing Ramp to handle nearly 100% of requests efficiently. This automation reduces operational costs and enhances the accuracy of the merchant database.
How does Ramp evaluate the performance of its AI agent?
Ramp evaluates its AI agent through a combination of manual reviews, follow-up request analysis, and rejection rate monitoring. The agent's performance is further assessed using a second LLM that judges the appropriateness of actions taken based on user requests.
Key Statistics & Figures
Percentage of requests handled by the AI agent
close to 100%
This represents a significant improvement over the previous manual handling, which was only 3% in 2023.
Time taken to resolve requests
under 10 seconds
This is a drastic reduction from the hours previously required for manual resolution.
Rejection rate of requests
1 out of 4
This indicates that the majority of requests are accepted and processed successfully by the AI agent.
Technologies & Tools
Backend
AI/ML
Used to power the AI agent that processes merchant classification requests.
Backend
Large Language Model (llm)
Forms the core of the AI agent, enabling it to understand and process user inputs effectively.
Backend
Online Analytical Processing (olap)
Utilized for rapid querying of transaction data to assist the AI agent in decision-making.
Backend
Multimodal Retrieval Augmented Generation (rag)
Employed to enhance the AI's ability to retrieve relevant merchant data during classification.
Key Actionable Insights
1Implementing an AI agent can drastically reduce the time taken to resolve customer requests.By automating the classification process, Ramp has improved its response time from hours to under 10 seconds, which enhances customer satisfaction and operational efficiency.
2Establishing guardrails for AI operations is crucial to prevent erroneous changes.Guardrails help ensure that the AI agent does not make inappropriate modifications to merchant records, maintaining data integrity and user trust.
3Utilizing user input effectively can enhance the accuracy of transaction classifications.By allowing users to provide additional context for their transactions, Ramp can improve the reliability of its merchant classifications, reducing frustration and errors.
Common Pitfalls
1
Users may provide incomplete or inaccurate information when requesting merchant classification changes.
This can lead to legitimate requests being rejected or misclassified, highlighting the importance of clear user guidance and form design.
2
Relying solely on card acceptor data without additional context can result in incorrect classifications.
The article emphasizes that user context is often necessary to accurately map transactions to merchants, indicating a need for robust data collection methods.