How Ramp used Retrieval-Augmented Generation (RAG) to build a state-of-the-art in-house industry classification model.
Overview
The article discusses how Ramp transitioned from a homegrown industry classification system to a standardized model using Retrieval-Augmented Generation (RAG) to improve data quality and operational efficiency. It highlights the challenges of industry classification and the benefits of adopting NAICS codes for better accuracy and control.
What You'll Learn
1
How to migrate from a homegrown classification system to a standardized industry classification using NAICS codes
2
Why using a Retrieval-Augmented Generation (RAG) model can enhance data classification accuracy
3
How to evaluate the performance of a multi-stage classification model using specific metrics
Prerequisites & Requirements
- Understanding of industry classification systems and their importance
- Familiarity with machine learning concepts, especially in classification tasks(optional)
Key Questions Answered
What were the challenges Ramp faced with their old industry classification system?
Ramp's old system suffered from inaccuracies, generic categories, and a lack of auditability. This led to difficulties in understanding customer needs and compliance requirements, as businesses were often misclassified or categorized too broadly.
How does Ramp's new system using NAICS codes improve classification?
The new system allows for a consistent and expressive taxonomy, enabling better communication with external partners. It also provides flexibility in categorization, allowing teams to choose the appropriate level of detail for their needs, thus enhancing data quality and operational efficiency.
What are the key stages in building a classification model with RAG?
The key stages include calculating text embeddings, generating recommendations based on similarity scores, and using a large language model (LLM) to make final predictions. This structured approach helps constrain outputs to relevant classifications, improving accuracy.
What metrics were used to evaluate the performance of Ramp's classification model?
The primary metrics included accuracy at k (acc@k) for the recommendation stage and a custom fuzzy-accuracy metric for the final prediction stage. These metrics ensure that the model's performance aligns with the overall classification goals.
Key Statistics & Figures
Performance improvement in acc@k
up to 60%
This improvement was observed as a result of optimizations in the recommendation generation stage.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Machine Learning
Retrieval-augmented Generation
Used as the classification model to improve accuracy in industry classification.
Classification System
Naics
Serves as the standardized taxonomy for classifying industries.
Database
Clickhouse
Used for fast retrieval of recommendations based on similarity scores.
Data Streaming
Kafka
Utilized for logging intermediate results to diagnose issues and iterate on prompts.
Key Actionable Insights
1Migrating to a standardized classification system like NAICS can significantly enhance data quality and operational efficiency.This transition allows teams to have a unified understanding of customer industries, which is crucial for effective targeting and compliance.
2Implementing a RAG model can provide better control over classification accuracy and adaptability to changing business needs.With the ability to adjust hyperparameters and audit decisions, organizations can respond more effectively to data quality issues.
3Utilizing hierarchical classification systems enables teams to extract broader categories from specific codes, enhancing flexibility in data analysis.This approach allows for tailored insights based on varying levels of detail, which can be crucial for different business contexts.
Common Pitfalls
1
Relying on a homegrown classification system can lead to inconsistencies and inaccuracies in data.
This often occurs due to the lack of a standardized approach, making it difficult to evaluate and compare classifications across different teams.
2
Neglecting the importance of auditing and interpretability in machine learning models can result in trust issues.
Without the ability to understand model decisions, stakeholders may be hesitant to rely on automated classifications, impacting overall adoption.
Related Concepts
Industry Classification Systems
Machine Learning In Data Classification
Hierarchical Classification Methods
Data Quality Improvement Strategies