Our journey building a message classification model for Shopify Inbox with the data-centric approach, from defining our classification taxonomy to training our annotators on labeling.
Overview
The article discusses the development of a message classification model for Shopify Inbox using a data-centric machine learning approach. It highlights the importance of improving training data quality to enhance model performance and provides insights into the methodologies and tools used throughout the process.
What You'll Learn
1
How to apply a data-centric approach to improve machine learning model accuracy
2
Why defining a clear taxonomy is essential for effective message classification
3
How to use semi-supervised techniques for building training datasets
Prerequisites & Requirements
- Understanding of natural language processing concepts
- Familiarity with TensorFlow and pre-trained models(optional)
Key Questions Answered
What is a data-centric approach in machine learning?
A data-centric approach focuses on improving the quality of training data to enhance model performance, rather than solely experimenting with different model architectures. This method aims to reduce inconsistencies in the dataset, which is crucial for achieving better results across various models.
What were the results of the message classification model's performance?
The final model achieved approximately 90% accuracy and 80% high confidence coverage, significantly improving from the initial prototype, which had around 70% accuracy and 35% coverage. This demonstrates the effectiveness of the refined training data and taxonomy.
How did the team handle sensitive information in buyer messages?
The team identified messages containing personal information (PI) and replaced it with mock data using the Presidio library. This ensured that sensitive data was anonymized while maintaining the integrity of the training dataset.
What tools were used for dimensionality reduction and clustering?
The team utilized UMAP for dimensionality reduction and HDBScan for clustering to identify groups of similar messages. This facilitated the rapid construction of a training dataset by finding related messages based on semantic similarity.
Key Statistics & Figures
Model accuracy
~90%
Achieved in the final version of the message classification model in production.
High confidence coverage
~80%
Indicates the percentage of messages classified with high confidence in the final model.
Size of training set
20,000
The number of messages used in the training dataset for the final model.
Taxonomy classes
45
The number of distinct topics defined in the new taxonomy for message classification.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Tensorflow
Used for implementing the Universal Sentence Encoder and training the classification model.
Data Processing
Umap
Utilized for dimensionality reduction in the analysis of message clusters.
Data Processing
Hdbscan
Used for clustering similar messages to aid in dataset construction.
Data Processing
Presidio
Employed for identifying and anonymizing personal information in buyer messages.
Data Processing
Faker
Used to generate realistic mock data for replacing personal information.
Backend
Distilbert
Chosen as the pre-trained model for fine-tuning in the message classification task.
Key Actionable Insights
1Investing time in refining your training data can lead to substantial improvements in model performance.The article illustrates how the Shopify team focused on enhancing their dataset quality, which resulted in a significant increase in model accuracy from 70% to 90%. This approach is applicable in any machine learning project.
2Creating a clear and comprehensive taxonomy is crucial for effective data annotation.The development of a new taxonomy at Shopify helped ensure consistent labeling of messages, which directly contributed to improved model performance. This practice can be beneficial in various domains where classification tasks are performed.
3Utilizing semi-supervised learning techniques can accelerate the dataset building process.By combining a small amount of labeled data with a larger set of unlabeled data, the Shopify team efficiently expanded their training dataset. This method can be particularly useful in scenarios where labeled data is scarce.
Common Pitfalls
1
Relying on a model-centric approach can lead to suboptimal performance due to poor data quality.
Many teams focus on tweaking model architectures without addressing the underlying data issues. This can result in models that perform well on training data but fail to generalize effectively.
2
Neglecting to define a clear taxonomy can cause confusion during the annotation process.
Without a well-structured labeling system, annotators may apply inconsistent labels, leading to a poorly trained model. Establishing a clear taxonomy is essential for achieving high-quality annotations.
Related Concepts
Natural Language Processing
Machine Learning Model Evaluation
Data Annotation Techniques
Semi-supervised Learning