•Huaixiu Zheng, Guoqin Zheng, Naveen Somasundaram, Basab Maulik, Hugh Williams, Jeremy Hermann•15 min read•advanced•
--
•View OriginalOverview
The article discusses the scaling of Uber's Customer Support Ticket Assistant (COTA) system using deep learning techniques. It highlights the improvements made from COTA v1 to COTA v2, including enhanced prediction accuracy and reduced ticket handling times, achieved through the integration of deep learning models into Uber's machine learning platform.
What You'll Learn
1
How to leverage deep learning to improve customer support ticket resolution
2
Why integrating Spark with deep learning can enhance model performance
3
How to implement a model lifecycle management pipeline for continuous improvement
Prerequisites & Requirements
- Understanding of machine learning and natural language processing concepts
- Familiarity with Spark and TensorFlow(optional)
Key Questions Answered
What improvements were made from COTA v1 to COTA v2?
COTA v2 improved the top-1 prediction accuracy by 16 percent for the Contact Type model and 8 percent for the Reply model compared to COTA v1. Additionally, it reduced ticket handling times and maintained or improved customer satisfaction levels.
How does Uber ensure the freshness of its models?
Uber built a model lifecycle management Pipeline (MLMP) that integrates an internal job scheduling tool to retrain and redeploy models at fixed intervals, ensuring that model performance remains optimal over time.
What challenges did Uber face when deploying COTA v2?
Uber faced challenges in integrating NLP transformations with deep learning training, particularly in utilizing both Spark for data processing and GPUs for model training. They addressed this by creating a deep learning Spark Pipeline (DLSP) to streamline the process.
What were the results of the online tests comparing COTA v1 and COTA v2?
The A/B tests showed a statistically significant improvement in model performance and a 6.6 percent reduction in average handle time per ticket for the treatment group using COTA v2 compared to COTA v1.
Key Statistics & Figures
Top-1 prediction accuracy improvement for Contact Type model
16 percent
Improvement from 49 percent to 65 percent accuracy
Top-1 prediction accuracy improvement for Reply model
8 percent
Improvement from 47 percent to 55 percent accuracy
Reduction in average handle time per ticket
6.6 percent
Observed during A/B testing of COTA v2
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Spark
Used for data preprocessing and building the deep learning pipeline
Backend
Tensorflow
Utilized for training deep learning models
Key Actionable Insights
1Integrating deep learning models into existing customer support systems can significantly enhance ticket resolution accuracy.By leveraging deep learning, Uber was able to improve the prediction accuracy for ticket responses, which directly impacts customer satisfaction and operational efficiency.
2Implementing a model lifecycle management pipeline is crucial for maintaining model performance over time.As business dynamics change, models can become outdated quickly. Regular retraining and deployment ensure that the models remain effective in providing accurate predictions.
3Utilizing Spark for data preprocessing while leveraging GPUs for deep learning training optimizes resource usage.This hybrid approach allows for faster data processing and efficient model training, which is essential for handling large volumes of customer support tickets.
Common Pitfalls
1
Overcomplicating the model retraining process can lead to reduced maintenance and model performance.
If the retraining process is too complex, teams may avoid regular updates, leading to outdated models that do not perform well in changing environments.
2
Neglecting to monitor model performance over time can result in degraded accuracy.
Models can become less effective as data patterns change. Regular monitoring and retraining are essential to maintain high performance.
Related Concepts
Machine Learning
Natural Language Processing
Model Lifecycle Management
Deep Learning