Overview
The article introduces Surus, a Netflix-OSS project, and its first user-defined function (UDF), ScorePMML, which allows for efficient scoring of predictive models in Apache Pig using the Predictive Modeling Markup Language (PMML). The article discusses the benefits of standardizing model representation and the practical implementation of scoring models at scale.
What You'll Learn
1
How to use ScorePMML for scoring predictive models in Apache Pig
2
Why adopting PMML can streamline the model deployment process
3
When to implement ScorePMML for large datasets
Prerequisites & Requirements
- Understanding of predictive modeling concepts
- Familiarity with Apache Pig and PMML
Key Questions Answered
What is ScorePMML and how does it work?
ScorePMML is a user-defined function (UDF) developed by Netflix that allows for efficient scoring of predictive models represented in PMML format within Apache Pig. It standardizes the model scoring process, reducing overhead and enabling consistent representation across different stages of model deployment.
How can ScorePMML improve model deployment at Netflix?
By adopting PMML as a standard representation for predictive models, ScorePMML helps streamline the model deployment process, allowing data scientists to iterate and deploy models more effectively, thus saving time and reducing the risk associated with custom code.
What are the limitations of ScorePMML?
ScorePMML is built on jPMML 1.0.19, which does not fully support the 4.2 PMML specification. This limitation affects the handling of certain missing value strategies, which can complicate the implementation of models like Gradient Boosting Machines (GBMs).
Technologies & Tools
Backend
Apache Pig
Used for processing and scoring predictive models with ScorePMML.
Data Format
Pmml
Standard format for representing predictive models used in ScorePMML.
Key Actionable Insights
1Implement ScorePMML to standardize model scoring across your data pipeline.Standardizing model representation with PMML reduces the complexity of custom scoring solutions and enhances collaboration among data scientists, making the deployment process more efficient.
2Utilize the error handling features of ScorePMML when scoring large datasets.By incorporating error indicators in the output tuples, you can avoid job failures due to individual record errors, ensuring smoother processing of large datasets.
3Consider the limitations of PMML when designing your predictive models.Understanding the constraints of the PMML version supported by ScorePMML can help you avoid potential issues with model implementation, especially for complex models like GBMs.
Common Pitfalls
1
Failing to match data types and column names between Pig and PMML can lead to exceptions.
This mismatch can cause delays as errors will only be caught when the Hadoop MapReduce job starts. It’s crucial to ensure compatibility before executing the job.
2
Using non-compliant column names in PMML can result in errors during scoring.
Since Hive has strict naming conventions, any non-alphanumeric characters in column names will lead to failures. Always adhere to Hive's naming rules when preparing PMML files.
Related Concepts
Predictive Modeling
User Defined Functions (udfs)
Data Processing With Apache Pig
Open Standards In Data Science