Overview
The article discusses Pinterest's approach to automating human evaluation for content relevance and safety through crowdsourcing. It highlights the implementation of a new system for managing Trending Searches, which enhances user experience by filtering out inappropriate queries while allowing for scalability across multiple languages.
What You'll Learn
1
How to automate human evaluation processes using crowdsourcing techniques
2
Why leveraging machine learning can improve content relevance on platforms
3
When to implement automated workflows for content moderation
Prerequisites & Requirements
- Understanding of machine learning concepts and content moderation
- Familiarity with Python and workflow management systems(optional)
Key Questions Answered
How does Pinterest automate human evaluation for Trending Searches?
Pinterest automates human evaluation by integrating a crowdsourcing platform, CrowdFlower, into its workflow. This allows for human evaluators to assess the appropriateness of trending search queries, which enhances the relevance and safety of content presented to users.
What challenges did Pinterest face with the initial Trending Searches implementation?
The initial implementation faced challenges such as the need for manual review by the Safety team, which was not scalable and could not support internationalization. Additionally, there were issues with inappropriate queries making it to the trending list, necessitating a more automated solution.
What improvements were made to the human evaluation process?
Improvements included the introduction of test questions for evaluators to ensure consistent quality in their assessments. This led to a higher accuracy in identifying inappropriate queries, reducing the need for secondary reviews by the Safety team.
How does Pinterest plan to enhance the Trending Searches feature in the future?
Pinterest aims to enhance Trending Searches by personalizing results based on historical data, integrating calendar data for timely suggestions, and predicting inappropriate queries using past evaluations. The goal is to support over 30 languages for a more global reach.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Crowdsourcing Platform
Crowdflower
Used for automating the human review process of trending search queries.
Programming Language
Python
Utilized for scripting the automation of candidate query submissions and processing.
Key Actionable Insights
1Implementing a crowdsourcing solution like CrowdFlower can significantly enhance the efficiency of content moderation processes.By leveraging crowdsourcing, teams can scale their evaluation efforts without overburdening internal resources, allowing for faster and more accurate content filtering.
2Incorporating test questions for evaluators can improve the reliability of human assessments.This practice ensures that evaluators understand the criteria for evaluation, leading to more consistent and accurate results, which is crucial for maintaining content quality.
3Automating workflows can help manage daily tasks more effectively, reducing manual intervention.By automating the review process, organizations can allocate resources more efficiently and focus on strategic initiatives rather than repetitive tasks.
Common Pitfalls
1
Failing to provide practice questions for evaluators can lead to inconsistent evaluations.
Without practice, evaluators may misinterpret the evaluation criteria, resulting in poor quality assessments that can compromise the integrity of the content moderation process.
Related Concepts
Crowdsourcing In Content Moderation
Machine Learning Applications In User-generated Content
Automation Of Workflows In Software Engineering