Visit the post for more.
Overview
The article discusses the engineering challenges and solutions involved in building a posts search feature at Facebook, which allows users to search through over one trillion posts. It covers data collection, indexing, updating, serving, and ranking results, highlighting the complexity and scale of the system.
What You'll Learn
1
How to manage large-scale data indexing using HBase and Hadoop
2
Why caching strategies are crucial for performance in search systems
3
How to implement real-time updates to search indexes using pub/sub systems
Prerequisites & Requirements
- Understanding of data indexing and search algorithms
- Familiarity with HBase and Hadoop ecosystems(optional)
Key Questions Answered
How does Facebook handle the indexing of over one trillion posts?
Facebook uses an HBase cluster to store harvested data and executes Hadoop map-reduce jobs to build a search index. This process involves separating document data for ranking and creating an inverted index for search queries, ensuring efficient data retrieval and ranking.
What challenges does Facebook face with its posts search feature?
The main challenges include managing a rapidly changing data schema, ensuring real-time updates without overloading MySQL databases, and serving a large index efficiently. The complexity arises from the need to sort and index various types of posts differently.
What techniques are used for ranking search results in Facebook's posts search?
Facebook employs query rewriting and dynamic result scoring to rank search results. This involves adding optional clauses to queries and calculating over a hundred distinct ranking features based on document data to surface the most relevant content.
How does Facebook update its search index in real-time?
Facebook uses a technology called Wormhole to subscribe to changes in MySQL databases. This allows the system to schedule updates for new, modified, or deleted posts, ensuring that the search index reflects the most current data.
Key Statistics & Figures
Total posts indexed
over one trillion
This figure highlights the scale of data that Facebook's search feature must manage.
New posts added daily
one billion
This statistic emphasizes the dynamic nature of the content being indexed and the need for efficient processing.
Size of the posts index
hundreds of terabytes
This indicates the substantial storage requirements for managing the posts index.
Size of the index stored in RAM
over 700 terabytes
This shows the limitations and challenges of using RAM for large datasets.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Hbase
Used to store harvested data for building the search index.
Data Processing
Hadoop
Utilized for executing map-reduce jobs to build the search index.
Database
Mysql
Serves as the production database from which data is harvested.
Data Streaming
Wormhole
A pub/sub system used for subscribing to changes in MySQL databases for real-time updates.
Search Infrastructure
Unicorn
The search infrastructure that works with the built index.
Key Actionable Insights
1Implementing a robust data indexing strategy is crucial for handling large datasets effectively.As seen in Facebook's approach, using technologies like HBase and Hadoop can facilitate the management of vast amounts of data while ensuring quick access and updates.
2Utilizing caching mechanisms can significantly improve the performance of search systems.Facebook's decision to cache frequently accessed data structures demonstrates the importance of optimizing data retrieval to enhance user experience.
3Real-time data updates are essential for maintaining the accuracy of search results.The use of pub/sub systems like Wormhole for updates allows Facebook to keep its search index current, which is vital for user satisfaction.
Common Pitfalls
1
Failing to account for the varying data schemas can lead to inefficiencies in indexing.
As Facebook experienced, different types of posts store information differently, complicating the sorting and indexing process. This can be mitigated by establishing a more uniform data schema.
2
Overloading production databases during data harvesting can degrade performance.
The article highlights the need for careful monitoring when harvesting data from MySQL databases, as they also serve production traffic. Implementing throttling or scheduling data collection during off-peak hours can help alleviate this issue.
Related Concepts
Data Indexing
Search Algorithms
Real-time Data Processing
Caching Strategies