by Varun Sekhri, Meenakshi Jindal
Overview
The article discusses Marken, a scalable annotation service developed by Netflix to allow various microservices to annotate their entities with metadata. It covers the goals, architecture, schema design, and search capabilities of the service, highlighting its importance in managing annotations for a vast array of media content.
What You'll Learn
1
How to create a scalable annotation service for microservices
2
Why versioning annotations is crucial for data integrity
3
How to implement spatial and temporal data storage in annotations
4
When to use Elasticsearch for efficient search capabilities
Prerequisites & Requirements
- Understanding of microservices architecture
- Familiarity with Elasticsearch and Cassandra(optional)
- Experience with JSON schema design
Key Questions Answered
What are the main goals of the Marken annotation service?
The main goals for the Marken annotation service include allowing any entity to be annotated, supporting versioning of annotations, ensuring low latency for CRUD and search operations, and making all data available for offline analytics in Hive/Iceberg.
How does Marken handle spatial and temporal data in annotations?
Marken uses Well Known Text (WKT) format to represent spatial data and allows temporal data to be stored as frame numbers or nanoseconds, ensuring that annotations can be accurately associated with specific locations and times in video content.
What technologies are used in the architecture of Marken?
Marken's architecture utilizes Cassandra as the primary database for storing annotations and Elasticsearch for supporting various search requirements, ensuring scalability and low latency for real-time applications.
What challenges does Marken face in scaling its annotation service?
Marken faces challenges in scaling due to the increasing volume of annotations and the need for efficient data ingestion from machine learning pipelines. The service has scaled from a 12-node to a 24-node Cassandra cluster to accommodate this growth.
Key Statistics & Figures
Total number of annotations
1.9 billion
This number reflects the scale of the annotations managed by the Marken service, indicating its capability to handle extensive data.
Data size of annotations
2.6 TB
The total data size shows the substantial storage requirements for the annotations processed by the service.
Nodes in Cassandra cluster
24 nodes
The cluster was scaled from 12 to 24 nodes to support the growing data size and query demands.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Cassandra
Used as the primary database for storing annotations.
Search Engine
Elasticsearch
Used to support various search requirements for annotations.
Key Actionable Insights
1Implement versioning for your annotations to maintain data integrity and track changes over time.Versioning allows teams to manage updates to annotations without losing historical data, which is essential for applications that rely on accurate metadata.
2Utilize Elasticsearch for implementing advanced search capabilities in your annotation service.Elasticsearch supports full-text search, stem search, and semantic search, which can significantly enhance the user experience by providing relevant results based on user queries.
3Consider using Well Known Text (WKT) for representing spatial data in your applications.WKT provides a standardized way to describe geometric shapes, making it easier to integrate spatial data into your annotation models.
4Plan for scalability from the outset by choosing databases that support horizontal scaling.Cassandra and Elasticsearch are designed to handle large volumes of data and high query loads, which is crucial for services like Marken that expect significant growth.
Common Pitfalls
1
Failing to implement proper versioning for annotations can lead to data integrity issues.
Without versioning, updates to annotations may overwrite previous data, making it difficult to track changes and maintain accurate metadata.
2
Neglecting to optimize search indices can result in high latency for query responses.
If search indices are not balanced, it can create hotspots that slow down query performance, especially during high data ingestion periods.
Related Concepts
Microservices Architecture
Annotation Data Management
Machine Learning Integration In Media Applications