v2.0 and beyond
Overview
The article discusses the evolution of Netflix Conductor, a workflow orchestration engine that has gained significant adoption within Netflix for managing core workflows. It highlights recent updates, use cases, and future enhancements, showcasing how Conductor integrates various technologies to optimize workflow management.
What You'll Learn
1
How to utilize Netflix Conductor for orchestrating microservices workflows
2
Why adopting gRPC can enhance performance in workflow orchestration
3
How to implement dynamic workflow executions to reduce overhead
4
When to use Cassandra for scalable data persistence in workflows
Prerequisites & Requirements
- Understanding of workflow orchestration concepts
- Familiarity with gRPC and Cassandra(optional)
Key Questions Answered
How does Netflix Conductor improve workflow orchestration?
Netflix Conductor enhances workflow orchestration by providing a robust framework that supports dynamic execution, scalability with Cassandra, and integration with gRPC for improved performance. It allows for efficient management of over 600 workflows across various teams, facilitating better content delivery and processing.
What are the new features introduced in Netflix Conductor v2.0?
The new features in Netflix Conductor v2.0 include the introduction of a gRPC framework, a Cassandra persistence layer for scalable data handling, external payload storage to offload data persistence, and dynamic workflow execution capabilities that streamline the workflow management process.
What use cases does Netflix Conductor address?
Netflix Conductor addresses several use cases including content ingest and delivery, content quality control, localization, encodes and deployments, and marketing technology. This versatility allows it to support various business flows effectively within Netflix.
How is Conductor deployed and maintained at Netflix?
Conductor is deployed using a wrapper that integrates the latest version of the open-source software with Netflix's infrastructure components. This approach ensures that updates are tested and functional before deployment, maintaining a robust orchestration engine without internal forks.
Key Statistics & Figures
Number of workflow definitions orchestrated
600+
Conductor orchestrates over 600 workflow definitions owned by more than 50 teams across Netflix.
Cluster size
12–18 instances of AWS EC2 m4.4xlarge
The Conductor cluster typically runs at around 30% capacity.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Database
Cassandra
Used for persisting execution data and enabling horizontal scaling.
Communication Framework
Grpc
Introduced as an alternative to REST for improved performance.
Search Engine
Elasticsearch
Used as a secondary datastore and indexer.
Database
Dynomite
Used for persisting metadata.
Queuing System
Dynoqueues
Built on top of Dynomite for queuing tasks.
Monitoring
Spectator
Used for metrics collection.
Monitoring
Atlas
Used alongside Spectator for metrics.
Key Actionable Insights
1Leverage the gRPC framework in Conductor to enhance communication efficiency between services.Using gRPC can significantly reduce latency and improve the performance of your microservices architecture, especially in high-throughput environments.
2Implement dynamic workflow executions to simplify the management of ad hoc workflows.This feature allows for the creation and execution of workflows without the need for prior registration, which can streamline testing and analytical tasks.
3Utilize Cassandra for data persistence to ensure scalability in handling large volumes of workflow executions.Cassandra's ability to scale horizontally makes it an ideal choice for applications requiring high throughput and availability.
Common Pitfalls
1
Failing to validate task definitions before registering workflows can lead to runtime errors.
Without proper validation, users may encounter issues during workflow execution due to incorrect task configurations, which can disrupt the entire workflow.
Related Concepts
Workflow Orchestration
Microservices Architecture
Scalability In Distributed Systems
Grpc Vs REST