Executing Cron Scripts Reliably At Scale

Cron scripts are responsible for critical Slack functionality. They ensure reminders execute on time, email notifications are sent, and databases are cleaned up, among other things. Over the years, both the number of cron scripts and the amount of data these scripts process have increased. While generally these cron scripts executed as expected, over time…

Slack logo
Slack
•
Claire Adams
•7 min read•beginner•
--
•View Original
ChefGolangKubernetesRedis

Overview

The article discusses the evolution of cron script execution at Slack, detailing the challenges faced with reliability and scalability. It introduces a new system designed to execute cron scripts more reliably at scale, leveraging existing services and technologies.

What You'll Learn

1

How to design a reliable cron execution system using Kubernetes

2

Why leveraging existing services can reduce maintenance overhead

3

How to implement job deduplication using a Vitess table

Key Questions Answered

What are the main components of Slack's new cron execution system?
The new cron execution system at Slack consists of three main components: a Scheduled Job Conductor service built in Golang, Slack's Job Queue for executing high volumes of work, and a Vitess table for job deduplication and monitoring. This architecture enhances reliability and scalability.
How does the Scheduled Job Conductor ensure reliability?
The Scheduled Job Conductor uses Kubernetes Leader Election to designate one pod for scheduling while keeping others in standby mode. This design minimizes downtime and allows for quick transitions between pods, ensuring that cron jobs are executed reliably.
What role does Slack's Job Queue play in the cron execution process?
Slack's Job Queue is an asynchronous compute platform that processes about 9 billion jobs daily. It efficiently manages job execution by utilizing Kafka for durable storage and Redis for short-term storage, allowing for quick execution of isolated jobs.
How does the Vitess table improve job tracking?
The Vitess table records each job execution as a new row, updating the job's state as it progresses. This allows users to check for active jobs and monitor the status of their scripts, enhancing visibility and control over cron job executions.

Key Statistics & Figures

Daily job executions
9 billion
This statistic highlights the scale at which Slack's Job Queue operates, emphasizing the need for a reliable cron execution system.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Backend
Golang
Used to build the Scheduled Job Conductor service.
Orchestration
Kubernetes
Used for managing the deployment and scaling of the Scheduled Job Conductor pods.
Database
Vitess
Used for job deduplication and monitoring.
Messaging
Kafka
Used for durable storage of jobs in the Job Queue.
Database
Redis
Used for short-term storage of job metadata.

Key Actionable Insights

1
Implement a Scheduled Job Conductor to manage cron jobs effectively.
Using a dedicated service allows for better control over scheduling and reduces the risk of failure associated with single-node setups.
2
Utilize existing infrastructure like Slack's Job Queue to handle job execution.
By offloading execution to a robust system already in place, you can enhance performance while minimizing the need for additional resources.
3
Incorporate a Vitess table for job monitoring and deduplication.
This approach not only prevents duplicate job executions but also provides a clear overview of job statuses, which is crucial for long-running scripts.

Common Pitfalls

1
Relying on a single node for cron job execution can lead to significant downtime.
This happens because any issues with provisioning or configuration can halt all scheduled tasks. A distributed approach mitigates this risk.
2
Neglecting job deduplication can result in multiple instances of the same job running simultaneously.
Without proper tracking, scripts that take longer than their recurrence may overlap, causing resource contention and unexpected behaviors.