There’s No Such Thing as a Free Lunch!

Incident Management takes time Incidents need responders that are trained and experienced.  At Slack, training is a foundation of our incident management program. Self-service training and live courses based mainly on prepared content are one piece of the puzzle, but there can be a missing piece in many organizations. How can staff get practical experience…

Slack logo
Slack
•
Scott Nelson Windels
•14 min read•intermediate•
--
•View Original
ChefPagerDuty

Overview

The article discusses how Slack trains its engineers in incident response through a unique exercise called the Incident Lunch. This approach combines practical experience with role-playing and time constraints, making the training both effective and engaging.

What You'll Learn

1

How to conduct an engaging incident response training exercise using real-life scenarios

2

Why role-playing and time constraints enhance incident response training effectiveness

3

How to incorporate unpredictability into training exercises with Chaos Cards

Key Questions Answered

How does Slack train engineers in incident response?
Slack trains engineers through a practical exercise called the Incident Lunch, where participants must order lunch under time constraints while following incident response protocols. This method allows engineers to gain hands-on experience in a fun and engaging way.
What are the key elements of the Incident Lunch exercise?
The Incident Lunch exercise includes time pressure, role-playing, constraints on lunch options, and the use of Chaos Cards to introduce unpredictability. These elements help simulate real incident scenarios and enhance the learning experience.
What resources are needed to run the Incident Lunch exercise?
To run the Incident Lunch, minimal resources are required: a conference room, a facilitator, a budget for lunch (approximately $300-500), and a GitHub repository for training materials. This makes it easy to implement regularly.
What insights have been gained from running the Incident Lunch exercises?
Insights from the Incident Lunch exercises include the importance of having a designated Incident Commander, the effectiveness of Chaos Cards in increasing unpredictability, and the need for quick decision-making in incident response scenarios.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Tools
Github
Used for managing training materials and facilitating the Incident Lunch exercise.

Key Actionable Insights

1
Incorporate time constraints into training exercises to simulate real incident scenarios.
Time pressure helps participants practice decision-making under stress, which is crucial during actual incidents.
2
Use role-playing to enhance engagement and learning in incident response training.
Role-playing allows participants to experience different incident roles, improving their understanding of the incident response process.
3
Implement Chaos Cards to introduce unpredictability in training exercises.
Adding variability through Chaos Cards mimics the unpredictable nature of real incidents, making training more relevant.

Common Pitfalls

1
Failing to designate an Incident Commander can lead to confusion during the exercise.
Without a clear leader, teams may struggle to make decisions quickly, which is critical in real incident scenarios.
2
Overcomplicating the exercise with too many rules or constraints can hinder learning.
Keeping the exercise simple allows participants to focus on the core objectives of incident response rather than getting bogged down in logistics.

Related Concepts

Incident Management
Incident Response
Training And Development