We built an agentic security investigation service to help us research alerts as part of our mission to keep Slack secure and protect our customers. Our service deploys teams of AI agents that collaboratively perform security investigations. AI agents free human analysts from tedious data gathering tasks. Over just the first quarter of their deployment, our agents have performed over 7,500 investigations, issuing over 500,000 tool calls. Our agents are enabling us to gain unprecedented real-time insight into Slack’s infrastructure in a way we could never do with human labor alone.
Overview
Slack's Security Engineering team describes how they built an AI agent-based system to automate and streamline security investigations. The system uses a multi-persona architecture with Director, Expert, and Critic agents that collaborate through structured investigation phases (Discovery, Trace, Conclude) to analyze security alerts, cross-reference evidence across data sources, and produce verifiable investigation reports with a knowledge pyramid approach that strategically uses different model tiers for cost optimization.
What You'll Learn
1
How to decompose a complex AI investigation prompt into a structured multi-agent system with defined personas and tasks
2
Why breaking single-prompt AI workflows into chained structured-output model invocations produces more consistent and controllable results
3
How to design a knowledge pyramid architecture that strategically assigns low, medium, and high-cost LLM models to different agent roles
4
How to implement a Critic agent pattern that provides adversarial review to mitigate hallucinations and improve finding quality
5
When to use phased investigation flows (Discovery, Trace, Conclude) to structure AI-driven security analysis
Prerequisites & Requirements
- Understanding of LLM prompting techniques and structured outputs (JSON schema constraints)
- Familiarity with security operations concepts such as alert triage, investigation workflows, and indicators of compromise
- Understanding of the Model Context Protocol (MCP) and tool-calling interfaces for LLMs(optional)
- Experience with multi-agent or multi-persona AI system design patterns(optional)
Key Questions Answered
How does Slack use AI agents to automate security investigations?
Slack built a multi-agent system with three persona types: a Director agent that orchestrates the investigation, domain Expert agents (Access, Cloud, Code, Threat) that query data sources and produce findings, and a Critic agent that reviews finding quality using a credibility rubric. These agents collaborate through structured phases, with the application orchestrating model invocations and propagating context between each step.
Why did Slack move from a single-prompt AI prototype to a multi-agent architecture?
The initial 300-word prompt prototype produced highly variable results—sometimes excellent cross-referenced findings, other times spurious conclusions. Prompt refinements proved insufficient because prompts are just guidelines, not fine-grained control mechanisms. Breaking the investigation into separate model invocations with structured outputs gave each step a well-defined purpose and predictable behavior, such as turning 'question your evidence' guidance into a dedicated Critic task.
What is the knowledge pyramid pattern in AI agent systems?
The knowledge pyramid is a cost-optimization architecture where domain experts at the bottom use low-cost models for token-intensive data querying and analysis. The Critic in the middle uses medium-cost models to review claims and assemble timelines. The Director at the top uses high-cost models but receives only condensed, high-quality findings. This design allows strategic model selection based on task complexity while progressively distilling investigation knowledge upward.
What are the investigation phases in Slack's AI security system?
There are three phases: Discovery, where the Director broadcasts questions to all experts to examine every data source; Trace, where the Director targets specific experts who produced relevant findings with focused questions and potentially enhanced token budgets; and Conclude, where sufficient information has been gathered to produce the final investigation report. The Director decides when to advance between phases.
How does the Critic agent help reduce AI hallucinations in security investigations?
The Critic acts as a meta-expert that assesses finding quality using a defined rubric. It inspects experts' claims along with the actual tool calls and results used to support them, assigning credibility scores to each finding. The weakly adversarial relationship between the Critic and expert group helps catch hallucinations and inconsistent evidence interpretation. In one case, the Critic identified a credential exposure that the expert had incorrectly assessed as secure.
What service architecture does Slack use for its AI investigation system?
The architecture has three components: a Hub that provides the service API, persistent storage, and metrics endpoint for monitoring token usage and cost; Workers that pick up queued investigation tasks and stream events back to the Hub (scalable for throughput); and a Dashboard that lets staff launch investigations, observe them in real-time via event streams, and inspect individual model invocations for debugging.
How can AI agents discover security issues beyond the original alert scope?
Slack's agents demonstrate emergent behavior by following evidence trails beyond the triggering alert. In an example investigation, agents were investigating a specific command sequence but independently traced the process ancestry tree and discovered a separate credential exposure in an ancestor process. The Critic noticed this during meta-analysis even though the expert missed it, and the Director pivoted the investigation to focus on the newly discovered issue.
What structured output approach works best for multi-step AI agent workflows?
Each agent/task pair should have a carefully defined JSON schema structured output applied to the model's last output. This constrains the model to produce predictable, parseable results. However, structured outputs aren't free—overly complex schemas can cause execution failures, and models may still hallucinate within the structure. The key is designing schemas that match the model's capability for each specific task in the chain.
Key Statistics & Figures
Security events processed daily
Billions
Slack's security event ingestion pipeline handles billions of events per day from diverse data sources
Initial prototype prompt size
~300 words
The rudimentary prototype consisted of a five-section prompt of approximately 300 words
Number of domain expert agents
4
Access, Cloud, Code, and Threat experts each with unique domain knowledge and data sources
Number of investigation phases
3
Discovery, Trace, and Conclude phases that structure the investigation flow
Technologies & Tools
AI Infrastructure
Mcp (model Context Protocol)
Stdio-mode MCP server used to safely expose security data sources through the tool call interface
Data Format
JSON Schema
Used for defining structured output formats that constrain model outputs for each agent/task pair
AI/ML
Llm (large Language Models)
Multiple model tiers (low, medium, high cost) used for expert, critic, and director functions respectively
Key Actionable Insights
1Break complex AI workflows into separate model invocations with structured outputs rather than relying on a single long prompt. Each invocation should have a single, well-defined purpose and output schema, chained together by your application logic. This gives you fine-grained control over each step that prompt engineering alone cannot achieve.Slack found that their initial single-prompt approach produced highly variable results despite extensive prompt refinement. Separating tasks like 'question evidence' into dedicated model calls with structured outputs made behavior much more predictable.
2Implement an adversarial Critic agent that reviews the output of other agents using a defined quality rubric and credibility scoring. This pattern creates a weakly adversarial relationship that catches hallucinations, misinterpretations, and blind spots that individual agents miss.In Slack's system, the Critic caught a credential exposure that the domain expert incorrectly assessed as secure, demonstrating that adversarial review can surface critical findings that pass through primary analysis.
3Design a knowledge pyramid that assigns model tiers strategically: use low-cost models for token-intensive data gathering at the bottom, medium-cost models for quality review and synthesis in the middle, and high-cost models for high-level decision-making at the top. Only pass condensed, high-quality findings upward.This approach optimizes cost by reserving expensive model capacity for tasks that require the most reasoning capability, while letting cheaper models handle the bulk of data processing and analysis.
4Structure multi-agent investigations into distinct phases (Discovery, Trace, Conclude) that allow you to vary agent behavior, model parameters, and token budgets as the investigation progresses. Let a Director agent decide when to transition between phases.In Discovery, questions broadcast to all experts ensure comprehensive data source coverage. In Trace, the Director targets specific experts with focused questions. This prevents premature conclusions while keeping investigations efficient.
5Use MCP (Model Context Protocol) servers to safely expose data sources to AI agents through the tool call interface, rather than giving agents direct access to production systems. This creates a controlled boundary between the AI system and your infrastructure.Slack implemented an stdio-mode MCP server that exposed a subset of their security data sources, providing the agents with investigation capabilities while maintaining security boundaries around sensitive infrastructure.
6Build observability into your agent system from the start with real-time event streaming, per-invocation inspection capabilities, and metrics for token usage and cost. The ability to watch investigations unfold and debug individual model calls is invaluable for iterating on agent behavior.Slack's Dashboard provides real-time observation of running investigations and detailed views of each model invocation, which they describe as invaluable when debugging the system.
Common Pitfalls
1
Relying on a single complex prompt to control an AI agent's investigation behavior leads to highly variable results. Even with extensive prompt refinements stressing assumptions-checking and multi-source verification, the agent may still jump to convenient or spurious conclusions because prompts are guidelines, not enforceable control mechanisms.
Slack experienced this firsthand with their prototype and solved it by decomposing the investigation into separate model invocations with structured outputs, giving them deterministic control over each investigation step.
2
Using overly complex JSON schemas for structured outputs can cause model execution failures. Structured outputs also don't eliminate hallucination—models can still produce fabricated data that conforms to the schema format, giving a false sense of reliability.
The solution is to keep each task's output schema appropriately scoped for the model's capabilities and implement adversarial review (like a Critic agent) to catch hallucinations within structured responses.
3
Domain expert agents can develop blind spots where they incorrectly assess security-relevant findings. In Slack's example, an expert assessed credential handling as secure when credentials were actually exposed in process command line parameters, representing a significant analysis failure.
The Critic agent pattern mitigates this by independently reviewing expert claims against the actual evidence (tool calls and results), but teams should be aware that no single agent layer is fully reliable on its own.
4
Using a coding agent CLI as an execution environment for production AI investigations lacks the observability, integration, and management capabilities needed for practical deployment. Without real-time monitoring, event streaming, and integration with existing detection tools, the system cannot scale beyond experimentation.
Slack replaced their CLI-based prototype with a Hub/Worker/Dashboard architecture that provides API integration, persistent storage, real-time observation, and scalable worker pools.
Related Concepts
Multi-agent AI Systems
Meta-prompting
Multi-persona Self-collaboration
Structured Output / Constrained Decoding
Model Context Protocol (mcp)
Security Operations Center (soc) Automation
Security Alert Triage
Adversarial AI Review Patterns
Process Ancestry Analysis
Indicator Of Compromise (ioc) Detection
Security Tabletop Exercises
Llm Orchestration Pipelines
Token Cost Optimization
Human-in-the-loop AI Systems
Event-driven Architecture