How we built an AI agent that answers 1,000+ data questions a month
Overview
Ramp built an internal AI agent called Ramp Research that serves as an agentic data analyst, answering data questions directly in Slack to eliminate the bottleneck of relying on a single on-call analyst. The agent leverages metadata from dbt, Looker, and Snowflake combined with domain-specific documentation to answer over 1,800 questions across 1,200 conversations, resulting in a 10-20x increase in the number of data questions employees ask.
What You'll Learn
1
How to build an agentic AI data analyst that answers questions from a large analytics warehouse via Slack
2
How to define and organize a context layer using dbt, Looker, and Snowflake metadata to make an AI agent effective over thousands of tables
3
Why giving an AI agent tools to inspect column values, branch, and backtrack outperforms generic keyword or vector search for data analysis
4
How to build an evaluation framework that asserts on intermediate agent steps, not just final answers
5
Why reducing the cost of asking data questions to near-zero creates a 10-20x increase in organizational data usage
Prerequisites & Requirements
- Understanding of data warehousing concepts and analytics workflows
- Familiarity with dbt, Looker, and Snowflake or similar data stack
- Understanding of AI agents and LLM-based tool use patterns
- Experience with Slack app development and integrations(optional)
Key Questions Answered
How do you build an AI agent that can answer data questions from a large analytics warehouse?
Ramp built their agent by aggregating and indexing metadata from dbt, Looker, and Snowflake to provide context about their data. Domain experts wrote technical documentation organized into a file system the agent can access. Rather than relying on keyword or vector search alone, the agent was given tools to inspect column values, branch, and backtrack through the data, reasoning like a human analyst would.
How does an agentic data analyst handle thousands of tables and views in a data warehouse?
The agent uses a combination of indexed metadata from the analytics stack (dbt, Looker, Snowflake) and domain-specific documentation written by analytics team experts. It has tools to inspect column values, branch between different approaches, and backtrack when needed. This agentic approach lets it navigate scale that would overwhelm generic compression methods like keyword or vector search alone.
What is the impact of making data questions free to ask in an organization?
Ramp saw a 10-20x increase in the number of data questions asked after deploying their AI analyst. Most growth came from questions that previously died in drafts or were never asked at all. In a four-week period, Ramp Research answered 1,476 questions compared to just 66 in the traditional help-data channel, demonstrating massive latent demand for data access.
How do you evaluate an AI data analyst agent beyond just checking final answers?
Ramp built a Python mini-framework within their dbt project that asserts not only on the final answer but also on intermediate steps including expected tool calls, table references, and query shape. This approach replaced their initial human-in-the-loop review and concept-level end-to-end tests, enabling a tight feedback loop: update context, run tests, and confirm improvements.
Why is Slack a good interface for an internal AI data analyst?
Slack was chosen because it's the internal communication hub, making data access zero-friction. The agent supports multi-turn conversations in threads for clarification and collaboration, provides in-thread CSV previews so users can validate results without switching to BI tools, and can be added to existing alert and project channels to diagnose issues and scope features in context.
How do you solve the analytics bottleneck where data questions queue up behind a single analyst?
Ramp deployed an AI agent that answers data questions directly in Slack, 24/7, in minutes instead of hours. This eliminated the dependency on a single on-call analyst in the #help-data channel. The agent answered 1,476 questions in four weeks compared to 66 in the traditional channel, demonstrating it can absorb the vast majority of data question volume.
What role does domain-specific documentation play in making AI agents effective for data analysis?
Domain-specific documentation is critical because much analytics knowledge is tacit and lives with domain owners. At Ramp, domain experts wrote technical documentation about their respective areas, organized into a file system the agent accesses as needed. This context layer bridges the gap between raw data metadata and the business understanding required to answer questions correctly.
Key Statistics & Figures
Total data questions answered
1,800+
Since launching in early August
Total conversations
1,200+
Since launch
Unique users
300+
Different users who have interacted with Ramp Research
Slack channel members
500+
#ramp-research-beta channel
Increase in questions asked
10-20x
Compared to prior question volume
AI agent vs traditional channel questions (4 weeks)
1,476 vs 66
Ramp Research answered 1,476 questions compared to 66 in #help-data over 4 weeks
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Interface
Slack
Primary user interface for the AI agent, supporting multi-turn conversations and CSV previews
Data Warehouse
Snowflake
Analytics warehouse containing thousands of tables and views that the agent queries
Data Transformation
Dbt
Source of metadata context and home of the Python evaluation mini-framework
Business Intelligence
Looker
Source of metadata context for the agent's search space
Business Intelligence
Redash
Previously used by users to inspect data and verify answers before CSV previews were added
Programming Language
Python
Used to build the evaluation mini-framework within the dbt project
Key Actionable Insights
1Invest in a structured context layer rather than relying solely on vector search or keyword matching for AI data agents. Aggregate metadata from your data stack (dbt, Looker, Snowflake) and have domain experts write technical documentation organized in an accessible file system. This domain knowledge is what enables the agent to bridge raw data understanding with business context.Ramp found that even with relevant data context, the agent struggled without domain-specific documentation. The combination of indexed metadata and expert-written docs was key to making the agent effective across different business domains.
2Give your AI agent tools to inspect data at the row level rather than compressing everything into embeddings or summaries. Providing the ability to inspect column values, branch between approaches, and backtrack when reasoning fails enables the agent to handle the complexity of large data warehouses with thousands of tables.Generic compression methods like keyword or vector search weren't sufficient for Ramp's scale. The agentic approach of exploring data iteratively, similar to how a human analyst works, proved more effective for answering complex data questions.
3Build evaluation frameworks that test intermediate reasoning steps, not just final answers. Assert on expected tool calls, table references, and query shape in addition to the final result. This diagnostic approach makes it possible to identify exactly why an agent fails and iterate quickly on improvements.Ramp's initial end-to-end tests exposed blind spots but gave little diagnosis. Their Python mini-framework in dbt enabled a tight feedback loop: update context, run tests, and confirm improvements, which was essential for rapid iteration.
4Deploy AI agents where users already work (like Slack) and support multi-turn conversations to dramatically increase adoption. Adding features like in-thread CSV previews removes friction for non-technical users and allows everyone to validate results without context-switching to specialized BI tools.Ramp's #ramp-research-beta channel grew to 500+ members. Multi-turn threads improved both user experience and agent performance by allowing clarification of intent and collaborative problem-solving.
5Recognize that reducing the cost of asking data questions creates massive latent demand. Plan for a 10-20x increase in question volume and design your system to handle questions that previously went unasked. Most growth will come from questions that people never bothered to submit to the traditional queue.Ramp saw 1,476 questions answered by their AI agent in four weeks compared to just 66 in the traditional help-data channel. The compounding effect of many small improvements in decision quality across pricing, GTM, and feature decisions becomes material at scale.
6Make your AI agent available as a headless API so teams can customize it for domain-specific workflows beyond ad-hoc questions. This enables automation of recurring tasks like generating customer case studies, detecting fraud patterns, and integrating with alert channels for incident diagnosis.Ramp is seeing teams proactively integrate the agent into existing workflows by adding it to alert and project channels, demonstrating that the value extends well beyond answering one-off data questions.
Common Pitfalls
1
Relying exclusively on generic compression methods like keyword or vector search to help an AI agent navigate a large data warehouse. These methods fail to capture the nuance needed when dealing with thousands of tables and views that require row-level inspection to produce correct answers.
Ramp found that giving the agent tools to inspect column values, branch, and backtrack was essential for handling scale and complexity that generic methods couldn't address.
2
Implementing human-in-the-loop evaluation by pinging domain owners for every in-domain question. This approach doesn't scale because effort still increases linearly with request volume and reintroduces the core bottleneck the AI agent was meant to solve.
Ramp initially tried this approach but shifted to automated evaluation with a Python mini-framework that tests intermediate steps, not just final answers.
3
Running only end-to-end tests that check final answers without asserting on intermediate steps like tool calls, table references, and query shape. These tests expose blind spots but provide little diagnostic information about why the agent passes or fails.
Ramp evolved their evaluation approach through multiple iterations before arriving at their mini-framework that tests both intermediate reasoning and final outputs.
4
Assuming that providing the AI agent with raw data metadata alone is sufficient for answering business questions. Much analytics knowledge is tacit and lives with domain owners, so without explicit domain documentation the agent will struggle to connect data understanding to business context.
Ramp had domain experts write technical documentation organized into a file system that the agent accesses as needed, which was critical for bridging the gap between data and business understanding.
Related Concepts
AI Agents For Data Analysis
Agentic Tool Use And Reasoning
Data Warehouse Metadata Indexing
Slack Bot Development
Llm Evaluation Frameworks
Data-driven Decision Making
Analytics Self-service
Domain Knowledge Management
Multi-turn Conversational AI
Dbt Project Organization