By Bonnie Xu, Aravind Suresh, and Emma Tang
Overview
OpenAI built a bespoke internal AI data agent powered by GPT-5.2 that enables employees across Engineering, Data Science, Go-To-Market, Finance, and Research to go from question to insight in minutes instead of days. The agent reasons over OpenAI's 600-petabyte data platform spanning 70,000 datasets, using six layers of context including table usage metadata, human annotations, Codex-enriched code analysis, institutional knowledge from Slack/Docs/Notion, a self-learning memory system, and live runtime queries. The article covers the architecture, context enrichment pipeline, evaluation strategy using OpenAI's Evals API, security model, and key lessons learned from building the agent.
What You'll Learn
1
How to architect a multi-layered context system for AI data agents using table metadata, code enrichment, institutional knowledge, memory, and runtime queries
2
Why code-level understanding of data pipelines is critical for agents to distinguish between similar-looking tables
3
How to build a self-learning memory system that retains corrections and non-obvious constraints for improved data accuracy over time
4
How to evaluate AI agent quality using curated question-answer pairs with golden SQL queries and OpenAI's Evals API
5
When to use high-level guidance versus prescriptive prompting for AI agents to improve robustness
Prerequisites & Requirements
- Understanding of SQL and data warehousing concepts including joins, CTEs, and query optimization
- Familiarity with AI/ML concepts including LLMs, embeddings, and retrieval-augmented generation (RAG)
- Understanding of data platform architecture including metadata services, data pipelines, and schema management
- Experience with building or using AI agents or LLM-powered tools(optional)
Key Questions Answered
How does OpenAI's internal data agent handle context for accurate query generation?
The agent uses six layers of context: table usage metadata (schema, lineage, historical queries), human annotations from domain experts, Codex-enriched code analysis that derives table definitions from pipeline source code, institutional knowledge from Slack/Docs/Notion embedded with permissions, a self-learning memory system that stores corrections and non-obvious constraints, and live runtime queries to the data warehouse for real-time validation.
Why did OpenAI build a custom data agent instead of using off-the-shelf tools?
OpenAI's data platform serves over 3,500 internal users across 600 petabytes of data spanning 70,000 datasets. Simply finding the right table among many similar-looking options was one of the most time-consuming parts of analysis. Generic tools couldn't handle the institutional context, internal terminology, overlapping table schemas, and complex SQL semantics needed to produce correct results at this scale.
How does the data agent's memory system improve over time?
When the agent receives corrections or discovers nuances during conversations, it prompts users to save these learnings as memories. Memories can be global or personal, and are manually editable. They retain non-obvious corrections, filters, and constraints critical for data correctness that are difficult to infer from other context layers. For example, the agent learned specific string patterns needed to filter analytics experiments correctly instead of fuzzy matching.
How does OpenAI evaluate the accuracy of their data agent's responses?
The agent uses OpenAI's Evals API with curated question-answer pairs where each question is paired with a manually authored 'golden' SQL query. The system sends natural language questions to the query-generation endpoint, executes the generated SQL, and compares outputs against expected results. Evaluation uses both SQL and dataframe comparison rather than naive string matching, with a grader producing scores and explanations to capture correctness and acceptable variation.
What role does Codex play in OpenAI's data agent architecture?
Codex crawls the OpenAI codebase to derive code-level definitions of tables, providing deeper understanding of what data actually contains. It extracts details like table purpose, grain and primary keys, downstream usage patterns, alternate table options, and data freshness. This enrichment is automatically refreshed and helps the agent distinguish between tables that look similar in schema but differ in critical ways, such as whether a table includes only first-party ChatGPT traffic.
What are the key lessons from building an enterprise AI data agent?
Three main lessons emerged: First, 'Less is More' — consolidate overlapping tools to reduce agent confusion. Second, 'Guide the Goal, Not the Path' — highly prescriptive prompting degraded results; high-level guidance with GPT-5's reasoning produced better outcomes. Third, 'Meaning Lives in Code' — pipeline source code captures assumptions, freshness guarantees, and business intent that never surface in SQL or metadata alone, making code analysis essential for accurate table understanding.
How does OpenAI's data agent handle security and access control?
The agent operates as a purely pass-through interface layer, inheriting and enforcing the same permissions that govern OpenAI's data platform. Users can only query tables they already have permission to access. When access is missing, the agent flags this or falls back to alternative authorized datasets. It exposes its reasoning process by summarizing assumptions and execution steps, and links directly to underlying query results for user verification.
Where is OpenAI's data agent available to employees?
The agent is available wherever employees already work: as a Slack agent, through a dedicated web interface, inside IDEs, via the Codex CLI through MCP (Model Context Protocol), and directly in OpenAI's internal ChatGPT app through an MCP connector. This multi-surface approach ensures the agent blends naturally into existing workflows rather than functioning as a separate tool.
Key Statistics & Figures
Internal data platform users
3,500+
Users working across Engineering, Product, and Research at OpenAI
Total data volume
600 petabytes
Data managed by OpenAI's internal data platform
Number of datasets
70,000
Datasets across OpenAI's data platform
Query time without memory
22 minutes 41 seconds
Time for agent to answer a question without memory context
Query time with memory
1 minute 22 seconds
Time for the same query using the agent's memory system
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
AI Model
Gpt-5.2
Powers the data agent's reasoning and query generation
AI Coding Tool
Codex
Crawls codebase to extract table-level knowledge and code-enriched context for data pipeline understanding
Evaluation Framework
Openai Evals API
Measures and protects the agent's response quality through curated question-answer pair evaluation
AI API
Openai Embeddings API
Converts enriched table context into embeddings for RAG-based retrieval
Protocol
Mcp
Model Context Protocol used for Codex CLI integration and ChatGPT app connector
Communication Platform
Slack
One of the agent's deployment surfaces and a source of institutional knowledge
Documentation Platform
Google Docs
Source of institutional knowledge including launch docs and metric definitions
Documentation Platform
Notion
Source of institutional knowledge
Query Language
SQL
Agent generates and executes SQL queries against the data warehouse
Data Processing
Spark
Part of the data platform systems the agent can interact with for broader data context
Workflow Orchestration
Airflow
Part of the data platform systems the agent can query for pipeline metadata
AI Architecture Pattern
Rag
Retrieval-augmented generation used to pull relevant embedded context at query time
Key Actionable Insights
1Build multiple layers of context rather than relying on schema metadata alone when creating data agents. Combine table usage patterns, human annotations, code-level enrichment, institutional knowledge, persistent memory, and runtime queries to ground agent reasoning in both technical and business context.OpenAI found that without rich context, even strong models produce wrong results like vastly misestimating user counts or misinterpreting internal terminology. Each layer addresses a different failure mode.
2Implement a self-learning memory system that captures corrections and non-obvious constraints from user interactions. Scope memories at both global and personal levels, and prompt users to save learnings during conversations rather than relying solely on pre-built context.Memory proved crucially important for retaining specific details like exact string patterns for experiment filtering that couldn't be inferred from other context layers. Without memory, identical queries took 22+ minutes versus 1 minute 22 seconds with memory.
3Use Codex or similar code analysis tools to enrich data catalog entries by crawling the source code of data pipelines. Extract table purpose, grain, primary keys, downstream usage patterns, alternate table options, and data freshness directly from the code that produces each dataset.Schemas and query history describe a table's shape and usage, but the true meaning lives in the pipeline code that produces it. This code captures assumptions, freshness guarantees, and business intent that never surface in SQL or metadata.
4Consolidate and restrict overlapping tool capabilities when building AI agents. Redundant functionality that seems helpful for manual use creates confusion for agents, leading to unreliable behavior and degraded results.OpenAI initially exposed their full tool set and quickly encountered problems with overlapping functionality. While redundancy can help humans making deliberate choices, it confuses agents that must choose between similar options.
5Prefer high-level goal-oriented guidance over highly prescriptive step-by-step prompting for AI agents. Let the model's reasoning capabilities determine the appropriate execution path rather than rigidly scripting the analysis flow.OpenAI discovered that prescriptive prompting degraded results because while many data questions share a general analytical shape, the details vary enough that rigid instructions pushed the agent down incorrect paths.
6Build continuous evaluation pipelines using curated question-answer pairs with golden SQL queries, and compare both the generated SQL and the resulting data using model-graded evaluation rather than naive string matching.Generated SQL can differ syntactically while still being correct, and result sets may include extra columns that don't affect the answer. OpenAI runs these evals as unit tests during development and as canaries in production to catch regressions early.
Common Pitfalls
1
Exposing too many overlapping tools to an AI agent creates confusion and unreliable behavior. While redundant functionality seems helpful for human users who can make deliberate choices, agents struggle to choose between similar tools and may select suboptimal paths.
OpenAI learned to restrict and consolidate tool calls to reduce ambiguity and improve reliability.
2
Using highly prescriptive, step-by-step prompting for data agents degrades results because analytical questions share a general shape but vary significantly in details. Rigid instructions push agents down incorrect paths when the question doesn't perfectly match the scripted flow.
Shifting to higher-level guidance and relying on the model's reasoning to choose execution paths made the agent more robust.
3
Relying solely on schema metadata and query history to understand data tables leads to incorrect results. Tables that look similar in structure can differ in critical ways (e.g., one includes logged-out users while another doesn't, or one covers only first-party traffic).
The true meaning of a table lives in the pipeline code that produces it. Code analysis captures assumptions, freshness guarantees, and business intent that never surface in metadata.
4
Building an evolving AI agent without systematic evaluation leads to invisible quality regressions. Without a tight feedback loop comparing agent outputs against golden references, quality can drift as easily as it improves.
OpenAI treats evals like unit tests that run continuously during development and as canaries in production to catch issues early.
5
Not implementing a memory system means the agent repeatedly encounters the same issues, such as not knowing how to filter for specific analytics experiments or which exact tables to use for common queries. This leads to dramatically slower response times (22+ minutes vs ~1.5 minutes).
Memory retains non-obvious corrections, filters, and constraints that are critical for data correctness but impossible to infer from schemas or metadata alone.
Related Concepts
Retrieval-augmented Generation (rag)
AI Agent Architecture
Data Catalog Enrichment
Embeddings And Semantic Search
Llm Evaluation And Evals
Text-to-sql Generation
Data Warehouse Query Optimization
Institutional Knowledge Management
Agent Memory Systems
Model Context Protocol (mcp)
Data Pipeline Lineage
Access Control And Data Security
Agentic Workflows
Prompt Engineering For Agents