GraphQL Data Mocking at Scale with LLMs and @generateMock

Key Actionable Insights

1

Integrate LLM-based generation directly into existing developer tooling rather than building separate tools. Airbnb embedded mock generation into their existing Niobe CLI code generator, so engineers trigger mock generation with the same command they already use for GraphQL code generation. This eliminates context-switching and drives adoption.

Engineers are more likely to adopt tools that fit into their existing workflows. By making mock generation a side effect of normal code generation, Airbnb ensured seamless adoption without requiring engineers to learn new tools or processes.

2

Always validate LLM output against a formal schema and implement a self-healing retry loop. Airbnb validates generated mock data using graphqlSync against the GraphQL schema, and if validation fails, feeds errors back to the LLM to correct the output. This provides strong guarantees that final output is fully valid.

LLMs can hallucinate invalid enum values or miss required fields. By placing the LLM within existing validation infrastructure rather than using it as a standalone tool, you can enforce guardrails that ensure correctness.

3

Prune the schema context sent to the LLM rather than including the entire schema. Airbnb traverses the schema and strips out types, fields, and whitespace not needed to resolve the specific query being mocked. This prevents context window overload while still providing the type information the LLM needs.

Large GraphQL schemas can exceed LLM context windows. Schema traversal and pruning ensures only relevant type definitions are included, improving both generation quality and performance.

4

Use hash-based versioning to detect when generated artifacts drift from their source definitions. Embedding hashes of both the query document and directive arguments in generated files enables smart regeneration that only updates mocks when the underlying query actually changes.

This approach prevents unnecessary regeneration, preserves manual engineer tweaks to mock data, and enables CI checks that guarantee mocks stay synchronized with evolving queries.

5

Provide the LLM with curated, valid resource URLs (like images) to prevent hallucination of non-existent resources. Airbnb feeds the LLM a list of real Airbnb-hosted image URLs with short descriptions so generated mock data contains loadable images at runtime.

LLMs commonly hallucinate URLs that don't exist. By constraining the LLM to choose from a known set of valid resources, you ensure mock data actually works when loaded in the application for prototyping or demos.

6

Support hybrid production/mock data at the field level rather than only full query mocking. Airbnb's @respondWithMock directive can be applied to individual fields, allowing the client to fetch real data from the server for existing fields while patching in mock data only for new, unimplemented fields.

This granular approach is more practical for iterative development where engineers add new fields to existing queries. It allows client teams to develop against partially implemented backends without losing access to real data for already-complete features.