How Meta discovers data flows via lineage at scale

Rishab Mangla

Data lineage is an instrumental part of Meta’s Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for s…

Overview

The article discusses how Meta implements data lineage as part of its Privacy Aware Infrastructure (PAI) initiative to enhance user privacy through scalable data flow discovery. It highlights the techniques used to collect data flow signals and the importance of understanding data lineage in developing privacy controls across Meta's systems.

What You'll Learn

1

How to implement data lineage tracking for sensitive data in large systems

2

Why understanding data flows is crucial for privacy control implementation

3

How to utilize static and runtime analysis for data flow discovery

Prerequisites & Requirements

Understanding of data privacy regulations and practices
Familiarity with static code analysis and runtime instrumentation tools(optional)
Experience in software development and data management

Key Questions Answered

How does Meta track data lineage for user information?

Meta tracks data lineage by collecting data flow signals through static code analysis, runtime instrumentation, and input-output data matching. This enables the creation of an end-to-end lineage graph that illustrates how user data, such as religious views in the Facebook Dating app, is processed across various systems.

What are the benefits of implementing data lineage in Meta's systems?

Implementing data lineage provides scalable data flow discovery, efficient rollout of privacy controls, and continuous compliance verification. These benefits empower developers to quickly identify data flows and apply necessary privacy measures, ensuring user data protection across Meta's applications.

What challenges did Meta face in developing data lineage technology?

Meta faced challenges in collecting lineage from diverse systems and ensuring comprehensive coverage. Initially, the approach required each system to collect lineage signals, which was slow. This was overcome by developing efficient PAI libraries that integrated lineage collection capabilities across various programming languages.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Backend

Privacy Probes

Automates data flow discovery by collecting runtime signals during request execution.

Backend

Presto

Used for logging SQL queries in data processing activities.

Backend

Spark

Another compute engine involved in processing data for the data warehouse.

Key Actionable Insights

1
Investing in data lineage early in the development process can significantly enhance the implementation of privacy controls.
By understanding data flows from the outset, teams can streamline the application of privacy measures, uncovering new opportunities for technology application beyond privacy, such as security.

2
Developing user-friendly tooling for lineage consumption can drastically reduce engineering efforts.
Initial complexity in using raw lineage signals led to inefficiencies. By creating iterative tools that guide engineers, Meta improved the discovery process for relevant data flows.

3
Integrating lineage collection into existing systems can enhance coverage and efficiency.
Meta's approach to embedding lineage collection logic into various programming languages allowed for smoother integration, overcoming initial challenges in collecting lineage signals.

Common Pitfalls

1

Relying solely on manual data flow diagrams can lead to inaccuracies and inefficiencies in large systems.

This traditional method does not scale well in dynamic environments like Meta, where code evolves continuously. Automating data flow tracking through tools like Privacy Probes is essential to maintain accuracy.

Related Concepts

Data Privacy

Data Management

Privacy Aware Infrastructure