Serverless Jupyter Notebooks at Meta

At Meta, Bento, our internal Jupyter notebooks platform, is a popular tool that allows our engineers to mix code, text, and multimedia in a single document. Use cases run the entire spectrum from w…

Meta logo
Meta
•
Steve Dini
•6 min read•intermediate•
--
•View Original
GraphQLServerlessSQLWebAssembly

Overview

The article discusses the implementation of serverless Jupyter notebooks at Meta, leveraging the Bento platform and Pyodide to enable in-browser code execution for lite workloads. This innovation aims to streamline the user experience by eliminating the need for remote compute provisioning, thereby enhancing productivity for engineers working on simple prototypes and machine learning workflows.

What You'll Learn

1

How to utilize Pyodide for in-browser code execution in Jupyter notebooks

2

Why serverless architecture enhances the user experience for lite workloads

3

When to implement serverless solutions for data operations in Bento

Prerequisites & Requirements

  • Understanding of Jupyter notebooks and their typical use cases
  • Familiarity with WebAssembly and Pyodide(optional)

Key Questions Answered

What are lite workloads in the context of Meta's Jupyter notebooks?
Lite workloads are defined as those that consume data from upstream systems without side effects, utilizing up to the maximum Chrome tab memory limit. They require a quick and intuitive startup process without the complexity of remote compute reservation.
How does Meta's serverless Jupyter notebooks architecture work?
Meta's architecture integrates Pyodide with the Bento platform, allowing code execution directly in the browser. This setup includes a kernel abstraction that seamlessly interacts with both traditional server-based and new browser-based kernels, enhancing user experience for lite workloads.
What integrations have been developed for Meta's serverless notebooks?
Meta has built integrations for SQL, Google Sheets, GraphQL, and dataframe uploads, allowing users to fetch and process data from various sources directly within the notebook environment, enhancing functionality and usability.

Technologies & Tools

Platform
Bento
Internal Jupyter notebooks platform used at Meta for data analysis and prototyping.
Library
Pyodide
Enables Python execution in the browser through WebAssembly.
Technology
Webassembly
Provides a compilation target for running code in web browsers, facilitating the serverless architecture.

Key Actionable Insights

1
Implementing serverless Jupyter notebooks can significantly reduce the time engineers spend on provisioning compute resources.
By allowing code execution directly in the browser, teams can focus more on development and less on setup, which is crucial for rapid prototyping and testing.
2
Utilizing Pyodide for in-browser execution can enhance the performance of data operations in Jupyter notebooks.
This approach minimizes the overhead associated with remote compute, making it ideal for lightweight tasks and improving overall efficiency.
3
Integrating existing tools like SQL and Google Sheets into the notebook environment can streamline data workflows.
This allows users to leverage familiar tools without leaving the notebook interface, promoting a more cohesive data analysis experience.

Common Pitfalls

1
Failing to recognize the limitations of WebAssembly can lead to performance issues in serverless notebooks.
Since not all libraries are compatible with WebAssembly, users may encounter functionality gaps. It's important to assess library compatibility before transitioning to a serverless architecture.

Related Concepts

Serverless Architecture
Webassembly
Data Analysis Workflows
Jupyter Notebooks