XARs: An efficient system for self-contained executables

Chip Turner

Distributing large pieces of software to thousands of machines with a wide variety of configurations can pose a significant operational challenge, requiring a process to identify and copy precisely…

Overview

XARs (Executable Archives) are a new system developed by Facebook for distributing self-contained executables that encapsulate both data and code dependencies, significantly improving the efficiency and speed of deployment across diverse machine configurations. This article details the evolution of XARs, their performance benefits compared to previous methods, and provides practical usage examples.

What You'll Learn

1

How to create self-contained executables using XARs

2

Why XARs improve deployment efficiency for Python applications

3

When to use XARs over traditional packaging methods like PARs and PEX files

Prerequisites & Requirements

Basic understanding of packaging and deployment in Python
FUSE filesystem for macOS or Linux(optional)

Key Questions Answered

What are XARs and how do they work?

XARs are self-contained executables that encapsulate both data and code dependencies, allowing them to run independently of the host operating system's libraries. They utilize a modified squashfs format to mount and unmount themselves, providing efficient on-demand access to the necessary files without the need for explicit decompression.

How do XARs compare to other executable formats like PARs and PEX?

Unlike PARs, XARs do not require explicit decompression and can mount themselves when executed, leading to faster start times and less overhead. They also provide better compatibility with modules and dependencies, making them a more efficient choice for deploying Python applications.

What performance benefits do XARs offer?

XARs demonstrate significant improvements in file size and execution speed compared to PEX and native installs. The use of zstd compression allows for smaller file sizes, while their on-demand mounting reduces cold and hot start times, making them ideal for short-running executables.

Key Statistics & Figures

File size reduction

XARs provide a more compressed format than PAR files

This compression is achieved through the use of zstd compression, which is particularly beneficial when deploying to numerous servers.

Cold start time

Improved compared to traditional methods

XARs mount only necessary files on demand, reducing the time taken to start applications.

Hot start time

Faster than PEX and PAR

The caching mechanism allows for quicker access to previously used files, enhancing performance.

Technologies & Tools

Packaging

Xar

Used for creating self-contained executables that encapsulate dependencies.

Filesystem

Fuse

Enables the mounting of user-space filesystems, allowing XARs to operate efficiently on macOS and Linux.

Compression

Zstd

Used for compressing XAR files to achieve better performance and smaller sizes.

Key Actionable Insights

1
Utilize XARs for deploying Python applications to streamline your deployment process.
XARs encapsulate all dependencies, eliminating the need for virtual environments and reducing compatibility issues with system libraries, making deployment faster and more reliable.

2
Adopt the use of FUSE-based filesystems to enhance the performance of your applications.
The FUSE-based architecture of XARs allows for efficient caching and on-demand access to files, which can significantly improve the performance of applications that rely on large datasets or multiple dependencies.

Common Pitfalls

1

Overlooking the need for FUSE dependencies on macOS or Linux systems.

Failing to install FUSE can lead to issues when trying to mount XAR files, resulting in errors during execution. Ensure that FUSE is properly set up before deploying XARs.

Related Concepts

Executable Packaging

Dependency Management

Performance Optimization Techniques