How Meta Uses Apache

The article discusses Meta's evolution in infrastructure over 21 years, highlighting the significant changes brought about by AI.

The article discusses Meta's transition to a composable data management architecture, emphasizing interoperability, reusability, and engineering efficiency.

The article discusses the collaboration between Meta, Voltron Data, and the Apache Arrow community to align Apache Arrow with Velox, Meta's open-source execution engine.

The article discusses the implementation of scheduled Jupyter notebooks at Meta, focusing on the integration of Bento with the Dataswarm batch ETL pipeline framework.

The article discusses the implementation and deployment of MySQL Raft at Meta, focusing on how it aims to replace semisynchronous databases with a more reliable and simpler distributed system.

Meta has introduced Velox, an open source unified execution engine designed to enhance data management systems and streamline their development.

Meta has open-sourced the Anonymous Credential Service (ACS), a multitenant service designed for de-identified client authentication, enhancing privacy and security while being compute-conscious.

The article discusses how Meta scales its data ingestion infrastructure to support machine learning training for various AI models.

The article introduces Zelos, a new ZooKeeper API built on the Delos platform, aimed at improving the performance and modularity of distributed systems at Meta.

BellJar is a new framework developed by Meta for testing system recoverability at scale, addressing the complexities of infrastructure outages.

The article discusses the migration of Apache ZooKeeper to the Twine container orchestration platform at Facebook, highlighting the challenges and solutions in managing stateful services within a s...

The article discusses Shard Manager, a platform developed by Facebook to efficiently manage sharded applications at scale.

The article discusses the third annual ZooKeeper Meetup hosted by Facebook, focusing on advancements in performance, scalability, and security within distributed systems.

The 2019 @Scale Conference brought together over 1,300 engineers to discuss challenges and innovations in building scalable applications and services.

The article provides a recap of the Systems @Scale 2019 conference, highlighting discussions on engineering challenges faced by operating systems serving millions of users.

The article provides a recap of the Data @Scale conference held in Boston, focusing on the challenges and advancements in large-scale data storage and analytics.

Horizon is the first open source end-to-end platform that employs applied reinforcement learning (RL) to optimize systems in large-scale production environments.

The 2018 @Scale Conference brought together over 2,500 engineers to explore the challenges and innovations in building scalable applications and services.

The article discusses the expansion of the Open Neural Network Exchange (ONNX) format, which enhances interoperability among various AI frameworks, thereby accelerating AI development.

The article discusses Facebook's advancements in artificial intelligence (AI) showcased at F8 2018, emphasizing open development frameworks like PyTorch 1. 0 and ONNX.

The article announces the release of ONNX V1, a production-ready open standard for deep learning frameworks, developed collaboratively by Facebook, AWS, and Microsoft.

LogDevice is a distributed data store designed specifically for logs, addressing the challenges of high availability and durability at Facebook's scale.

The article discusses the implications of the Apache Software Foundation's decision to categorize the BSD + Patents license as a Category-X license, which affects Facebook's open source projects, p...

The article discusses how Facebook utilized Apache Spark for large-scale language model training, highlighting the transition from a Hive-based solution to a Spark-based pipeline.

This article provides a comprehensive comparison of two state-of-the-art graph processing systems, Apache Giraph and GraphX, focusing on their performance, scalability, and usability for large-scal...

The article discusses Facebook's experience in scaling Apache Spark to handle a 60 TB+ production use case, focusing on the migration from a Hive-based pipeline to a more efficient Spark implementa...

The article recaps the Data @Scale conference held in June 2016, focusing on large-scale storage systems and analytics.

The article recaps the second Networking @Scale conference held on May 10 and 11, 2016, featuring discussions on large-scale networking solutions from industry leaders like Microsoft, Google, and F...

The article discusses Box's migration to HHVM (HipHop Virtual Machine) to enhance performance and reduce latency in their PHP application.

The article discusses the challenges and solutions in implementing collaborative filtering (CF) at Facebook to recommend items to over a billion users.

Proxygen is a high-performance C++ HTTP framework developed by Facebook, designed to facilitate the creation of HTTP services with sensible defaults. It supports protocols like HTTP/1.

The article discusses Facebook's innovative approach to configuration management using Chef, highlighting the need for a scalable system that allows engineers to make changes through data-driven AP...

In the first half of 2014, Facebook's open source program experienced significant growth, launching 64 new projects and increasing its codebase to 9. 9 million lines of code.

The article discusses HydraBase, an evolution of HBase developed at Facebook to enhance reliability and availability in distributed data storage.

The article discusses Facebook's strategies for scaling its data warehouse to manage over 300 PB of data, focusing on the evolution of storage formats from RCFile to ORCFile.

The article discusses the implementation of large-scale graph partitioning using Apache Giraph at Facebook, addressing the challenges of high latency in distributed systems.

The article discusses the evolution of Thrift at Facebook, highlighting the enhancements made to improve performance and functionality, leading to the release of fbthrift as an open-source project.

The article discusses Facebook's commitment to open source in 2013, highlighting significant projects and contributions across various domains such as mobile, web, data, and infrastructure.

The article discusses the scaling of Apache Giraph to handle a trillion edges, detailing the challenges faced and the improvements made to the framework.

The article discusses TAO, Facebook's distributed data store designed to efficiently manage the social graph's complex data relationships.

The article discusses Facebook's journey in scaling memcached since its initial implementation in 2005, highlighting the challenges faced due to the unique nature of the social graph and the high r...

The article discusses the development and implementation of Corona, a new scheduling framework at Facebook designed to improve the efficiency of MapReduce job scheduling.

The article discusses Facebook's approach to maintaining site reliability while rapidly deploying new features.

The article discusses a collaborative project between Facebook Engineering and a team of undergraduate students from Harvey Mudd College aimed at enhancing indexing capabilities in Apache Hive.

The article discusses the architecture and functionality of Facebook Messages' application server, detailing its integration with various services and the management of message operations.

The article discusses the architecture and infrastructure developed by Facebook to scale the Messages application back end, integrating various communication channels such as email, SMS, and Facebo...

The article discusses the potential of HTML5 as a gaming platform, highlighting the performance challenges and the introduction of JSGameBench, a tool designed to measure game performance metrics, ...

The article discusses join optimization techniques in Apache Hive, focusing on improving performance for join operations, which are critical for processing large datasets.

The article discusses the process of visualizing friendships within a social graph of 500 million people, focusing on geographical and political influences on friendships.

The article discusses Facebook's use of Apache Hadoop in its data warehouse, detailing three primary applications: web analytics, distributed database storage, and MySQL database backups.