How Google Uses JAX

HeyGen ported their 18B+ parameter Avatar IV video generation model to Google Cloud's Trillium (v6e) TPUs via torchax and XLA, utilizing FSDP and Ulysses sequence parallelism across an eight-chip mesh. To achieve a 1.86x speedup for real-time streaming, the engineering team pipelined exposed all-to-all collectives, aligned sparse attention block sizes to eliminate mask padding, and bypassed softmax serial dependencies using a precomputed Cauchy-Schwarz upper bound. These custom Pallas kernel and compiler optimizations were deployed only after passing rigorous two-tier quality gates to guarantee byte-identical or mathematically equivalent pixel outputs.

Google's open-source TPU microbenchmark suite provides developers with granular performance metrics across Network, Compute, HBM, Host Transfer, and Attention components to validate real-world hardware capabilities. By leveraging these benchmarks to establish a Roofline model, engineers can accurately diagnose whether their machine learning workloads are compute-, memory-, or network-bound. This empirical baseline directly guides targeted software optimizations—such as kernel tuning, mesh sharding, and rematerialization—to maximize hardware utilization for large-scale model deployments.

This second installment explores how Ray’s higher-level libraries—Serve, Data, and Train—abstract the complexities of running AI workloads on Google's TPU slices. Ray Serve uses a simple topology configuration to correctly gang-schedule large multi-host models, while Ray Data eliminates data-loading bottlenecks by feeding accelerators directly with native JAX batches. Finally, JaxTrainer streamlines distributed training across TPUs by automatically handling cross-slice coordination, checkpointing, and fault tolerance.

Tunix is Google’s new JAX-native post-training library designed to eliminate TPU idling bottlenecks when training multi-turn, tool-using LLM reasoning agents. It maximizes hardware throughput by combining highly concurrent, asynchronous rollouts with a decoupled producer-consumer pipeline, ensuring the trainer is constantly fed even while agents wait on network I/O or environment steps. Additionally, Tunix provides plug-and-play abstractions and continuous macro-level profiling, allowing developers to easily integrate custom open-source environments and optimize complex distributed workflows without massive code rewrites.

Distributed AI training is notoriously fragile because losing a single machine typically crashes the entire multi-node job, forcing a time-consuming, full-workload infrastructure restart. To address this, Google’s JAX ecosystem utilizes elastic training via Pathways, which converts a hardware failure into a catchable Python exception so the running process can survive. When an unplanned failure occurs, the system automatically replaces only the broken worker, restores the last viable checkpoint from Cloud Storage, and resumes training in place—minimizing total downtime to under two minutes without ever restarting the main controller process.

DiffusionGemma is an experimental text-generation model built on the Gemma 4 architecture that uses diffusion-based parallel generation instead of token-by-token autoregression, enabling much faster inference, bidirectional context awareness, and real-time self-correction while remaining deployable on consumer GPUs. Its architecture generates and refines 256-token blocks in parallel through iterative denoising, allowing it to handle complex constraint-based tasks such as Sudoku more effectively than traditional language models and demonstrating strong gains from fine-tuning. The model integrates with vLLM and other popular inference frameworks, giving developers access to a new non-autoregressive approach that combines high performance, efficient long-context scaling, and straightforward customization and deployment.

Researchers at UCSD have successfully implemented DFlash, a block-diffusion speculative decoding method, on Google TPUs to bypass the sequential bottlenecks of traditional autoregressive drafting. By "painting" entire blocks of candidate tokens in a single forward pass rather than predicting them one-by-one, the system achieved average speedups of 3.13x, with peak performance nearly doubling that of existing methods like EAGLE-3. This open-source integration into the vLLM ecosystem optimizes TPU hardware by leveraging "free" parallel verification and high-quality draft predictions for complex reasoning tasks.

MaxText has introduced new support for Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on single-host TPU configurations, leveraging JAX and the Tunix library for high-performance model refinement. These features enable developers to easily adapt pre-trained models for specialized tasks and complex reasoning using efficient algorithms like GRPO and GSPO. This update streamlines the post-training workflow, offering a scalable path from single-host setups to larger multi-host configurations.

TorchTPU is a new engineering stack designed to provide a native, high-performance experience for running PyTorch workloads on Google’s TPU infrastructure with minimal code changes. It features an "Eager First" approach with multiple execution modes and utilizes the XLA compiler to optimize distributed training across massive clusters. Moving into 2026, the project aims to further reduce compilation overhead and expand support for dynamic shapes and custom kernels to ensure seamless scalability for the next generation of AI.

Google has introduced Finish Changes and Outlines for Gemini Code Assist in IntelliJ and VS Code to reduce developer friction and eliminate the need for long, manual prompting. Finish Changes acts as an AI pair programmer that completes code, implements pseudocode, and applies refactoring patterns by observing your current edits and context. Meanwhile, Outlines improves code comprehension by generating interactive, high-level English summaries interleaved directly within the source code to help engineers navigate and understand complex files.

Google has officially launched LiteRT, the successor to TFLite, which offers significantly faster GPU and NPU acceleration alongside seamless support for PyTorch and JAX. The update also introduces lower-precision data type support for increased efficiency and a commitment to more frequent security and dependency updates across the TensorFlow ecosystem. This transition solidifies LiteRT as Google's primary high-performance framework for deploying GenAI and advanced on-device inference.

This tutorial demonstrates how to fine-tune FunctionGemma, a small language model for translating natural language into API calls, using Google's Tunix library on TPUs.

LiteRT has evolved from its TensorFlow Lite foundation into a universal on-device AI inference framework, now offering production-ready GPU acceleration across six platforms and streamlined NPU int...

This article demonstrates how to fine-tune FunctionGemma, a specialized 270M parameter Gemma 3 model designed for function calling in agentic AI systems.

This article serves as a practical guide for developers working with JAX on Cloud TPUs, focusing on essential tools and techniques for debugging and profiling machine learning workflows.

The article introduces A2UI, an open-source project designed for agent-driven interfaces that allows agents to generate contextually relevant user interfaces.

The article discusses the advancements in on-device AI powered by MediaTek's Neural Processing Unit (NPU) and the introduction of the LiteRT NeuroPilot Accelerator.

The article discusses optimizing performance on Qualcomm's Neural Processing Unit (NPU) using LiteRT, Google's high-performance on-device ML framework.

The article discusses the JAX AI Stack, a modular framework for building production AI models on Google Cloud TPUs.

Metrax is a high-performance library designed for efficient and robust model evaluation metrics in JAX, addressing the need for standardized metrics during the migration from TensorFlow.

Agent Garden is a platform designed to facilitate the development and deployment of AI agents, making it accessible to all users, not just those on Google Cloud.

The article introduces Coral NPU, a full-stack, open-source platform designed to enhance Edge AI capabilities on low-power devices.

The article discusses building high-performance data pipelines using Grain, a data loading library for JAX, and ArrayRecord, an efficient file format.

The article introduces Tunix, a new open-source, JAX-native library designed for post-training of large language models (LLMs).

The article discusses how JAX, a popular framework for AI model development, is being increasingly adopted in scientific computing, particularly for solving complex Partial Differential Equations (...

The article introduces Gemma 3 270M, a compact AI model designed for hyper-efficient task-specific fine-tuning.

This article provides a comprehensive guide on how to train a GPT-2 model using JAX on TPU, highlighting the ease of leveraging Google TPUs for free.

The article discusses the increasing adoption of JAX in robotics, highlighting its efficiency in optimal control and simulation. It features insights from Max Muchen Sun, a Robotics Ph. D.

Stanford's Marin project introduces the first fully open foundation model developed using JAX, emphasizing transparency in the scientific process behind AI models.

The article discusses how to use KerasHub for loading model weights from SafeTensors into Keras, enabling flexible end-to-end machine learning workflows across different frameworks like JAX, PyTorc...

The article introduces Keras Recommenders, a new library designed to simplify the creation of state-of-the-art recommendation systems using Keras with JAX, TensorFlow, or PyTorch.

Gemma 3 is the latest version of the Gemma open-model family, boasting enhanced capabilities such as multimodality, longer context windows, and improved reasoning.

The article discusses the launch of ShieldGemma 2, a safety content classifier model built on Gemma 3, aimed at detecting harmful content in both synthetic and natural images.

PaliGemma 2 mix is an advanced vision-language model designed for multiple tasks, allowing developers to utilize a single model for various applications such as image captioning, object detection, ...

PaliGemma 2 is the latest vision-language model from Google, designed to simplify the process of building advanced AI that can interpret visual inputs.

The article announces that Francois Chollet, the creator of Keras, is leaving Google to pursue new opportunities.

The article introduces Keras Hub, a unified library for pretrained models that simplifies access to both natural language processing (NLP) and computer vision (CV) architectures.

The article discusses the PaliGemma architecture, a lightweight open vision-language model (VLM) inspired by PaLI-3.

LiteRT, formerly known as TensorFlow Lite, is a high-performance runtime for on-device AI that now supports models from multiple frameworks including PyTorch, JAX, and Keras.

The article discusses the release of Gemma 2, a new suite of open models that sets a new standard for performance and accessibility in conversational AI.

The article discusses the advancements in responsible AI through the introduction of Gemma 2, which includes models with 27 billion and 9 billion parameters, emphasizing safety and accessibility.

The article discusses the release of the Gemma 2 model with 27 billion parameters, highlighting its capabilities in Keras and integration with JAX for efficient model training.

Model Explorer is a powerful graph visualization tool designed to simplify the development and optimization of machine learning models for edge devices.

The article recaps the Google I/O 2024 event, highlighting advancements in AI technologies aimed at making AI accessible for developers.

The article introduces PaliGemma, an open vision-language model, along with Gemma 2, the next generation of the Gemma models, and updates to the Responsible AI Toolkit.

This article discusses how to publish Keras models on Kaggle and Hugging Face, highlighting the ease of sharing fine-tuned models with the community.

The article reveals the program lineup for Google I/O, highlighting key sessions and events for developers.

The article introduces the expansion of the Gemma family with two new models, CodeGemma and RecurrentGemma, designed specifically for developers and researchers.

The article introduces Gemma models in Keras, a family of lightweight, state-of-the-art open models that leverage the same technology as the Gemini models.

The Women in Machine Learning Symposium 2023 is set to take place on December 7, 2023, featuring a diverse lineup of female presenters who will discuss advancements in AI/ML and their practical app...