Google logo

How Google Uses Transformers

27 engineering articles about Transformers from Google's engineering team

Articles

Filter:
Google logo
Google
Advanced
Google AI Edge Model Explorer now supports custom compiler intermediate representations and open-source adapters like TOSA, VGF, and PTE for on-device machine learning. Developers can leverage hierarchical graph visualization, split-pane views, and per-node trace overlays to catch export regressions and verify hardware delegation. These tools enable engineering teams to transparently debug execution graphs and pinpoint cycle-level latency bottlenecks locally.
Eric Yang, Alan Ma
4 min read
Includes Code
--
Google logo
Google
Advanced
TorchTPU integrates a native torch.device("tpu") backend directly into PyTorch, allowing developers to train large-scale models on Google Cloud TPUs without rewriting code in JAX. The TorchTitan TPU fork enhances this with production-grade scaling primitives like SimpleFSDP, Graph Trainer, and custom Pallas kernels that achieve high Model Flops Utilization (MFU) across dense and MoE architectures. Developers can leverage these battle-tested optimization playbooks, recipes, and the Hugging Face modeling backend to efficiently scale their own workloads on TPU infrastructure.
Aleksey Vlasenko, Rob Torres, Will Constable
22 min read
Includes Code
--
Google logo
Google
Intermediate
EmbeddingGemma 2 is a compact, open-source multimodal embedding model that maps text, code, images, video, and audio into a unified 768-dimensional space. Developers can use the sentence-transformers library to selectively load modular modality encoders—ranging from 270M to 740M parameters—to optimize memory usage. Additionally, Matryoshka Representation Learning enables dynamic dimension truncation down to 128d, significantly reducing vector database storage requirements while maintaining high retrieval performance.
Maarten Grootendorst, Ian Ballantyne
7 min read
Includes Code
--
Google logo
Google
Intermediate
DiffusionGemma is an experimental text-generation model built on the Gemma 4 architecture that uses diffusion-based parallel generation instead of token-by-token autoregression, enabling much faster inference, bidirectional context awareness, and real-time self-correction while remaining deployable on consumer GPUs. Its architecture generates and refines 256-token blocks in parallel through iterative denoising, allowing it to handle complex constraint-based tasks such as Sudoku more effectively than traditional language models and demonstrating strong gains from fine-tuning. The model integrates with vLLM and other popular inference frameworks, giving developers access to a new non-autoregressive approach that combines high performance, efficient long-context scaling, and straightforward customization and deployment.
Ian Ballantyne, Omar Sanseviero
6 min read
Includes Code
--
Google logo
Google
Advanced
The newly released Gemma 4 12B is a dense, multimodal model designed for high-performance local AI execution on consumer devices. By introducing a novel, encoder-free architecture, it bypasses traditional visual and audio encoders to feed multimodal data directly into the LLM backbone.
André Susano Pinto, Andreas Steiner, Karolis Misiunas, Karsten Roth, Michael Tschannen, Omar Sanseviero
6 min read
Includes Code
--
Google logo
Google
Intermediate
Integration of Arm Scalable Matrix Extension 2 (SME2) and the Google AI Edge software stack enables high-performance, on-device generative AI by turning the CPU into a powerful matrix-compute accelerator. Using Stability AI’s "stable-audio-open-small" model as a case study, it outlines a streamlined "Convert, Optimize, and Deploy" pipeline that utilizes LiteRT, XNNPACK, and KleidiAI to automate hardware acceleration. The resulting implementation achieves over a 2x speedup in audio generation and a 4x reduction in memory usage while maintaining high audio quality on Arm-powered mobile devices and laptops.
Chintan Parikh, Dillon Sharlet, Na Li, Gian Marco Iodice
8 min read
Includes Code
--
Google logo
Google
Intermediate
The article discusses how to fine-tune the Gemma 3 270M model for on-device applications, enabling developers to create custom AI models without the need for expensive hardware.
Ian Ballantyne, Jason Mayes
5 min read
Includes Code
Has Summary
--
Google logo
Google
Intermediate
The article provides an in-depth exploration of the EmbeddingGemma architecture, detailing its origins, embedding generation process, and the comprehensive training methodology.
Henrique Schechter Vera, Juyeong Ji, Sahil Dua
7 min read
Includes Code
Has Summary
--
Google logo
Google
Intermediate
EmbeddingGemma is an innovative open embedding model designed for on-device AI applications, featuring 308 million parameters for efficient performance.
Google logo
Google
Intermediate
The article introduces Gemma 3 270M, a compact AI model designed for hyper-efficient task-specific fine-tuning.
Olivier Lacombe, Kathleen Kenealy, Kat Black, Ravin Kumar, Francesco Visin, Jiageng Zhang
5 min read
Has Summary
--
Google logo
Google
Intermediate
The article introduces Gemma 3n, a mobile-first architecture designed for on-device AI, highlighting its multimodal capabilities and architectural innovations.
Omar Sanseviero, Ian Ballantyne
9 min read
Includes Code
Has Summary
--
Google logo
Google
Intermediate
The article discusses the new features and improvements in Gemma 3, highlighting its vision-language capabilities, architectural changes for memory efficiency, and enhanced multilingual support.
Ju-yeong Ji, Ravin Kumar
9 min read
Includes Code
Has Summary
--
Google logo
Google
Intermediate
Gemma 3 is the latest version of the Gemma open-model family, boasting enhanced capabilities such as multimodality, longer context windows, and improved reasoning.
Omar Sanseviero, Philipp Schmid
5 min read
Includes Code
Has Summary
--
Google logo
Google
Beginner
The article discusses the launch of ShieldGemma 2, a safety content classifier model built on Gemma 3, aimed at detecting harmful content in both synthetic and natural images.
Dana Kurniawan, Wenjun Zeng, Ryan Mullins
3 min read
Has Summary
--
Google logo
Google
Beginner
PaliGemma 2 mix is an advanced vision-language model designed for multiple tasks, allowing developers to utilize a single model for various applications such as image captioning, object detection, ...
Omar Sanseviero, Andreas Steiner
3 min read
Includes Code
Has Summary
--
Google logo
Google
Intermediate
PaliGemma 2 is the latest vision-language model from Google, designed to simplify the process of building advanced AI that can interpret visual inputs.
Daniel Keysers, Andreas Steiner
3 min read
Has Summary
--
Google logo
Google
Advanced
The Web AI Summit 2024, hosted by Google on October 18, 2024, focused on client-side AI for developers, showcasing how machine learning models can operate offline in web browsers.
Google logo
Google
Intermediate
The article discusses the expansion of the Responsible Generative AI Toolkit, introducing new tools designed for various large language models (LLMs) like Gemma and Gemini.
Google logo
Google
Advanced
The article explores the RecurrentGemma architecture, a hybrid model that combines gated linear recurrences with local sliding window attention, enhancing performance for long context prompts.
Ju-yeong Ji, Ravin Kumar
6 min read
Includes Code
Has Summary
--
Google logo
Google
Intermediate
The article provides an overview of the Gemma model family architectures, detailing its lightweight, state-of-the-art open models derived from Gemini research.
Ju-yeong Ji, Ravin Kumar
9 min read
Includes Code
Has Summary
--
Google logo
Google
Advanced
This article provides a comprehensive guide on using Gemma with Ray on Vertex AI, detailing the steps to set up, fine-tune, and deploy machine learning models.
Google logo
Google
Intermediate
The article discusses the release of the Gemma 2 model with 27 billion parameters, highlighting its capabilities in Keras and integration with JAX for efficient model training.
Martin Görner
5 min read
Includes Code
Has Summary
--
Google logo
Google
Intermediate
The article introduces PaliGemma, an open vision-language model, along with Gemma 2, the next generation of the Gemma models, and updates to the Responsible AI Toolkit.
Tris Warkentin, Xiaohua Zhai, Ludovic Peran
4 min read
Has Summary
--
Google logo
Google
Intermediate
This article discusses how to publish Keras models on Kaggle and Hugging Face, highlighting the ease of sharing fine-tuned models with the community.
Google logo
Google
Intermediate
The article introduces the expansion of the Gemma family with two new models, CodeGemma and RecurrentGemma, designed specifically for developers and researchers.
Google logo
Google
Intermediate
The article highlights the achievements and activities of Google Machine Learning communities in the second quarter of 2023, showcasing various training campaigns, community events, and innovative ...
Google logo
Google
Intermediate
The article highlights the achievements of Machine Learning Google Developer Experts (GDEs) in Q2 2021, showcasing their contributions to the global ML ecosystem through various events, projects, a...

You've reached the end! All 27 articles loaded.