On-device GenAI in Chrome, Chromebook Plus, and Pixel Watch with LiteRT-LM

Google AI Edge provides the tools to run AI features on-device, and its new LiteRT-LM runtime is a significant leap forward for generative AI. LiteRT-LM is an open-source C++ API, cross-platform compatibility, and hardware acceleration designed to efficiently run large language models like Gemma and Gemini Nano across a vast range of hardware. Its key innovation is a flexible, modular architecture that can scale to power complex, multi-task features in Chrome and Chromebook Plus, while also being lean enough for resource-constrained devices like the Pixel Watch. This versatility is already enabling a new wave of on-device generative AI, bringing capabilities like WebAI and smart replies to users.

Google logo
Google
•
Yu-hui Chen, Ram Iyengar
•9 min read•advanced•
--
•View Original
ChiGeminiKotlinSwiftTransformer

Overview

The article discusses the deployment of on-device generative AI (GenAI) using LiteRT-LM in Chrome, Chromebook Plus, and Pixel Watch. It highlights the advantages of running large language models (LLMs) directly on user devices, including offline availability and cost efficiency, while addressing the technical challenges involved in achieving high performance and low latency.

What You'll Learn

1

How to leverage LiteRT-LM to build custom AI pipelines for on-device applications

2

Why using shared foundation models can optimize resource usage in LLM deployments

3

When to implement lightweight LoRAs for feature-specific customization in LLMs

Prerequisites & Requirements

  • Understanding of large language models and AI concepts
  • Familiarity with C++ programming language

Key Questions Answered

What is LiteRT-LM and how does it function?
LiteRT-LM is a production-tested inference framework designed for running large language models like Gemini Nano on edge devices. It provides an easy-to-integrate API and reusable modules, allowing developers to create customized LLM pipelines tailored to their product requirements.
How does LiteRT-LM optimize resource usage on devices like Pixel Watch?
LiteRT-LM allows developers to build custom pipelines by selecting only the necessary components for specific tasks. This modular approach minimizes binary size and memory usage, making it suitable for resource-constrained devices like the Pixel Watch.
What are the key features of LiteRT-LM?
Key features of LiteRT-LM include cross-platform deployment, hardware acceleration support for CPU, GPU, and NPU, and enhanced flexibility through its modular design, which allows for customized inference pipelines and multi-modality support.
What architectural patterns does LiteRT-LM use for managing LLM features?
LiteRT-LM employs an Engine and Session architecture where the Engine manages shared resources and Sessions handle individual tasks. This separation allows multiple features to utilize a single foundation model efficiently while maintaining their unique states.

Technologies & Tools

Framework
Litert-lm
Used for running large language models on edge devices.
API
Mediapipe Llm Inference API
Provides high-level access to LLM functionalities for developers.

Key Actionable Insights

1
Utilize LiteRT-LM to create tailored AI solutions that meet specific application needs.
By leveraging LiteRT-LM's modular architecture, developers can optimize their applications for performance and resource efficiency, especially on devices with varying capabilities.
2
Implement lightweight LoRAs to enhance the functionality of shared foundation models.
This approach allows for feature-specific customization without the overhead of deploying multiple large models, making it a practical solution for applications requiring diverse AI capabilities.
3
Explore the MediaPipe LLM Inference API for high-level access to LLM functionalities.
This API simplifies the integration of AI features into applications, enabling developers to quickly implement powerful AI capabilities without delving into low-level details.

Common Pitfalls

1
Failing to optimize model size and memory usage for low-compute devices can lead to performance issues.
Developers need to be mindful of the resource constraints of devices like the Pixel Watch and should tailor their implementations to avoid bloated binaries and excessive memory consumption.

Related Concepts

Large Language Models
On-device AI
Modular AI Frameworks