Unlocking Peak Performance on Qualcomm NPU with LiteRT

Lu Wang, Weiyi Wang, Andrew Zhang

LiteRT's new Qualcomm AI Engine Direct (QNN) Accelerator unlocks dedicated NPU power for on-device GenAI on Android. It offers a unified mobile deployment workflow, SOTA performance (up to 100x speedup over CPU), and full model delegation. This enables smooth, real-time AI experiences, with FastVLM-0.5B achieving over 11,000 tokens/sec prefill on Snapdragon 8 Elite Gen 5 NPU.

Google

•

Lu Wang, Weiyi Wang, Andrew Zhang

•9 min read•intermediate•

--

•View Original

JAXKotlinShell

Overview

The article discusses optimizing performance on Qualcomm's Neural Processing Unit (NPU) using LiteRT, Google's high-performance on-device ML framework. It highlights the advantages of leveraging NPUs for AI tasks, the introduction of the LiteRT Qualcomm AI Engine Direct (QNN) Accelerator, and provides insights into deployment and performance benchmarks.

What You'll Learn

1

How to leverage NPU for on-device AI applications

2

Why using LiteRT simplifies mobile deployment workflows

3

How to achieve state-of-the-art performance with LiteRT QNN Accelerator

Prerequisites & Requirements

Understanding of AI/ML concepts and mobile application development
Familiarity with LiteRT and Qualcomm AI Hub(optional)

Key Questions Answered

How does the LiteRT Qualcomm AI Engine Direct Accelerator improve performance?

The LiteRT Qualcomm AI Engine Direct Accelerator enhances performance by providing a unified and simplified mobile deployment workflow, enabling developers to utilize the NPU's capabilities without dealing with low-level SDKs. It supports an extensive range of LiteRT operations, allowing for full model delegation to the NPU, resulting in significant speedups over CPU and GPU.

What performance improvements can be expected when using NPU acceleration?

NPU acceleration can provide up to a 100x speedup over CPU and a 10x speedup over GPU. On the Snapdragon 8 Elite Gen 5, over 56 models can run in under 5ms with the NPU, compared to only 13 models achieving that on the CPU, enabling smoother real-time AI experiences.

What are the steps to deploy a .tflite model on NPU using LiteRT?

To deploy a .tflite model on NPU using LiteRT, first compile the model for the target SoCs, then export the compiled models into a Google Play AI Pack. Finally, integrate the model into your Android app project and configure it to use the NPU for inference.

Key Statistics & Figures

Speedup over CPU

100x

NPU acceleration provides this speedup when compared to CPU performance.

Speedup over GPU

10x

This speedup is achieved through the use of NPU acceleration.

Models running under 5ms

56 models

This performance is observed on the Snapdragon 8 Elite Gen 5 with NPU acceleration.

Technologies & Tools

Framework

Litert

Used for high-performance on-device ML applications.

Hardware

Qualcomm Snapdragon 8 Elite Gen 5

Provides the NPU capabilities for enhanced AI processing.

Key Actionable Insights

1
Utilizing the NPU for AI tasks can significantly enhance the performance of mobile applications, especially for compute-intensive tasks like text-to-image generation.
This is crucial for applications that require real-time processing, as it allows for smoother user experiences without frame drops.

2
The LiteRT framework simplifies the deployment process across various Qualcomm SoCs, reducing the complexity for developers.
By abstracting away vendor-specific SDKs and targeting multiple SoCs simultaneously, developers can save time and effort in the deployment phase.

3
Implementing ahead-of-time (AOT) compilation is recommended for larger models to optimize initialization times and memory usage.
This approach can lead to better performance and resource management, especially on devices with limited capabilities.

Common Pitfalls

1

Neglecting to use AOT compilation for large models can lead to longer initialization times and higher memory consumption.

This can negatively impact the user experience, especially on devices with limited resources, making it essential to consider AOT for larger models.

2

Failing to properly configure the AI Pack in the gradle settings may result in deployment issues.

This can prevent the application from accessing the compiled models correctly, leading to runtime errors.

Related Concepts

Mobile AI Application Development

Npu Architecture And Capabilities

Performance Optimization Techniques For AI Models

Google's open-source TPU microbenchmark suite provides developers with granular performance metrics across Network, Compute, HBM, Host Transfer, and Attention components to validate real-world hardware capabilities. By leveraging these benchmarks to establish a Roofline model, engineers can accurately diagnose whether their machine learning workloads are compute-, memory-, or network-bound. This empirical baseline directly guides targeted software optimizations—such as kernel tuning, mesh sharding, and rematerialization—to maximize hardware utilization for large-scale model deployments.

ShellJAXengineering

6 min read

Includes Code

--

Google

Intermediate

MaxText Expands Post-Training Capabilities: Introducing SFT and RL on Single-Host TPUs

MaxText has introduced new support for Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on single-host TPU configurations, leveraging JAX and the Tunix library for high-performance model refinement. These features enable developers to easily adapt pre-trained models for specialized tasks and complex reasoning using efficient algorithms like GRPO and GSPO. This update streamlines the post-training workflow, offering a scalable path from single-host setups to larger multi-host configurations.

ShellJAXHugging Face

3 min read

Includes Code

--

Google

Intermediate

DiffusionGemma: The Developer Guide

DiffusionGemma is an experimental text-generation model built on the Gemma 4 architecture that uses diffusion-based parallel generation instead of token-by-token autoregression, enabling much faster inference, bidirectional context awareness, and real-time self-correction while remaining deployable on consumer GPUs. Its architecture generates and refines 256-token blocks in parallel through iterative denoising, allowing it to handle complex constraint-based tasks such as Sudoku more effectively than traditional language models and demonstrating strong gains from fine-tuning. The model integrates with vLLM and other popular inference frameworks, giving developers access to a new non-autoregressive approach that combines high performance, efficient long-context scaling, and straightforward customization and deployment.

ShellGoogle CloudApache

6 min read

Includes Code

--

These articles from Google and other leading engineering teams share similar topics with "Unlocking Peak Performance on Qualcomm NPU with LiteRT". Explore more engineering insights on Shell, JAX, Google Cloud.