LiteRT: The Universal Framework for On-Device AI

Key Actionable Insights

1

Migrate from TFLite GPU delegate to LiteRT's new CompiledModel API to achieve 1.4x average GPU performance improvement. The new API provides a modern interface that unlocks full GPU and NPU acceleration potential, while the legacy interpreter API remains available for existing production models.

This is especially impactful for latency-sensitive applications like real-time segmentation and speech recognition, where the combination of asynchronous execution and zero-copy buffer interoperability can yield up to 2x speedups.

2

Use AOT compilation for production deployments targeting known device SoCs to minimize initialization time and memory footprint. Pre-compile your .tflite models for specific target NPUs using the LiteRT Python library, then leverage Google Play for On-device AI (PODAI) for automatic delivery of the model and runtime.

AOT compilation is particularly important for complex models where first-run latency matters. For simpler models distributed across many device types, on-device JIT compilation may be more practical despite higher first-run costs.

3

Leverage LiteRT's NPU acceleration for compute-intensive GenAI workloads to unlock up to 100x faster performance than CPU. MediaTek and Qualcomm NPU integrations are now production-ready, with LiteRT handling automatic delegation and providing robust fallback to GPU or CPU when NPU is unavailable.

This is critical for deploying large language models like Gemma on-device, where NPU acceleration provides an additional 2x gain over GPU for compute-bound prefill operations.

4

Use the LiteRT Torch library to convert PyTorch models directly to .tflite format in a single step, eliminating complex intermediate translation workflows. This enables PyTorch-based architectures to immediately benefit from LiteRT's advanced hardware acceleration across CPU, GPU, and NPU backends.

This is valuable for teams training models in PyTorch who want to deploy to edge devices without rewriting models in TensorFlow, enabling higher research-to-production velocity.

5

Implement zero-copy buffer interoperability when building GPU-accelerated pipelines to eliminate unnecessary CPU overhead. Use TensorBuffer::CreateFromGlBuffer to wrap existing OpenGL buffers directly, and access outputs as AHardwareBuffers for efficient downstream processing.

This optimization is demonstrated in the LiteRT Segmentation sample app and is essential for real-time use cases like background segmentation and automatic speech recognition where end-to-end latency is critical.

6

Evaluate LiteRT as a replacement for llama.cpp when deploying GenAI models on mobile devices, given the substantial performance advantages of 3x on CPU and up to 19x on GPU. The LiteRT-LM orchestration layer handles LLM-specific complexities and is the same infrastructure powering Gemini Nano in Google products.

Benchmarked with Gemma 3 1B on Samsung Galaxy S25 Ultra, these gains are particularly significant for prefill (compute-bound) operations where GPU acceleration shows the largest improvement.