How to Build a Distributed Inference Cache with NVIDIA Triton and Redis

Explore the benefits of the new Redis implementation of the Triton Caching API, including best practices for using Redis to supercharge your NVIDIA Triton…

NVIDIA logo
NVIDIA
•
Steve Lorello
•12 min read•advanced•
--
•View Original
CachingDockerGoogle CloudRedis

Overview

This article discusses how to build a distributed inference cache using NVIDIA Triton and Redis, highlighting the benefits and drawbacks of local versus distributed caching. It provides insights into setting up Redis with Triton to enhance performance and scalability for machine learning inference tasks.

What You'll Learn

1

How to implement distributed caching with Redis

2

Why to use Redis for distributed caching in NVIDIA Triton

3

When to choose local cache versus distributed cache

Prerequisites & Requirements

  • Basic understanding of caching concepts
  • Familiarity with NVIDIA Triton Inference Server and Redis(optional)

Key Questions Answered

What are the benefits of using Redis for caching in NVIDIA Triton?
Using Redis for caching in NVIDIA Triton allows for horizontal scaling, high availability, and resource sharing among multiple Triton instances. Redis is faster than local caching, with execution times typically in sub-milliseconds, making it a more efficient option for managing inference workloads.
How does local caching work in NVIDIA Triton?
Local caching in NVIDIA Triton uses an in-memory cache to store inference results. It involves hashing the input query to check for previously computed results, returning them if available, or performing the inference and caching the result if not. This method increases throughput but is limited to the process memory.
What are the drawbacks of using local cache in Triton?
The main drawbacks of local caching in Triton include cold start issues upon process restarts and resource contention since the cache is tied to the process memory. This limits horizontal scaling and can lead to duplicated caching across multiple Triton instances.
What is the speed comparison between Redis and local cache in Triton?
In testing, Redis achieved a throughput of 329 inference/sec with an average latency of 3,030 µs for the DenseNet model, while local caching provided 355 inference/sec at 2,817 µs. Although local cache was slightly faster, Redis offers significant advantages in scalability and resource management.

Key Statistics & Figures

DenseNet throughput without cache
80 inference/sec
Measured with an average latency of 12,680 µs.
DenseNet throughput with Redis
329 inference/sec
Achieved with an average latency of 3,030 µs.
DenseNet throughput with local cache
355 inference/sec
Resulted in an average latency of 2,817 µs.
Simple model throughput without cache
1,358 inference/sec
Measured with a latency of 735 µs.
Simple model throughput with Redis
1,639 inference/sec
Achieved with a latency of 608 µs.
Simple model throughput with local cache
2,753 inference/sec
Resulted in a latency of 363 µs.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Backend
Nvidia Triton Inference Server
Used for running inferences on tensors and implementing caching.
Database
Redis
Utilized as a distributed caching layer to enhance performance.

Key Actionable Insights

1
Implementing Redis as a distributed cache can significantly enhance the performance of your NVIDIA Triton instance.
By offloading caching to Redis, you can improve throughput and manage heavier workloads, allowing multiple Triton instances to share the same cache and reduce resource contention.
2
Minimize round-trip time by colocating your Redis and Triton servers.
Keeping Redis servers physically close to Triton servers can reduce latency and improve overall performance, especially in high-throughput applications.
3
Consider the computational expense of your queries when choosing between local and distributed caching.
For computationally expensive queries, distributed caching with Redis is likely to yield better performance, while simpler queries may benefit from local caching.

Common Pitfalls

1
Over-relying on local caching can lead to resource contention and limited scalability.
Since local caches are tied to the process memory, they cannot be shared across instances, leading to duplicated caching and inefficient resource usage.

Related Concepts

Caching Strategies In Distributed Systems
Performance Optimization Techniques For Machine Learning
Scalability Challenges In AI/ML Applications