Bringing Generative AI to Life with NVIDIA Jetson

Recently, NVIDIA unveiled Jetson Generative AI Lab, which empowers developers to explore the limitless possibilities of generative AI in a real-world setting…

NVIDIA logo
NVIDIA
•
Chitoku Yato
•9 min read•intermediate•
--
•View Original
CLIPGenerative AIGitHub ActionsGPTGPT-4GradioHugging FaceModalOobaboogaRLHFSegment Anything ModelStable DiffusionTransformers

Overview

NVIDIA has introduced the Jetson Generative AI Lab, enabling developers to leverage generative AI capabilities on Jetson edge devices. The lab supports running large language models (LLMs), vision transformers, and diffusion models locally, including the Llama-2-70B model on Jetson AGX Orin at interactive rates.

What You'll Learn

1

How to run the stable-diffusion-webui on Jetson devices

2

How to implement text-generation-webui for local LLMs

3

How to utilize llamaspeak for voice conversations with LLMs

4

How to optimize models using NVIDIA TensorRT for real-time performance

Prerequisites & Requirements

  • Basic understanding of generative AI concepts
  • Familiarity with Git and Docker(optional)

Key Questions Answered

What generative AI applications can be run on NVIDIA Jetson devices?
The article details several applications including stable-diffusion-webui for image generation, text-generation-webui for local LLM interactions, and llamaspeak for voice conversations. Each application leverages the capabilities of Jetson devices to run advanced AI models locally.
How does the Jetson Generative AI Lab support developers?
The Jetson Generative AI Lab provides tutorials, resources, and prebuilt containers, enabling developers to quickly test and deploy generative AI models on Jetson devices. This support helps in exploring real-world applications of generative AI.
What is the performance of the Llama-2-70B model on Jetson AGX Orin?
The Llama-2-70B model can run on Jetson AGX Orin at interactive rates, showcasing the device's capability to handle large language models locally without relying on cloud infrastructure.
What is NanoOWL and how does it enhance object detection?
NanoOWL is a project that optimizes the Open World Localization with Vision Transformers (OWL-ViT) model for real-time performance on Jetson platforms, allowing for text-prompted object detection at high frame rates.

Key Statistics & Figures

Llama-2-70B model performance
interactive rates
This performance is achievable on Jetson AGX Orin, indicating the device's capability to handle large models locally.
NanoOWL encoding speed
~95FPS
This speed is achieved on Jetson AGX Orin, allowing for real-time object detection.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Hardware
Nvidia Jetson
Used for running generative AI models locally.
Software
Tensorrt
Optimizes AI models for real-time performance on Jetson devices.
Tools
Github Actions
Used for continuous integration and deployment of jetson-containers.

Key Actionable Insights

1
Leverage the Jetson Generative AI Lab to experiment with various generative AI models locally.
This allows developers to test and iterate on their applications without the latency and bandwidth issues associated with cloud computing.
2
Utilize the jetson-containers project to simplify the deployment of AI models on Jetson devices.
This open-source project automates the containerization process, making it easier for developers to focus on building applications rather than managing dependencies.
3
Explore the potential of multimodal AI applications using the capabilities of Jetson devices.
With models like Llama-2 and NanoOWL, developers can create applications that integrate text and visual data, enhancing user interactions.

Common Pitfalls

1
Failing to optimize models for the specific hardware can lead to suboptimal performance.
Developers should ensure they use tools like TensorRT to maximize the efficiency of their AI models on Jetson devices.

Related Concepts

Generative AI
Large Language Models
Vision Transformers
Real-time Object Detection