Getting Started with Edge AI on NVIDIA Jetson: LLMs, VLMs, and Foundation Models for Robotics

Running advanced AI and computer vision workloads on small, power-efficient devices at the edge is a growing challenge. Robots, smart cameras…

NVIDIA logo
NVIDIA
•
Chitoku Yato
•9 min read•intermediate•
--
•View Original
Hugging FaceOllamaWebRTC

Overview

The article discusses the implementation of Edge AI on the NVIDIA Jetson platform, focusing on the use of Large Language Models (LLMs), Vision Language Models (VLMs), and Foundation Models in robotics. It provides tutorials for deploying these models locally on Jetson devices, emphasizing privacy, low latency, and the transition from simulation to real-world applications.

What You'll Learn

1

How to run LLMs and VLMs locally on NVIDIA Jetson devices

2

Why using local models enhances privacy and reduces latency

3

How to deploy robotics policies using NVIDIA Isaac Sim

4

When to choose between different Jetson models based on project needs

Prerequisites & Requirements

  • Basic understanding of AI and robotics concepts
  • Familiarity with Docker and command line interfaces(optional)

Key Questions Answered

What are the advantages of running LLMs and VLMs locally on Jetson?
Running LLMs and VLMs locally on Jetson devices provides complete privacy since data does not leave the device, and it eliminates network latency, resulting in instantaneous interactions. This setup is ideal for applications requiring real-time intelligence without cloud dependency.
How can I deploy robotics policies using NVIDIA Isaac Sim?
You can generate synthetic training data and validate policies in a physics-accurate virtual environment using NVIDIA Isaac Sim. This allows for hardware-in-the-loop testing, enabling you to ensure that your robot's control policy works effectively before deploying it to physical hardware.
Which Jetson model should I choose for my project?
Choose the Jetson Orin Nano Super (8GB) for small LLMs or VLMs and early-stage robotics. The Jetson AGX Orin (64GB) is suitable for more complex models and personal pipelines, while the Jetson AGX Thor (128GB) is ideal for large models and real-time applications requiring high performance.
What VLMs can I run on different Jetson models?
The Jetson Orin Nano 8GB can run VLMs and LLMs up to nearly 4B parameters, while the Jetson AGX Orin 64GB supports models in the 4B–20B range. The Jetson AGX Thor 128GB is designed for workloads involving models from about 20B up to around 120B parameters.

Key Statistics & Figures

Memory capacity of Jetson AGX Orin
64GB
This model supports larger, more complex AI workloads, making it suitable for deep reasoning tasks.
Memory capacity of Jetson AGX Thor
128GB
This model enables running massive 100B+ parameter models, providing data center-class intelligence at the edge.
Token generation speed of gpt-oss-20b
40 tokens/sec
This speed was achieved during inference on NVIDIA Jetson AGX Orin using vLLM.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Hardware
Nvidia Jetson
Used for running AI and computer vision workloads at the edge.
Software
Docker
Utilized for containerizing applications and managing dependencies.
Software
Nvidia Isaac Sim
Provides a simulation environment for training robotics policies.

Key Actionable Insights

1
Utilize local LLMs and VLMs on Jetson devices to enhance privacy and performance.
By running models locally, you maintain control over your data and reduce latency, making it suitable for applications like personal assistants and autonomous robots.
2
Leverage NVIDIA Isaac Sim for training robotics policies in a simulated environment.
This approach allows you to gather necessary interaction data without the costs and risks associated with physical trials, enabling efficient development cycles.
3
Select the appropriate Jetson model based on your project requirements.
Understanding the memory and processing capabilities of each model helps in making informed decisions that align with your project's complexity and performance needs.

Common Pitfalls

1
Failing to choose the right Jetson model for your project can lead to performance issues.
Selecting a model without understanding its memory and processing capabilities may result in inadequate performance for your specific use case, leading to frustration and wasted resources.

Related Concepts

Edge AI
Local AI Deployment
Robotics Simulation
Imitation Learning