Skip to main content

Edge AI on NVIDIA Jetson

This section guides you through running local AI models at the edge on NVIDIA Jetson devices managed by Palette. Palette manages each Jetson device as an Edge host. You define the Kubernetes distribution and network in a cluster profile, deploy it the same way as the rest of your Edge fleet, and then run the AI serving workload on the cluster.

On ARM64 devices such as the Jetson family, Palette registers the host using agent mode. Appliance mode is not available on ARM64. Refer to Edge Hardware Requirements for the current ARM64 support statement.

Edge AI Use Cases​

Edge AI runs model inference on hardware at the edge, close to where data is generated, instead of sending that data to a data center or the cloud. It suits environments that need low-latency inference, operate with limited or intermittent connectivity, or must keep data on-site for privacy or compliance. Common examples include computer vision on remote equipment, real-time analysis in the field, and AI-assisted decisions where a round trip to the cloud is too slow.

For a real-world example, refer to the RapidAI case study, which describes running life-critical AI workloads on edge Kubernetes with Palette.

Get Started​

  • Jetson Requirements - Review the hardware, operating system, and Palette requirements for running Edge AI workloads on an NVIDIA Jetson device.

  • Prepare the Jetson Host - Install the software prerequisites and prepare the device so it can register with Palette.

  • Register a Jetson Host and Serve a Model - Register the device with Palette in agent mode, deploy an Edge Native cluster profile, and serve a local AI model on the device GPU.

  • Day 2 Operations - Monitor, update, reset, troubleshoot, and decommission a Jetson Edge AI deployment after it is registered and serving a model.

  • Run a Local AI Model on a Jetson Thor - Follow an end-to-end tutorial that takes a Jetson AGX Thor from a bare device to a local AI model served on the device GPU.