Skip to main content
Version: v1.0.x

PaletteAI Inference Launchpad Inference Engines

This page explains what an inference engine is in PaletteAI Inference Launchpad, how the appliance selects one automatically, and when you might choose one yourself. It gives you the background to make an informed choice when you deploy a model, where the engine is an optional setting.

Inference Engine Overview​

An inference engine is the runtime that loads a model and serves its requests. The appliance exposes each loaded model as an OpenAI-compatible endpoint, but the engine behind that endpoint is what actually runs the model on a node's CPU or GPUs. The engine determines how the model runs, which hardware it can use, and which serving features are available.

Each engine has a kind that identifies the underlying runtime, and each engine is either GPU-capable or CPU-only. A GPU model can only run on a GPU-capable engine, and a model intended for CPU nodes runs on a CPU-only engine.

Automatic Engine Selection​

When you deploy a model, the engine setting defaults to engine (auto). With the automatic option, the appliance selects an engine that fits the model you chose, a GPU-capable engine for a GPU model and a CPU-only engine for a CPU model. Leaving the setting on automatic is the recommended choice for most deployments, because the appliance matches the engine to the model and the target node for you.

The engines you can choose from depend on how your appliance is configured. The deploy panel lists the automatic option first, followed by any named engines the appliance exposes, each labeled with its kind, such as default · vllm.

Supported Engine Kinds​

PaletteAI Inference Launchpad supports one engine kind, vLLM, which provides high-throughput GPU serving for large models. Because it is the only supported kind, the automatic option and any manual selection both resolve to a vLLM engine.

Manual Engine Selection​

Leave the engine on the automatic option unless you have a specific reason to pin a model to a particular engine, such as a serving feature that a specific kind provides. When you select an engine yourself, the appliance only offers engines that fit the model's hardware, so you cannot select a CPU-only engine for a GPU model or the reverse.

Resources​