Deploy a Model
This guide explains how to deploy a model to a running PaletteAI Inference Launchpad appliance and verify that the model is serving requests. The appliance places the model automatically on the best-fit node. For how placement and the deploy lifecycle work, refer to Model Provisioning Lifecycle.
Prerequisites
- A running PaletteAI Inference Launchpad appliance, with the admin console reachable and operator access.
- At least one node with free capacity for the model you intend to deploy.
- The model present in the appliance catalog and able to fit the appliance's available GPU resources. To confirm support, refer to Certified Models by Hardware and Suggested Hardware. To place a model in the appliance catalog, follow Upload a Model first.
Deploy a Model
-
From the left main menu, select Cluster.
-
Select Deploy model to open the deploy panel.
-
Review the cluster capacity line to find which nodes have free GPUs. Each node shows either a free count, such as
N of T free, orallocation unknown. -
Open the model drop-down menu and select the model to deploy.
-
(Optional) Open the engine drop-down menu and select an engine. Leave it on engine (auto) to let the appliance choose the engine. For what an engine is and when you would override the automatic choice, refer to Inference Engines.
-
Confirm that the will deploy to line shows a best-fit node. If the panel shows Deploy held with a reason, or the Deploy button is unavailable, no node currently fits the model. Refer to Resolve a Blocked Deployment before you continue.
-
Select Deploy, review the deployment preview, and then select Confirm & apply.
Verify the Model Is Serving
Confirm the model is serving before you route traffic to it.
-
Locate the model in the Model table on the Cluster page.
-
Confirm that the model health reads
N/N healthyand that its state reachesreadyorserving. A state ofdeployingorsmoke-testingmeans the model is still coming online, and a state offailedorverification failedmeans the model did not come online.
While a model is deploying, a node hosting it can briefly report a Degraded or Not serving state. This is expected
during deployment and clears once the model finishes coming online, at which point the node returns to a healthy state.
For why a model becomes routable only after it passes its smoke test, refer to Model Provisioning Lifecycle.
Resolve a Blocked Deployment
If no node can host the model you selected, the Deploy button is unavailable and the panel explains why. The reason is one of the following:
- The model is already deployed on every node in the cluster.
- The model needs GPUs, but no node in the cluster has a GPU.
- No node has enough free GPUs for the model because every GPU node is full.
- No node has enough free GPUs for the model, and some nodes report
allocation unknown.
To resolve the reason, free GPUs on a node by draining or shutting down another model, add capacity to the cluster, or resolve the unknown allocation on the affected nodes. Then deploy the model again.