Skip to main content

Model Placement

When you deploy a model, you decide which nodes run it. On a single-node appliance that choice is one node. On a multi-node appliance, mixed hardware, reserved capacity, and locally staged weights often mean that not every ready node should run every model.

This page explains that choice. To deploy a model and change which nodes run it, refer to Deploy a Model.

Why Choose Specific Nodes​

Choosing nodes tells the appliance exactly where a model may run. Pin a model when nodes are not interchangeable:

  • Mixed hardware. An H200 pool beside L40S nodes, or GPU workers beside a CPU-only control plane. Deploying the model on every node puts engines on machines that cannot serve the model.
  • Reserved capacity. Keep a large model off nodes you want free for other work.
  • Staged weights. For a model whose weights live on disk, only nodes that already hold the weights can serve it.

The appliance runs the model only on the nodes you choose. A node you add to the cluster later does not receive the model until you add it, as described in Add a Model to More Nodes.

How Placement Works​

The appliance creates one inference engine per chosen node and exposes those engines through a single per-model endpoint. Clients keep calling the model by name. They do not choose a node.

The Nodes field on the deploy wizard's Model and placement step offers the cluster's nodes. For why a node cannot be chosen, refer to Why a Node Cannot Be Chosen. The field can open with nodes already selected.

If a node cannot run the model when the deploy runs, the appliance reports the outcome on the model's condition rather than placing the engine somewhere else. Nothing falls back to another node, and unreadable capacity counts as unusable rather than free.

You change where a model runs by adding or removing nodes, as described in Deploy a Model.

How the Cluster View Reports Placement​

A model you placed on a subset of the cluster reads N of M nodes. N is how many nodes you chose. M is the cluster size. The health chip counts serving against the chosen set, so 1/2 healthy means one chosen node is not up yet.

Expanding the row lists only the nodes that belong to the model:

  • Chosen nodes always appear.
  • A node you never selected does not appear. That absence is not a fault.
  • A node you de-selected stays visible while its engine is still shutting down. It reads Removing with the message No longer chosen. Its engine is shutting down. The row disappears once the engine is gone.
  • A chosen node that has not started an engine yet reads Waiting to start. Investigate that state if it persists.

Why a Node Cannot Be Chosen​

An ineligible node is not selectable and states why. A node cannot run a model for one of the following reasons.

ReasonWhat it means
Too few free GPUsAll the node's GPUs are in use, or the node has fewer free GPUs than the GPU width. A model that shares a GPU can still use a node whose GPUs a sharing group holds.
A different GPU productThe model requires a GPU product that this node does not have.
CordonedNo new engine can be scheduled on the node.
Not readyThe node does not report that it is ready to run workloads.
Not in the clusterThe node is not part of this cluster.
Unknown GPU capacity or productThe appliance cannot confirm that the node can run the model. Unknown capacity counts as unusable, never as free.
Weights not stagedThe model's weights are not on this node, or only its metadata is.
Unknown weights stateThe appliance cannot confirm that the weights are on this node.

How Local Weights Affect the Choice​

Choosing a node does not copy weights to it. For a model you uploaded to the appliance, only nodes that already hold the weights can run it. A node appears in the Nodes list once the weights are staged on it. For the upload procedure, refer to Upload a Model.

What Model Placement Is Not​

  • Not a weight copy. Stage weights on a node before you select it for a model you uploaded.
  • Not automatic rebalancing. A model does not follow new hardware. Add the node, as described in Add a Model to More Nodes.
  • Not a substitute for per-node maintenance. Per-node Replace swaps the engine on one node. Per-node Remove stops the model on one node while it keeps running elsewhere. Retry deploy appears only on a node whose deploy failed.

Next Steps​