Upload a Model
PaletteAI Inference Launchpad is a self-contained appliance with no outbound internet access, so you bring models to it rather than having the appliance pull them at deploy time. For a certified model, you work from a model recipe that you download from Artifact Studio. You upload the recipe to the appliance, then download the model weights onto a jumpbox and upload them to the appliance over SSH, after which the model appears in the appliance's deploy catalog. For what a recipe contains and why the upload takes two steps, refer to Model Recipes.
This guide is for a certified model, which comes with a model recipe. If you bring your own model, follow Bring Your Own Model instead.
After the upload, the model appears in the deploy catalog, and you serve it by following Deploy a Model. For the full command flags and metadata file fields, refer to Model Upload Reference.
You run every step in this guide from a jumpbox: a separate machine (also called the administrative workstation) with outbound internet access and SSH access to the appliance. The jumpbox downloads the model recipe, which includes the vLLM engine image, and the model weights from the internet. The appliance never downloads models itself. For how to provision the jumpbox, refer to Administrative Workstation.
Prerequisites
-
A jumpbox (administrative workstation) on the appliance network, provisioned with the Palette CLI, an SSH client and key pair (or password authentication), and enough local disk to stage model downloads.
-
Outbound internet access from the jumpbox. The jumpbox downloads the model recipe, including its vLLM engine image, from Artifact Studio and the model weights from Hugging Face. For details, refer to Administrative Workstation and Model Download Access.
-
rsyncon the jumpbox. The Palette CLI uses it to transfer the model weights to the appliance over SSH. -
The appliance's SSH host address (IP or DNS name) and an SSH user with
sudoaccess, which you use to read the node's upload token and to transfer the weights. -
Network access from the jumpbox to the appliance node on TCP port
5082, which the Palette CLI uses to upload the model recipe to the node's Local UI API, or on TCP port5080to upload it from Local UI in a browser. -
Access to Artifact Studio, Spectro Cloud's artifact download portal, to download the model recipe.
-
(Gated or private Hugging Face repositories) A Hugging Face access token.
Download the Model Recipe
-
In Artifact Studio, download the model recipe for the model you intend to run. NVIDIA and AMD recipes are separate artifacts, so choose the recipe that matches the GPU vendor of your hardware.
-
Save the recipe on the jumpbox. You use it twice. First, you upload the recipe to the appliance. Then the weights commands read the model's name, version, and Hugging Face source from it.
Upload the Model Recipe
Upload the recipe to the appliance as content. The appliance loads the inference engine image from the recipe and places the model's metadata file in the model's directory. On a multi-node appliance, upload the recipe to the leader node. Content uploaded on the leader synchronizes automatically to every linked host. You can upload with the Palette CLI or from Local UI. Select the tab for your method.
- Palette CLI
- Local UI
The Palette CLI authenticates to the node with a per-node upload token, so you do not sign in to Palette for this step.
-
From the jumpbox, read the upload token from the leader node and store it in an environment variable.
export AIL_NODE_TOKEN=$(ssh <ssh-user>@<appliance-host> sudo cat /opt/spectrocloud/.upload-auth-token)Replace
<ssh-user>with the appliance's SSH user and<appliance-host>with the leader node's address.The command prints nothing when it succeeds. If SSH prompts for a password, enter the SSH user's password.
-
Upload the recipe to the leader node.
palette content upload \
--file <model-recipe-file> \
--token "$AIL_NODE_TOKEN" \
<appliance-host>Replace
<model-recipe-file>with the path to the recipe on the jumpbox and<appliance-host>with the leader node's address. The command targets port5082by default.Expected outputresponse: Uploaded content successfullyAfter the transfer completes, the node unpacks the recipe and imports the engine image into its local registry. Wait for the node to finish before you continue.
-
Open Local UI on the leader node at
https://<appliance-host>:5080and sign in. Replace<appliance-host>with the leader node's address. -
From the left main menu, select Content > Actions > Upload Content.
-
Select the recipe file and start the upload.
-
After the upload reaches 100 percent, wait for the node to finish unpacking the recipe before you continue.
Download and Upload the Model Weights
After the recipe upload, the model does not appear in the deploy catalog until its weights arrive. This is expected. For the reason, refer to The State Between the Two Uploads.
Choose one of the following options. Both options download the weights from Hugging Face onto the jumpbox and upload them to the appliance over SSH, so use only one. For when to use each, refer to Two Commands or One for the Weights.
- Option 1: Download, Then Upload runs two commands.
- Option 2: Download and Upload in One Command runs one command.
On a multi-node appliance, upload the weights to a single node; the appliance syncs the model to the remaining nodes automatically. You can use the leader node again.
Option 1: Download, Then Upload
Download the weights into a writable local directory on the jumpbox, then ship them to the appliance. Do not use a read-only NFS mount. The Palette CLI reads the model name, version, and Hugging Face source from the recipe.
-
Run the download command with the path to the recipe and the directory to download into.
palette content model download \
--model-recipe <model-recipe-file> \
--model-dir ./modelsReplace
<model-recipe-file>with the path to the recipe on the jumpbox.Expected outputModel: <name>@<version> (HuggingFace: <hugging-face-repository>@<revision>)
...
Downloaded <name>@<version> to models/<name>/<version>The model downloads to
<model-dir>/<name>/<version>/, where<name>and<version>come from the metadata inside the recipe.For a gated or private Hugging Face repository, set the
HF_TOKENenvironment variable to your Hugging Face access token before you run the command. -
Run the upload command with the recipe, the model directory, and the appliance's SSH details.
palette content model upload \
--model-recipe <model-recipe-file> \
--model-dir ./models \
--ssh-user <ssh-user> \
--ssh-host <appliance-host> \
--ssh-key <private-key-path>Replace
<model-recipe-file>with the path to the recipe,<ssh-user>and<appliance-host>with the appliance's SSH user and address, and<private-key-path>with the path to your private key, such as~/.ssh/id_ed25519.Expected outputModel: <name>@<version> (HuggingFace: <hugging-face-repository>@<revision>)
Model dir models/<name>/<version> already complete (per .download-complete); skipping HuggingFace download
...
metadata.yaml: already up to date on remote, nothing sent
Uploaded <name>@<version> to <appliance-host>:/opt/data/spectrocloud/models/<name>/<version> (full)Because the recipe upload already placed the metadata file, the metadata line usually reports that nothing was sent. The line
metadata.yaml: transferredalso indicates success.warning--model-dirpoints at the parent directory that contains<name>/<version>/, not at the directory named after the model. The Palette CLI composes the model path as<model-dir>/<name>/<version>/from thenameandversionfields in the model metadata. For a model at./models/my-model/1.0.0/, pass--model-dir ./models. Passing--model-dir ./models/my-modelproduces an error that beginsmodel dir models/my-model/my-model/1.0.0 is not a complete download for my-model@1.0.0, with the model name doubled in the path.
The upload command accepts other flags, including metadata-only sync (--metadata-only). For the full list, refer to
Model Upload Reference.
Option 2: Download and Upload in One Command
Use this option instead of Option 1. The upload command downloads any weights that are missing from the model directory, then transfers them to the appliance.
-
Run the upload command with
--download. This example uses password authentication, which needs--insecure-skip-host-key-checkwhen the jumpbox does not yet trust the node's SSH host key.palette content model upload \
--model-recipe <model-recipe-file> \
--model-dir ./models \
--download \
--ssh-user <ssh-user> \
--ssh-password <ssh-password> \
--ssh-host <appliance-host> \
--insecure-skip-host-key-checkReplace
<model-recipe-file>with the path to the recipe,<ssh-user>and<appliance-host>with the appliance's SSH user and address, and<ssh-password>with the SSH user's password.For a gated or private Hugging Face repository, set the
HF_TOKENenvironment variable to your Hugging Face access token before you run the command.Expected outputModel: <name>@<version> (HuggingFace: <hugging-face-repository>@<revision>)
...
metadata.yaml: already up to date on remote, nothing sent
Uploaded <name>@<version> to <appliance-host>:/opt/data/spectrocloud/models/<name>/<version> (full)
Verify the Model and Deploy It
-
In the appliance console, select Cluster from the left main menu, then select the Models tab.
-
Select Deploy Model to open the Deploy a model wizard.
-
Open the model drop-down menu and confirm the model you uploaded is listed. Each entry shows the model name, followed by details such as its engine and size. The model can appear in the list before all its weights have landed. Wait until the upload command reports
Uploaded <name>@<version>before you deploy it. -
Deploy the model by following Deploy a Model.
Move from a Metadata File to a Model Recipe
Artifact Studio publishes a model recipe in place of the standalone metadata file. If your workflow downloads
metadata.yaml for a certified model, change it as follows.
-
Download the model recipe for your GPU vendor instead of the metadata file.
-
Upload the recipe as content, as described in Upload the Model Recipe. This step is new. It brings the model's inference engine image to the appliance.
-
In the download and upload commands, replace
--metadata <metadata-file>with--model-recipe <model-recipe-file>. Every other flag stays the same.
A metadata file that you already have keeps working with --metadata. It does not carry an engine image. For what that
means for the appliance's network access, refer to Upload with a Metadata File. For a
certified model that has a recipe, use the recipe.
Upload with a Metadata File
Use a metadata file for a model you bring yourself, which has no recipe. For a certified model, this is the prior flow,
which the recipe replaces. The --metadata and --model-recipe flags are mutually exclusive, so pass exactly one of
them.
A metadata file does not carry the vLLM engine image that the model runs on, and the appliance's air-gapped content includes no vLLM image. The engine image must be downloaded from the internet, so this flow works only in an environment with internet access. The jumpbox also needs internet access to download the model weights from Hugging Face. For a certified model, use the model recipe, which brings the engine image to the appliance.
-
On the jumpbox, download the model from Hugging Face with the path to your metadata file.
palette content model download \
--metadata <metadata-file> \
--model-dir ./modelsReplace
<metadata-file>with the path to your metadata file. For a gated or private Hugging Face repository, set theHF_TOKENenvironment variable to your Hugging Face access token before you run the command.Expected outputModel: <name>@<version> (HuggingFace: <hugging-face-repository>@<revision>)
...
Downloaded <name>@<version> to models/<name>/<version> -
Upload the model to the appliance over SSH.
palette content model upload \
--metadata <metadata-file> \
--model-dir ./models \
--ssh-user <ssh-user> \
--ssh-host <appliance-host> \
--ssh-key <private-key-path>Replace
<metadata-file>with the path to your metadata file,<ssh-user>and<appliance-host>with the appliance's SSH user and address, and<private-key-path>with the path to your private key.Expected outputModel: <name>@<version> (HuggingFace: <hugging-face-repository>@<revision>)
Model dir models/<name>/<version> already complete (per .download-complete); skipping HuggingFace download
...
metadata.yaml: transferred
Uploaded <name>@<version> to <appliance-host>:/opt/data/spectrocloud/models/<name>/<version> (full)To download and upload in one command, add
--download. For when to use each form, refer to Two Commands or One for the Weights. Then verify and deploy the model as described in Verify the Model and Deploy It.
Next Steps
-
To deploy the uploaded model and verify that it is serving, follow Deploy a Model.
-
To remove the current model from a node and deploy this one instead, follow Replace a Model.
-
If you uploaded a vision model to use with a text-only model, follow Enable Vision Preprocessing.