Skip to main content

Upload a Model

PaletteAI Inference Launchpad is a self-contained appliance with no outbound internet access, so you bring models to it rather than having the appliance pull them at deploy time. For a certified model, you work from a model recipe that you download from Artifact Studio. You upload the recipe to the appliance, then download the model weights onto a jumpbox and upload them to the appliance over SSH, after which the model appears in the appliance's deploy catalog. For what a recipe contains and why the upload takes two steps, refer to Model Recipes.

This guide is for a certified model, which comes with a model recipe. If you bring your own model, follow Bring Your Own Model instead.

After the upload, the model appears in the deploy catalog, and you serve it by following Deploy a Model. For the full command flags and metadata file fields, refer to Model Upload Reference.

info

You run every step in this guide from a jumpbox: a separate machine (also called the administrative workstation) with outbound internet access and SSH access to the appliance. The jumpbox downloads the model recipe, which includes the vLLM engine image, and the model weights from the internet. The appliance never downloads models itself. For how to provision the jumpbox, refer to Administrative Workstation.

Prerequisites​

  • A jumpbox (administrative workstation) on the appliance network, provisioned with the Palette CLI, an SSH client and key pair (or password authentication), and enough local disk to stage model downloads.

  • Outbound internet access from the jumpbox. The jumpbox downloads the model recipe, including its vLLM engine image, from Artifact Studio and the model weights from Hugging Face. For details, refer to Administrative Workstation and Model Download Access.

  • rsync on the jumpbox. The Palette CLI uses it to transfer the model weights to the appliance over SSH.

  • The appliance's SSH host address (IP or DNS name) and an SSH user with sudo access, which you use to read the node's upload token and to transfer the weights.

  • Network access from the jumpbox to the appliance node on TCP port 5082, which the Palette CLI uses to upload the model recipe to the node's Local UI API, or on TCP port 5080 to upload it from Local UI in a browser.

  • Access to Artifact Studio, Spectro Cloud's artifact download portal, to download the model recipe.

  • (Gated or private Hugging Face repositories) A Hugging Face access token.

Download the Model Recipe​

  1. In Artifact Studio, download the model recipe for the model you intend to run. NVIDIA and AMD recipes are separate artifacts, so choose the recipe that matches the GPU vendor of your hardware.

  2. Save the recipe on the jumpbox. You use it twice. First, you upload the recipe to the appliance. Then the weights commands read the model's name, version, and Hugging Face source from it.

Upload the Model Recipe​

Upload the recipe to the appliance as content. The appliance loads the inference engine image from the recipe and places the model's metadata file in the model's directory. On a multi-node appliance, upload the recipe to the leader node. Content uploaded on the leader synchronizes automatically to every linked host. You can upload with the Palette CLI or from Local UI. Select the tab for your method.

The Palette CLI authenticates to the node with a per-node upload token, so you do not sign in to Palette for this step.

  1. From the jumpbox, read the upload token from the leader node and store it in an environment variable.

    export AIL_NODE_TOKEN=$(ssh <ssh-user>@<appliance-host> sudo cat /opt/spectrocloud/.upload-auth-token)

    Replace <ssh-user> with the appliance's SSH user and <appliance-host> with the leader node's address.

    The command prints nothing when it succeeds. If SSH prompts for a password, enter the SSH user's password.

  2. Upload the recipe to the leader node.

    palette content upload \
    --file <model-recipe-file> \
    --token "$AIL_NODE_TOKEN" \
    <appliance-host>

    Replace <model-recipe-file> with the path to the recipe on the jumpbox and <appliance-host> with the leader node's address. The command targets port 5082 by default.

    Expected output
    response: Uploaded content successfully

    After the transfer completes, the node unpacks the recipe and imports the engine image into its local registry. Wait for the node to finish before you continue.

Download and Upload the Model Weights​

After the recipe upload, the model does not appear in the deploy catalog until its weights arrive. This is expected. For the reason, refer to The State Between the Two Uploads.

Choose one of the following options. Both options download the weights from Hugging Face onto the jumpbox and upload them to the appliance over SSH, so use only one. For when to use each, refer to Two Commands or One for the Weights.

On a multi-node appliance, upload the weights to a single node; the appliance syncs the model to the remaining nodes automatically. You can use the leader node again.

Option 1: Download, Then Upload​

Download the weights into a writable local directory on the jumpbox, then ship them to the appliance. Do not use a read-only NFS mount. The Palette CLI reads the model name, version, and Hugging Face source from the recipe.

  1. Run the download command with the path to the recipe and the directory to download into.

    palette content model download \
    --model-recipe <model-recipe-file> \
    --model-dir ./models

    Replace <model-recipe-file> with the path to the recipe on the jumpbox.

    Expected output
    Model: <name>@<version> (HuggingFace: <hugging-face-repository>@<revision>)
    ...
    Downloaded <name>@<version> to models/<name>/<version>

    The model downloads to <model-dir>/<name>/<version>/, where <name> and <version> come from the metadata inside the recipe.

    For a gated or private Hugging Face repository, set the HF_TOKEN environment variable to your Hugging Face access token before you run the command.

  2. Run the upload command with the recipe, the model directory, and the appliance's SSH details.

    palette content model upload \
    --model-recipe <model-recipe-file> \
    --model-dir ./models \
    --ssh-user <ssh-user> \
    --ssh-host <appliance-host> \
    --ssh-key <private-key-path>

    Replace <model-recipe-file> with the path to the recipe, <ssh-user> and <appliance-host> with the appliance's SSH user and address, and <private-key-path> with the path to your private key, such as ~/.ssh/id_ed25519.

    Expected output
    Model: <name>@<version> (HuggingFace: <hugging-face-repository>@<revision>)
    Model dir models/<name>/<version> already complete (per .download-complete); skipping HuggingFace download
    ...
    metadata.yaml: already up to date on remote, nothing sent
    Uploaded <name>@<version> to <appliance-host>:/opt/data/spectrocloud/models/<name>/<version> (full)

    Because the recipe upload already placed the metadata file, the metadata line usually reports that nothing was sent. The line metadata.yaml: transferred also indicates success.

    warning

    --model-dir points at the parent directory that contains <name>/<version>/, not at the directory named after the model. The Palette CLI composes the model path as <model-dir>/<name>/<version>/ from the name and version fields in the model metadata. For a model at ./models/my-model/1.0.0/, pass --model-dir ./models. Passing --model-dir ./models/my-model produces an error that begins model dir models/my-model/my-model/1.0.0 is not a complete download for my-model@1.0.0, with the model name doubled in the path.

The upload command accepts other flags, including metadata-only sync (--metadata-only). For the full list, refer to Model Upload Reference.

Option 2: Download and Upload in One Command​

Use this option instead of Option 1. The upload command downloads any weights that are missing from the model directory, then transfers them to the appliance.

  1. Run the upload command with --download. This example uses password authentication, which needs --insecure-skip-host-key-check when the jumpbox does not yet trust the node's SSH host key.

    palette content model upload \
    --model-recipe <model-recipe-file> \
    --model-dir ./models \
    --download \
    --ssh-user <ssh-user> \
    --ssh-password <ssh-password> \
    --ssh-host <appliance-host> \
    --insecure-skip-host-key-check

    Replace <model-recipe-file> with the path to the recipe, <ssh-user> and <appliance-host> with the appliance's SSH user and address, and <ssh-password> with the SSH user's password.

    For a gated or private Hugging Face repository, set the HF_TOKEN environment variable to your Hugging Face access token before you run the command.

    Expected output
    Model: <name>@<version> (HuggingFace: <hugging-face-repository>@<revision>)
    ...
    metadata.yaml: already up to date on remote, nothing sent
    Uploaded <name>@<version> to <appliance-host>:/opt/data/spectrocloud/models/<name>/<version> (full)

Verify the Model and Deploy It​

  1. In the appliance console, select Cluster from the left main menu, then select the Models tab.

  2. Select Deploy Model to open the Deploy a model wizard.

  3. Open the model drop-down menu and confirm the model you uploaded is listed. Each entry shows the model name, followed by details such as its engine and size. The model can appear in the list before all its weights have landed. Wait until the upload command reports Uploaded <name>@<version> before you deploy it.

  4. Deploy the model by following Deploy a Model.

Move from a Metadata File to a Model Recipe​

Artifact Studio publishes a model recipe in place of the standalone metadata file. If your workflow downloads metadata.yaml for a certified model, change it as follows.

  1. Download the model recipe for your GPU vendor instead of the metadata file.

  2. Upload the recipe as content, as described in Upload the Model Recipe. This step is new. It brings the model's inference engine image to the appliance.

  3. In the download and upload commands, replace --metadata <metadata-file> with --model-recipe <model-recipe-file>. Every other flag stays the same.

A metadata file that you already have keeps working with --metadata. It does not carry an engine image. For what that means for the appliance's network access, refer to Upload with a Metadata File. For a certified model that has a recipe, use the recipe.

Upload with a Metadata File​

Use a metadata file for a model you bring yourself, which has no recipe. For a certified model, this is the prior flow, which the recipe replaces. The --metadata and --model-recipe flags are mutually exclusive, so pass exactly one of them.

warning

A metadata file does not carry the vLLM engine image that the model runs on, and the appliance's air-gapped content includes no vLLM image. The engine image must be downloaded from the internet, so this flow works only in an environment with internet access. The jumpbox also needs internet access to download the model weights from Hugging Face. For a certified model, use the model recipe, which brings the engine image to the appliance.

  1. On the jumpbox, download the model from Hugging Face with the path to your metadata file.

    palette content model download \
    --metadata <metadata-file> \
    --model-dir ./models

    Replace <metadata-file> with the path to your metadata file. For a gated or private Hugging Face repository, set the HF_TOKEN environment variable to your Hugging Face access token before you run the command.

    Expected output
    Model: <name>@<version> (HuggingFace: <hugging-face-repository>@<revision>)
    ...
    Downloaded <name>@<version> to models/<name>/<version>
  2. Upload the model to the appliance over SSH.

    palette content model upload \
    --metadata <metadata-file> \
    --model-dir ./models \
    --ssh-user <ssh-user> \
    --ssh-host <appliance-host> \
    --ssh-key <private-key-path>

    Replace <metadata-file> with the path to your metadata file, <ssh-user> and <appliance-host> with the appliance's SSH user and address, and <private-key-path> with the path to your private key.

    Expected output
    Model: <name>@<version> (HuggingFace: <hugging-face-repository>@<revision>)
    Model dir models/<name>/<version> already complete (per .download-complete); skipping HuggingFace download
    ...
    metadata.yaml: transferred
    Uploaded <name>@<version> to <appliance-host>:/opt/data/spectrocloud/models/<name>/<version> (full)

    To download and upload in one command, add --download. For when to use each form, refer to Two Commands or One for the Weights. Then verify and deploy the model as described in Verify the Model and Deploy It.

Next Steps​