Skip to main content
Version: v1.1.x

Upload a Model

PaletteAI Inference Launchpad is a self-contained appliance with no outbound internet access, so you bring models to it rather than having the appliance pull them at deploy time. You download a model onto a jumpbox and transfer it to the appliance over SSH, after which the model appears in the appliance's deploy catalog.

This guide comes before Deploy a Model: the upload puts the model in the catalog, and the deploy serves it. For a model that is not in the certified catalog, author metadata.yaml first, then follow this guide. Refer to Bring Your Own Model. For the full command flags and metadata file fields, refer to Model Upload Reference.

info

You run every step in this guide from a jumpbox: a separate machine (also called the administrative workstation) with network access to your model source, such as Hugging Face, and SSH access to the appliance. The appliance never downloads models itself. For how to provision the jumpbox, refer to Administrative Workstation.

Prerequisites​

  • A jumpbox (administrative workstation) on the appliance network, provisioned with the Palette CLI, an SSH client and key pair (or password authentication), enough local disk to stage model downloads, and network access to your model source, such as Hugging Face. For details, refer to Administrative Workstation and Model Download Access.
  • rsync on the jumpbox. The Palette CLI uses it to transfer the model to the appliance over SSH.
  • The metadata YAML for the model you intend to upload. For a certified model, obtain it from Artifact Studio. For a model that is not certified, author it yourself. Refer to Bring Your Own Model. For its fields, refer to Model Metadata File.
  • The appliance's SSH host address (IP or DNS name) and an SSH user.
  • (Gated or private Hugging Face repositories) A Hugging Face access token.

Download the Model​

On the jumpbox, download the model from Hugging Face into a writable local directory. Do not use a read-only NFS mount.

  1. Run the download command, using the path to your metadata file and the directory to download into.

    palette content model download \
        --metadata model.yaml \
        --model-dir ./models

    The model downloads to <model-dir>/<name>/<version>/, where <name> and <version> come from the metadata file.

  2. (Gated or private Hugging Face repositories) Provide a Hugging Face token through the HF_TOKEN environment variable.

    HF_TOKEN=<hugging-face-token> palette content model download \
        --metadata model.yaml \
        --model-dir ./models

Upload the Model to the Appliance​

Ship the downloaded directory from the jumpbox to the appliance over SSH. On a multi-node appliance, upload to a single node; the appliance syncs the model to the remaining nodes automatically.

  1. Run the upload command with the model directory and the appliance's SSH details.

    palette content model upload \
        --metadata model.yaml \
        --model-dir ./models \
        --ssh-user <ssh-user> \
        --ssh-host <appliance-host> \
        --ssh-key <private-key-path>

    Replace <ssh-user> and <appliance-host> with the appliance's SSH user and address, and <private-key-path> with the path to your private key, such as ~/.ssh/id_ed25519.

    --model-dir points at the parent, not the model's own directory

    --model-dir is the directory that contains <name>/<version>/, not the directory named after the model. The Palette CLI composes the model path as <model-dir>/<name>/<version>/ from the name and version fields in your metadata YAML. For a model at ./models/my-model/1.0.0/, pass --model-dir ./models. Passing --model-dir ./models/my-model produces the error model dir ./models/my-model/my-model/1.0.0 is not a complete download for my-model@1.0.0, with the model name doubled in the path.

The upload command accepts other flags, including password authentication (--ssh-password, optionally with --insecure-skip-host-key-check), one-step download and upload (--download), and metadata-only sync (--metadata-only). For the full list, refer to Model Upload Reference.

Verify the Model and Deploy It​

  1. In the appliance console, select Cluster from the left main menu, then select the Models tab and Deploy New Model to open the Deploy model dialog.

  2. Open the model drop-down menu and confirm the model you uploaded is listed. Each entry renders as <Name> · <N>+GPU · <size>GB. The model appears on the next catalog scan after the upload finishes.

  3. Deploy the model by following Deploy a Model.

Next Steps​

  • Deploy the model: Follow Deploy a Model to deploy the uploaded model and verify it is serving.
  • Bring your own model: Follow Bring Your Own Model to author metadata for a model that is not in the certified catalog, then download, upload, and deploy it.
  • Replace a serving model: Follow Replace a Model to remove the current model from a node and deploy this one instead.
  • Review the reference: Refer to Model Upload Reference for the full command flags and metadata file fields.
  • Enable vision preprocessing: If you uploaded a vision model to use with a text-only model, follow Enable Vision Preprocessing.