Skip to main content
Version: latest

Upgrade Airgap Palette Installed with Kubernetes

This guide takes you through the process of upgrading a self-hosted airgap Palette instance installed on Kubernetes.

warning

Before upgrading Palette to a new major version, you must first update it to the latest patch version of the latest minor version available. Refer to the Supported Upgrade Paths section for details.

If your setup includes a PCG, you must also allow the PCG to upgrade automatically before each major or minor Palette upgrade.

Prerequisites

  • Access to the x86 Linux jumpbox or bastion host with connectivity to the target platform where you have installed Palette.

  • An OCI registry such as Harbor or AWS ECR configured and available to store the new Palette images and packs.

  • Access to the latest Palette airgap setup binary. Refer to Access Palette for more details.

  • kubectl and helm available in your system.

  • Access to the kubeconfig file of the target Kubernetes cluster. You must be able to interact with the cluster through kubectl and have sufficient permissions to upgrade Palette. We recommend using a role with the cluster-admin permissions.

  • unzip or a similar tool available in your system.

  • Access to the latest Palette Helm Chart. Refer to Access Palette for more details.

Upgrade

  1. Log in to the Linux environment from which you can access your self-hosted airgap Palette instance.

  2. Authenticate with your OCI registry and get credentials to your spectro-packs and spectro-images repositories.

    info

    Be mindful of the auth token timeout. Uploading Palette images and packages to the OCI registry can take about an hour. If your auth token expires before everything is uploaded, you will need to re-authenticate and start the upload again.

    Select one of the following tabs for instructions on how to authenticate with Harbor and AWS ECR.

    Use the following command template to authenticate with your Harbor OCI registry through oras. Refer to the oras login docs for more details about the available CLI flags and examples.

    oras login X.X.X.X --user 'yourUserNameHere' --password 'yourPasswordHere'
    tip

    If your Harbor registry has a self-signed certificate, use the --insecure flag.

    oras login X.X.X.X --insecure --user 'yourUserNameHere' --password 'yourPasswordHere'
  3. Prepare and export the environment variables required by the Palette airgap binary. Select one of the following tabs for instructions on how to prepare these variables for Harbor and AWS ECR.

    • OCI_IMAGE_REGISTRY: The IP address or domain name of the spectro-images registry.

    • OCI_IMAGE_BASE: The namespace or repository that hosts the images.

    • OCI_PACK_REGISTRY: The IP address or domain name of the spectro-packs registry.

    • OCI_PACK_BASE: The namespace or repository that hosts the packages.

      export OCI_IMAGE_REGISTRY=<harbor-endpoint>
      export OCI_IMAGE_BASE=spectro-images
      export OCI_PACK_REGISTRY=<harbor-endpoint>
      export OCI_PACK_BASE=spectro-packs

      Consider the following example for reference.

      export OCI_IMAGE_REGISTRY=example.internal.com
      export OCI_IMAGE_BASE=spectro-images
      export OCI_PACK_REGISTRY=10.10.100.48
      export OCI_PACK_BASE=spectro-packs
  4. Download the airgap setup binary using the credentials our support team provided along with the binary version. Use the following command template to set up a VERSION variable you'll use in the following steps.

    export VERSION=<x.y.z>

    Consider the following example for reference.

    export VERSION=4.2.7
  5. Use the following command template to download the new Palette airgap installation bin. Enter the username, password, and the Palette airgap installation URL you received from our support team. In the output file name, replace <version> with the Palette version you're downloading.

    curl --user <username>:<password> --output airgap-<version>.bin <url-to-airgap-installation-bin>

    Consider the following command example for reference.

    curl --user <username>:<password> --output airgap-4.2.7.bin https://software.spectrocloud.com/airgap-v4.2.7.bin
  6. Use the following command template to execute the new Palette airgap installation bin.

    chmod +x airgap-v$VERSION.bin && ./airgap-v$VERSION.bin

    After the binary execution is done, you should receive the following success message (the example is shortened for brevity). As a result of starting the binary, you should have a spectro-manifests archive in your temporary directory.

    Verifying archive integrity...  100%   MD5 checksums are OK. All good.
    Uncompressing Airgap Setup - Version 4.0.17 100%
    Setting up Packs
    - Pushing Pack cni-calico:3.25.1
    ...
    Setting up Images
    - Pushing image docker.io/kindest/kindnetd:v20230227-15197099
    - Pushing image gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.22.8
    ...
    Preparing Manifests Archive
    Manifests are available in /tmp/spectro-manifests-1696971110.zip. Extract the archive to a file server to serve as a Spectro Cloud Repository
    Setup Completed
  7. Move the spectro-manifests archive to a directory that your file server can access and use the following command template to unzip it.

    unzip spectro-manifests-<file-id>.zip -d /target/folder
    warning

    Do not remove or replace the existing files inside your target folder that is served by the file server. The previous content is necessary for the upgrade process.

  8. Refer to the Additional Packs page and update the packages you are currently using. You must update each package separately.

    info

    Depending on your underlying infrastructure provider and Kubernetes distribution, you may need to modify the following Palette upgrade steps to match your environment.

  9. Navigate to the directory with the Palette installation zip file. Unzip the file to a palette-install directory.

    unzip release-*.zip -d palette-install
  10. Navigate to the release directory inside palette-install.

    cd palette-install/charts/release-*
  11. In a code editor of your choice, open the extras/cert-manager/values.yaml file and replace the controllerImage, webhookImage, and amceResolverImage image URLs with your OCI image registry URLs.

    image:
    cainjectorImage: "gcr.io/spectro-images-public/release-fips/jetstack/cert-manager-cainjector:spectro-v1.11.0-20230427"
    controllerImage: "<your-oci-registry-url>/spectro-images-public/release-fips/jetstack/cert-manager-controller:spectro-v1.11.0-20230427"
    webhookImage: "<your-oci-registry-url>/spectro-images-public/release-fips/jetstack/cert-manager-webhook:spectro-v1.11.0-20230808"
    amceResolverImage: "<your-oci-registry-url>/spectro-images-public/release-fips/jetstack/cert-manager-acmesolver:spectro-v1.11.0-20230427"

    featureGates: "AdditionalCertificateOutputFormats=true"

    Consider the following example for reference.

    image:
    cainjectorImage: "gcr.io/spectro-images-public/release-fips/jetstack/cert-manager-cainjector:spectro-v1.11.0-20230427"
    controllerImage: "harbor.docs.spectro.dev/spectro-images-public/release-fips/jetstack/cert-manager-controller:spectro-v1.11.0-20230427"
    webhookImage: "harbor.docs.spectro.dev/spectro-images-public/release-fips/jetstack/cert-manager-webhook:spectro-v1.11.0-20230808"
    amceResolverImage: "harbor.docs.spectro.dev/spectro-images-public/release-fips/jetstack/cert-manager-acmesolver:spectro-v1.11.0-20230427"

    featureGates: "AdditionalCertificateOutputFormats=true"
  12. Update the cert-manager chart using the following command.

    helm upgrade --values extras/cert-manager/values.yaml \
    cert-manager extras/cert-manager/cert-manager-*.tgz --install

    You should receive an output similar to the following.

    Release "cert-manager" has been upgraded. Happy Helming!
    NAME: cert-manager
    LAST DEPLOYED: Thu Feb 22 19:42:33 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None
  13. Prepare the Palette configuration file values.yaml. If you saved values.yaml used during the Palette installation, you can reuse it for the upgrade. Alternatively, follow the Kubernetes Installation Instructions to populate your values.yaml.

    warning

    Ensure that the values.yaml file is ready before proceeding. Specifically, make sure that the ociPackEcrRegistry and ociImageRegistry configurations include the parameters necessary to interact with your spectro-images and spectro-packs repositories.

  14. Upgrade the image-swap chart with the following command. Point to the palette/values.yaml file from step twelve.

    helm upgrade --values palette/values.yaml \
    image-swap extras/image-swap/image-swap-*.tgz --install

    You should receive an output similar to the following.

    Release "image-swap" has been upgraded. Happy Helming!
    NAME: image-swap
    LAST DEPLOYED: Thu Feb 22 19:44:13 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None
  15. Upgrade the reach-system chart with the following command. Point to the palette/values.yaml file from step twelve.

    helm upgrade --values palette/values.yaml \
    reach-system extras/reach-system/reach-system-\*.tgz --install

    You should receive an output similar to the following.

    Release "reach-system" has been upgraded. Happy Helming!
    NAME: reach-system
    LAST DEPLOYED: Thu Feb 22 19:47:10 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None
  16. Upgrade Palette with the following command.

    helm upgrade --values palette/values.yaml \
    hubble palette/spectro-mgmt-plane-\*.tgz --install

    You should receive an output similar to the following.

    Release "hubble" has been upgraded. Happy Helming!
    NAME: hubble
    LAST DEPLOYED: Thu Feb 22 20:05:24 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None
  17. Use the following command to track the upgrade process.

    kubectl get pods --all-namespaces --watch
    tip

    For a more user-friendly experience, consider using K9s or a similar tool to track the upgrade.

    The upgrade usually takes up to five minutes. Palette is upgraded when the deployments in the namespaces cp-system, hubble-system, ingress-nginx, jet-system , and ui-system are in the Ready status.

Validate

  1. In your terminal, issue the following command.

    helm ls

    You should receive an output with the version and other details of the currently deployed apps. Check the App Version column of cert-manager, image-swap, reach-system, and hubble to verify that they have the expected versions.

    NAME        	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART                   	APP VERSION
    cert-manager default 2 2024-02-22 19:42:33.776829 +0100 CET deployed cert-manager-1.11.0 1.11.0
    image-swap default 2 2024-02-22 19:44:13.209592 +0100 CET deployed image-swap-v1.5.2-spectro-4.1.1 1.5.2
    reach-system default 2 2024-02-22 19:47:10.558061 +0100 CET deployed reach-system-4.2.0 4.2.0
    hubble default 2 2024-02-22 19:47:24.085305 +0100 CET deployed spectro-mgmt-plane-4.2.7 4.2.7
  2. Issue the following command to verify that the Palette clusters work as expected.

    kubectl get pods --all-namespaces --output custom-columns="NAMESPACE:metadata.namespace,NAME:metadata.name,STATUS:status.phase" \
    | grep -E '^(cp-system|hubble-system|ingress-nginx|jet-system|ui-system)\s'

    The command should return a list of deployments in the cp-system, hubble-system, ingress-nginx, jet-system, and ui-system namespaces. All deployments should have the status Running.

    cp-system       spectro-cp-ui-689984f88d-54wsw             Running
    hubble-system auth-85b748cbf4-6drkn Running
    hubble-system auth-85b748cbf4-dwhw2 Running
    hubble-system cloud-fb74b8558-lqjq5 Running
    hubble-system cloud-fb74b8558-zkfp5 Running
    hubble-system configserver-685fcc5b6d-t8f8h Running
    hubble-system event-68568f54c7-jzx5t Running
    hubble-system event-68568f54c7-w9rnh Running
    hubble-system foreq-6b689f54fb-vxjts Running
    hubble-system hashboard-897bc9884-pxpvn Running
    hubble-system hashboard-897bc9884-rmn69 Running
    hubble-system hutil-6d7c478c96-td8q4 Running
    hubble-system hutil-6d7c478c96-zjhk4 Running
    hubble-system mgmt-85dbf6bf9c-jbggc Running
    hubble-system mongo-0 Running
    hubble-system mongo-1 Running
    hubble-system mongo-2 Running
    hubble-system msgbroker-6c9b9fbf8b-mcsn5 Running
    hubble-system oci-proxy-7789cf9bd8-qcjkl Running
    hubble-system packsync-28205220-bmzcg Succeeded
    hubble-system spectrocluster-6c57f5775d-dcm2q Running
    hubble-system spectrocluster-6c57f5775d-gmdt2 Running
    hubble-system spectrocluster-6c57f5775d-sxks5 Running
    hubble-system system-686d77b947-8949z Running
    hubble-system system-686d77b947-cgzx6 Running
    hubble-system timeseries-7865bc9c56-5q87l Running
    hubble-system timeseries-7865bc9c56-scncb Running
    hubble-system timeseries-7865bc9c56-sxmgb Running
    hubble-system user-5c9f6c6f4b-9dgqz Running
    hubble-system user-5c9f6c6f4b-hxkj6 Running
    ingress-nginx ingress-nginx-controller-2txsv Running
    ingress-nginx ingress-nginx-controller-55pk2 Running
    ingress-nginx ingress-nginx-controller-gmps9 Running
    jet-system jet-6599b9856d-t9mr4 Running
    ui-system spectro-ui-76ffdf67fb-rkgx8 Running