Upgrade Airgap Palette Installed with Kubernetes
This guide takes you through the process of upgrading a self-hosted airgap Palette instance installed on Kubernetes.
Before upgrading Palette to a new major version, you must first update it to the latest patch version of the latest minor version available. Refer to the Supported Upgrade Paths section for details.
If your setup includes a PCG, you must also allow the PCG to upgrade automatically before each major or minor Palette upgrade.
Prerequisites
-
Access to the x86 Linux jumpbox or bastion host with connectivity to the target platform where you have installed Palette.
-
An OCI registry such as Harbor or AWS ECR configured and available to store the new Palette images and packs.
-
Access to the latest Palette airgap setup binary. Refer to Access Palette for more details.
-
Access to the kubeconfig file of the target Kubernetes cluster. You must be able to interact with the cluster through
kubectl
and have sufficient permissions to upgrade Palette. We recommend using a role with the cluster-admin permissions. -
unzip
or a similar tool available in your system. -
Access to the latest Palette Helm Chart. Refer to Access Palette for more details.
Upgrade
-
Log in to the Linux environment from which you can access your self-hosted airgap Palette instance.
-
Authenticate with your OCI registry and get credentials to your
spectro-packs
andspectro-images
repositories.infoBe mindful of the auth token timeout. Uploading Palette images and packages to the OCI registry can take about an hour. If your auth token expires before everything is uploaded, you will need to re-authenticate and start the upload again.
Select one of the following tabs for instructions on how to authenticate with Harbor and AWS ECR.
- Harbor
- AWS ECR
Use the following command template to authenticate with your Harbor OCI registry through
oras
. Refer to the oras login docs for more details about the available CLI flags and examples.oras login X.X.X.X --user 'yourUserNameHere' --password 'yourPasswordHere'
tipIf your Harbor registry has a self-signed certificate, use the
--insecure
flag.oras login X.X.X.X --insecure --user 'yourUserNameHere' --password 'yourPasswordHere'
To get the authentication commands for your specific registries, navigate to the AWS ECR console and click View push commands. Refer to the AWS ECR Authentication docs for more details.
Alternatively, you can use the following command templates:
-
Authenticate with your private
spectro-packs
registry:-
<ecr-region>
: Replace with the region where your ECR repository is hosted. -
<aws-account-id>
: Replace with your AWS account ID.aws ecr get-login-password --region <ecr-region> | oras login --username AWS --password-stdin <aws-account-id>.dkr.ecr.<ecr-region>.amazonaws.com
Consider the following example for reference.
aws ecr get-login-password --region us-east-1 | oras login --username AWS --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com
-
-
Authenticate with your public
spectro-images
registry:-
<ecr-region>
: Replace with the region where your ECR repository is hosted. -
<public-ecr-url>
: Replace with the URL to your public ECR registry.aws ecr-public get-login-password --region <ecr-region> | docker login --username AWS --password-stdin <public-ecr-url>
Consider the following example for reference.
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/dfa987f
-
-
Prepare and export the environment variables required by the Palette airgap binary. Select one of the following tabs for instructions on how to prepare these variables for Harbor and AWS ECR.
- Harbor
- AWS ECR
-
OCI_IMAGE_REGISTRY
: The IP address or domain name of thespectro-images
registry. -
OCI_IMAGE_BASE
: The namespace or repository that hosts the images. -
OCI_PACK_REGISTRY
: The IP address or domain name of thespectro-packs
registry. -
OCI_PACK_BASE
: The namespace or repository that hosts the packages.export OCI_IMAGE_REGISTRY=<harbor-endpoint>
export OCI_IMAGE_BASE=spectro-images
export OCI_PACK_REGISTRY=<harbor-endpoint>
export OCI_PACK_BASE=spectro-packsConsider the following example for reference.
export OCI_IMAGE_REGISTRY=example.internal.com
export OCI_IMAGE_BASE=spectro-images
export OCI_PACK_REGISTRY=10.10.100.48
export OCI_PACK_BASE=spectro-packs
-
ECR_IMAGE_REGISTRY
: The IP address or domain name of the publicspectro-images
registry. -
ECR_IMAGE_BASE
: The namespace or repository that hosts the images. -
ECR_IMAGE_REGISTRY_REGION
: The AWS region where the ECR registry is located. -
ECR_PACK_REGISTRY
: The IP address or domain name of the publicspectro-packs
registry. -
ECR_PACK_BASE
: The namespace or repository that hosts the packages. -
ECR_PACK_REGISTRY_REGION
: The AWS region where the ECR registry is located.export ECR_IMAGE_REGISTRY=<ecr-endpoint>
export ECR_IMAGE_BASE=spectro-images
export ECR_IMAGE_REGISTRY_REGION=<ecr-region>
export ECR_PACK_REGISTRY=<ecr-endpoint>
export ECR_PACK_BASE=spectro-packs
export ECR_PACK_REGISTRY_REGION=<ecr-region>Consider the following example for reference.
export ECR_IMAGE_REGISTRY=public.ecr.aws/1234567890
export ECR_IMAGE_BASE=spectro-images
export ECR_IMAGE_REGISTRY_REGION=us-east-1
export ECR_PACK_REGISTRY=123456789.dkr.ecr.us-east-1.amazonaws.com
export ECR_PACK_BASE=spectro-packs
export ECR_PACK_REGISTRY_REGION=us-east-1
-
Download the airgap setup binary using the credentials our support team provided along with the binary version. Use the following command template to set up a
VERSION
variable you'll use in the following steps.export VERSION=<x.y.z>
Consider the following example for reference.
export VERSION=4.2.7
-
Use the following command template to download the new Palette airgap installation bin. Enter the username, password, and the Palette airgap installation URL you received from our support team. In the output file name, replace
<version>
with the Palette version you're downloading.curl --user <username>:<password> --output airgap-<version>.bin <url-to-airgap-installation-bin>
Consider the following command example for reference.
curl --user <username>:<password> --output airgap-4.2.7.bin https://software.spectrocloud.com/airgap-v4.2.7.bin
-
Use the following command template to execute the new Palette airgap installation bin.
chmod +x airgap-v$VERSION.bin && ./airgap-v$VERSION.bin
After the binary execution is done, you should receive the following success message (the example is shortened for brevity). As a result of starting the binary, you should have a
spectro-manifests
archive in your temporary directory.Verifying archive integrity... 100% MD5 checksums are OK. All good.
Uncompressing Airgap Setup - Version 4.0.17 100%
Setting up Packs
- Pushing Pack cni-calico:3.25.1
...
Setting up Images
- Pushing image docker.io/kindest/kindnetd:v20230227-15197099
- Pushing image gcr.io/cloud-provider-vsphere/cpi/release/manager:v1.22.8
...
Preparing Manifests Archive
Manifests are available in /tmp/spectro-manifests-1696971110.zip. Extract the archive to a file server to serve as a Spectro Cloud Repository
Setup Completed -
Move the
spectro-manifests
archive to a directory that your file server can access and use the following command template to unzip it.unzip spectro-manifests-<file-id>.zip -d /target/folder
warningDo not remove or replace the existing files inside your target folder that is served by the file server. The previous content is necessary for the upgrade process.
-
Refer to the Additional Packs page and update the packages you are currently using. You must update each package separately.
infoDepending on your underlying infrastructure provider and Kubernetes distribution, you may need to modify the following Palette upgrade steps to match your environment.
-
Navigate to the directory with the Palette installation zip file. Unzip the file to a palette-install directory.
unzip release-*.zip -d palette-install
-
Navigate to the release directory inside palette-install.
cd palette-install/charts/release-*
-
In a code editor of your choice, open the extras/cert-manager/values.yaml file and replace the
controllerImage
,webhookImage
, andamceResolverImage
image URLs with your OCI image registry URLs.image:
cainjectorImage: "gcr.io/spectro-images-public/release-fips/jetstack/cert-manager-cainjector:spectro-v1.11.0-20230427"
controllerImage: "<your-oci-registry-url>/spectro-images-public/release-fips/jetstack/cert-manager-controller:spectro-v1.11.0-20230427"
webhookImage: "<your-oci-registry-url>/spectro-images-public/release-fips/jetstack/cert-manager-webhook:spectro-v1.11.0-20230808"
amceResolverImage: "<your-oci-registry-url>/spectro-images-public/release-fips/jetstack/cert-manager-acmesolver:spectro-v1.11.0-20230427"
featureGates: "AdditionalCertificateOutputFormats=true"Consider the following example for reference.
image:
cainjectorImage: "gcr.io/spectro-images-public/release-fips/jetstack/cert-manager-cainjector:spectro-v1.11.0-20230427"
controllerImage: "harbor.docs.spectro.dev/spectro-images-public/release-fips/jetstack/cert-manager-controller:spectro-v1.11.0-20230427"
webhookImage: "harbor.docs.spectro.dev/spectro-images-public/release-fips/jetstack/cert-manager-webhook:spectro-v1.11.0-20230808"
amceResolverImage: "harbor.docs.spectro.dev/spectro-images-public/release-fips/jetstack/cert-manager-acmesolver:spectro-v1.11.0-20230427"
featureGates: "AdditionalCertificateOutputFormats=true" -
Update the cert-manager chart using the following command.
helm upgrade --values extras/cert-manager/values.yaml \
cert-manager extras/cert-manager/cert-manager-*.tgz --installYou should receive an output similar to the following.
Release "cert-manager" has been upgraded. Happy Helming!
NAME: cert-manager
LAST DEPLOYED: Thu Feb 22 19:42:33 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None -
Prepare the Palette configuration file
values.yaml
. If you savedvalues.yaml
used during the Palette installation, you can reuse it for the upgrade. Alternatively, follow the Kubernetes Installation Instructions to populate yourvalues.yaml
.warningEnsure that the
values.yaml
file is ready before proceeding. Specifically, make sure that theociPackEcrRegistry
andociImageRegistry
configurations include the parameters necessary to interact with yourspectro-images
andspectro-packs
repositories. -
Upgrade the image-swap chart with the following command. Point to the
palette/values.yaml
file from step twelve.helm upgrade --values palette/values.yaml \
image-swap extras/image-swap/image-swap-*.tgz --installYou should receive an output similar to the following.
Release "image-swap" has been upgraded. Happy Helming!
NAME: image-swap
LAST DEPLOYED: Thu Feb 22 19:44:13 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None -
Upgrade the reach-system chart with the following command. Point to the
palette/values.yaml
file from step twelve.helm upgrade --values palette/values.yaml \
reach-system extras/reach-system/reach-system-\*.tgz --installYou should receive an output similar to the following.
Release "reach-system" has been upgraded. Happy Helming!
NAME: reach-system
LAST DEPLOYED: Thu Feb 22 19:47:10 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None -
Upgrade Palette with the following command.
helm upgrade --values palette/values.yaml \
hubble palette/spectro-mgmt-plane-\*.tgz --installYou should receive an output similar to the following.
Release "hubble" has been upgraded. Happy Helming!
NAME: hubble
LAST DEPLOYED: Thu Feb 22 20:05:24 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None -
Use the following command to track the upgrade process.
kubectl get pods --all-namespaces --watch
tipFor a more user-friendly experience, consider using K9s or a similar tool to track the upgrade.
The upgrade usually takes up to five minutes. Palette is upgraded when the deployments in the namespaces
cp-system
,hubble-system
,ingress-nginx
,jet-system
, andui-system
are in the Ready status.
Validate
-
In your terminal, issue the following command.
helm ls
You should receive an output with the version and other details of the currently deployed apps. Check the
App Version
column ofcert-manager
,image-swap
,reach-system
, andhubble
to verify that they have the expected versions.NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
cert-manager default 2 2024-02-22 19:42:33.776829 +0100 CET deployed cert-manager-1.11.0 1.11.0
image-swap default 2 2024-02-22 19:44:13.209592 +0100 CET deployed image-swap-v1.5.2-spectro-4.1.1 1.5.2
reach-system default 2 2024-02-22 19:47:10.558061 +0100 CET deployed reach-system-4.2.0 4.2.0
hubble default 2 2024-02-22 19:47:24.085305 +0100 CET deployed spectro-mgmt-plane-4.2.7 4.2.7 -
Issue the following command to verify that the Palette clusters work as expected.
kubectl get pods --all-namespaces --output custom-columns="NAMESPACE:metadata.namespace,NAME:metadata.name,STATUS:status.phase" \
| grep -E '^(cp-system|hubble-system|ingress-nginx|jet-system|ui-system)\s'The command should return a list of deployments in the
cp-system
,hubble-system
,ingress-nginx
,jet-system
, andui-system
namespaces. All deployments should have the statusRunning
.cp-system spectro-cp-ui-689984f88d-54wsw Running
hubble-system auth-85b748cbf4-6drkn Running
hubble-system auth-85b748cbf4-dwhw2 Running
hubble-system cloud-fb74b8558-lqjq5 Running
hubble-system cloud-fb74b8558-zkfp5 Running
hubble-system configserver-685fcc5b6d-t8f8h Running
hubble-system event-68568f54c7-jzx5t Running
hubble-system event-68568f54c7-w9rnh Running
hubble-system foreq-6b689f54fb-vxjts Running
hubble-system hashboard-897bc9884-pxpvn Running
hubble-system hashboard-897bc9884-rmn69 Running
hubble-system hutil-6d7c478c96-td8q4 Running
hubble-system hutil-6d7c478c96-zjhk4 Running
hubble-system mgmt-85dbf6bf9c-jbggc Running
hubble-system mongo-0 Running
hubble-system mongo-1 Running
hubble-system mongo-2 Running
hubble-system msgbroker-6c9b9fbf8b-mcsn5 Running
hubble-system oci-proxy-7789cf9bd8-qcjkl Running
hubble-system packsync-28205220-bmzcg Succeeded
hubble-system spectrocluster-6c57f5775d-dcm2q Running
hubble-system spectrocluster-6c57f5775d-gmdt2 Running
hubble-system spectrocluster-6c57f5775d-sxks5 Running
hubble-system system-686d77b947-8949z Running
hubble-system system-686d77b947-cgzx6 Running
hubble-system timeseries-7865bc9c56-5q87l Running
hubble-system timeseries-7865bc9c56-scncb Running
hubble-system timeseries-7865bc9c56-sxmgb Running
hubble-system user-5c9f6c6f4b-9dgqz Running
hubble-system user-5c9f6c6f4b-hxkj6 Running
ingress-nginx ingress-nginx-controller-2txsv Running
ingress-nginx ingress-nginx-controller-55pk2 Running
ingress-nginx ingress-nginx-controller-gmps9 Running
jet-system jet-6599b9856d-t9mr4 Running
ui-system spectro-ui-76ffdf67fb-rkgx8 Running