Upgrade Airgap Palette VerteX Installed with Kubernetes

This guide takes you through the process of upgrading a self-hosted airgap Palette VerteX instance installed on Kubernetes.

Upgrade Notes

All Versions

Before upgrading Palette VerteX to a new major version, you must first update it to the latest patch version of the latest minor version available. Refer to the Supported Upgrade Paths section for details.
If your setup includes a PCG, you must also allow the PCG to upgrade automatically before each major or minor Palette VerteX upgrade.

Specific Versions

(pre-4.9.38 to 4.9.38+, proxy environments only) If you are upgrading Palette VerteX from a version earlier than 4.9.38 and your instance uses the reach-system chart for network proxy support, the first upgrade to reach-system-4.9.38-v4 or later requires a one-time migration. Earlier chart versions created the reach-controller-manager Deployment through a Helm pre-install hook, which did not record Helm ownership metadata on the Deployment. As a result, a plain helm upgrade cannot update it and, with Helm 3.14 or later, fails with an ownership-import error.

To adopt the existing Deployment into the release, add the --take-ownership flag to the standard reach-system upgrade command. The --take-ownership flag requires Helm 3.17 or later.
```
helm upgrade --values <values.yaml> \
reach-system extras/reach-system/reach-system-*.tgz --install --take-ownership
```
After the upgrade completes, remove the leftover hook annotations from the adopted Deployment.
```
kubectl --namespace reach-system annotate deployment reach-controller-manager \
  helm.sh/hook- helm.sh/hook-weight- --overwrite
```
Subsequent helm upgrade invocations of the reach-system chart do not require --take-ownership.
(pre-4.9.14 to 4.9.14+) If your self-hosted Palette VerteX environment is installed on a GKE cluster and you are upgrading Palette to version 4.9.14 or later, the principal that runs helm upgrade must have the following Cloud Identity and Access Management (IAM) permissions so the upgrade can prune leftover ingress-nginx Role-Based Access Control (RBAC) resources:
- container.roles.delete
- container.roleBindings.delete
- container.clusterRoles.delete
- container.clusterRoleBindings.delete
If the upgrade principal is missing these permissions, the upgrade fails to remove the deprecated ingress-nginx RBAC objects.
(4.8.47 - 4.9.8 to 4.9.14+) The ingress.type and ingress.ingress.internal fields have been removed from the Palette Helm chart. Any references that remain in your override file are ignored; however, we recommend removing both fields for hygiene purposes.
(pre-4.8.47 to 4.9.14+) If you are upgrading from a version of Palette VerteX earlier than 4.8.47 to 4.9.14 or later, you must update your DNS records after upgrading. Beginning with version 4.8.47, Traefik replaced Ingress Nginx as the primary ingress controller. The upgrade creates a new service of type LoadBalancer in the ingress-traefik namespace with a new external IP address or hostname. If you do not update your DNS records, Palette VerteX will be unreachable at its custom domain.

Use the following command to retrieve the new Traefik load balancer address.
```
kubectl get service traefik-ingress-controller --namespace ingress-traefik \
--output jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}'
```
Update your DNS records for your Palette VerteX domain to resolve to the address returned by the previous command. The specific steps to update DNS records depend on your DNS provider. After updating DNS, verify that your domain resolves to the new address.
```
nslookup <your-palette-domain>
```
(Any to 4.8.47 - 4.9.14) If upgrading an IP-based environment to any version between 4.8.47 - 4.9.14, Palette may replace custom Transport Security Layer (TLS) certificates with newly generated certificates. This results in Edge hosts and other clients that trusted the original, custom certificates to stop communicating with Palette. Before upgrading, we recommend backing up your existing certificates so that you can restore them through the system console. Refer to Scenario - Custom Certificate Replaced After Upgrade for more information.
(4.7.x to 4.8.x) If the MongoDB data for your clusters was first created on Palette VerteX 4.5.x through 4.6.9, the MongoDB Feature Compatibility Version (FCV) might be at 6.0, which causes the upgrade to fail. Your current Palette VerteX version does not indicate whether this applies to your cluster; you must check the FCV directly. You must raise the FCV to 7.0 before upgrading because Palette VerteX 4.8 uses MongoDB 8.0.

To learn how to check the FCV version and upgrade to FCV 7.0 if necessary, refer to Scenario - MongoDB Feature Compatibility Version Mismatch after Palette Upgrade.

Prerequisites

warning

Use Helm client v3.14.0 or later. Older Helm clients do not wait for pre-upgrade hook resources to finish terminating before recreating them. When you run helm upgrade against the hubble chart with a client earlier than v3.14.0, the hubble-system namespace can be inadvertently deleted mid-upgrade, which results in loss of MongoDB data — including tenants and cluster metadata — with no automatic recovery.

The fix for this hook behavior was released in Helm v3.14.0. Confirm your client version with helm version before you continue.

Access to the x86 Linux jumpbox or bastion host with connectivity to the target platform where you have installed Palette VerteX.
An OCI registry such as Harbor or AWS ECR configured and available to store the new Palette VerteX images and packs.
Access to the latest Palette VerteX airgap setup binary. Refer to Access Palette VerteX for more details.
kubectl and helm available in your system. Helm must be v3.14.0 or later.
Access to the kubeconfig file of the target Kubernetes cluster. You must be able to interact with the cluster through kubectl and have sufficient permissions to upgrade Palette VerteX. We recommend using a role with the cluster-admin permissions.
unzip or a similar tool available in your system.
Access to the latest Palette VerteX Helm Chart. Refer to Access Palette VerteX for more details.

Upgrade

Log in to the Linux environment from which you can access your self-hosted airgap Palette VerteX instance.
Authenticate with your OCI registry and get credentials to your spectro-packs and spectro-images repositories.

info
Be mindful of the auth token timeout. Uploading Palette VerteX images and packages to the OCI registry can take about an hour. If your auth token expires before everything is uploaded, you will need to re-authenticate and start the upload again.

Select one of the following tabs for instructions on how to authenticate with Harbor and AWS ECR.
- Harbor
- AWS ECR
Use the following command template to authenticate with your Harbor OCI registry through oras. Refer to the oras login docs for more details about the available CLI flags and examples.
oras login X.X.X.X --username 'yourUserNameHere' --password 'yourPasswordHere'
tip
If your Harbor registry has a self-signed certificate, use the --insecure flag.
oras login X.X.X.X --insecure --username 'yourUserNameHere' --password 'yourPasswordHere'
To get the authentication commands for your specific registries, navigate to the AWS ECR console and click View push commands. Refer to the AWS ECR Authentication docs for more details.
Alternatively, you can use the following command templates:
- Authenticate with your private spectro-packs registry:
  
  <ecr-region>: Replace with the region where your ECR repository is hosted.
  
  <aws-account-id>: Replace with your AWS account ID.
  
  aws ecr get-login-password --region <ecr-region> | oras login --username AWS --password-stdin <aws-account-id>.dkr.ecr.<ecr-region>.amazonaws.com
  
  Consider the following example for reference.
  
  aws ecr get-login-password --region us-east-1 | oras login --username AWS --password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com
- Authenticate with your public spectro-images registry:
  
  <ecr-region>: Replace with the region where your ECR repository is hosted.
  
  <public-ecr-url>: Replace with the URL to your public ECR registry.
  
  aws ecr-public get-login-password --region <ecr-region> | docker login --username AWS --password-stdin <public-ecr-url>
  
  Consider the following example for reference.
  
  aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/dfa987f
Prepare and export the environment variables required by the Palette VerteX airgap binary. Select one of the following tabs for instructions on how to prepare these variables for Harbor and AWS ECR.
- Harbor
- AWS ECR
- OCI_IMAGE_REGISTRY: The IP address or domain name of the spectro-images registry.
- OCI_IMAGE_BASE: The namespace or repository that hosts the images.
- OCI_PACK_REGISTRY: The IP address or domain name of the spectro-packs registry.
- OCI_PACK_BASE: The namespace or repository that hosts the packages.
  
  export OCI_IMAGE_REGISTRY=<harbor-endpoint> export OCI_IMAGE_BASE=spectro-images export OCI_PACK_REGISTRY=<harbor-endpoint> export OCI_PACK_BASE=spectro-packs
  
  Consider the following example.
  
  Example values
  export OCI_IMAGE_REGISTRY=example.internal.com export OCI_IMAGE_BASE=spectro-images export OCI_PACK_REGISTRY=10.10.100.48 export OCI_PACK_BASE=spectro-packs
- ECR_IMAGE_REGISTRY: The IP address or domain name of the public spectro-images registry.
- ECR_IMAGE_BASE: The namespace or repository that hosts the images.
- ECR_IMAGE_REGISTRY_REGION: The AWS region where the ECR registry is located.
- ECR_PACK_REGISTRY: The IP address or domain name of the public spectro-packs registry.
- ECR_PACK_BASE: The namespace or repository that hosts the packages.
- ECR_PACK_REGISTRY_REGION: The AWS region where the ECR registry is located.
  
  export ECR_IMAGE_REGISTRY=<ecr-endpoint> export ECR_IMAGE_BASE=spectro-images export ECR_IMAGE_REGISTRY_REGION=<ecr-region> export ECR_PACK_REGISTRY=<ecr-endpoint> export ECR_PACK_BASE=spectro-packs export ECR_PACK_REGISTRY_REGION=<ecr-region>
  
  Consider the following example.
  
  Example values
  export ECR_IMAGE_REGISTRY=public.ecr.aws/1234567890 export ECR_IMAGE_BASE=spectro-images export ECR_IMAGE_REGISTRY_REGION=us-east-1 export ECR_PACK_REGISTRY=123456789.dkr.ecr.us-east-1.amazonaws.com export ECR_PACK_BASE=spectro-packs export ECR_PACK_REGISTRY_REGION=us-east-1
Download the airgap setup binary using the credentials our support team provided along with the binary version. Use the following command template to set up a VERSION variable you'll use in the following steps.
```
export VERSION=<x.y.z>
```
Consider the following example for reference.
```
export VERSION=4.2.7
```
Use the following command template to download the new Palette VerteX airgap installation bin. Enter the username, password, and the Palette VerteX airgap installation URL you received from our support team. In the output file name, replace <version> with the Palette VerteX version you're downloading.
```
curl --user <username>:<password> --output airgap-<version>.bin <url-to-airgap-installation-bin>
```
Consider the following command example for reference.
```
curl --user <username>:<password> --output airgap-4.2.7.bin https://software.spectrocloud.com/airgap-v4.2.7.bin
```

Use the following command template to execute the new Palette VerteX airgap installation bin.

chmod +x airgap-v$VERSION.bin && ./airgap-v$VERSION.bin

After the binary execution is done, you should receive the following success message (the example is shortened for brevity). As a result of starting the binary, you should have a spectro-manifests archive in your temporary directory.

Example output
Verifying archive integrity...  100%   MD5 checksums are OK. All good.
Uncompressing Airgap Setup - Version 4.0.17  100%
Setting up Packs
- Pushing Pack cni-calico:3.25.1
...
Setting up Images
- Pushing image docker.io/kindest/kindnetd:v20230227-15197099
...
Preparing Manifests Archive
Manifests are available in /tmp/spectro-manifests-1696971110.zip.
Setup Completed

Refer to the Additional Packs page and update the packages you are currently using. You must update each package separately.

info
Depending on your underlying infrastructure provider and Kubernetes distribution, you may need to modify the following Palette VerteX upgrade steps to match your environment.
Navigate to the directory where you downloaded the Palette VerteX install zip file provided by our support team. Unzip the file to a directory named vertex-install.
```
unzip charts.zip -d vertex-install
```
Navigate to the vertex-install directory.
```
cd vertex-install
```

Open the file extras/cert-manager/values.yaml with a text editor of your choice. This example uses Vim.

vim extras/cert-manager/values.yaml

Append <your-registry-url> to each image, along with the <repository> where you want to store your images.

image:
  cainjectorImage: "<your-registry-url>/<repository>/us-docker.pkg.dev/palette-images-fips/palette/spectro-cert-manager/cert-manager-cainjector:v1.19.3-spectro-4.8.b"
  controllerImage: "<your-registry-url>/<repository>/us-docker.pkg.dev/palette-images-fips/palette/spectro-cert-manager/cert-manager-controller:v1.19.3-spectro-4.8.b"
  webhookImage: "<your-registry-url>/<repository>/us-docker.pkg.dev/palette-images-fips/palette/spectro-cert-manager/cert-manager-webhook:v1.19.3-spectro-4.8.b"
  amceResolverImage: "<your-registry-url>/<repository>/us-docker.pkg.dev/palette-images-fips/palette/spectro-cert-manager/cert-manager-acmesolver:v1.19.3-spectro-4.8.b"

In the example below, we used harbor.docs.spectro.dev for the registry and spectro-images for the repository.

Example output
image:
  cainjectorImage: "harbor.docs.spectro.dev/spectro-images/us-docker.pkg.dev/palette-images-fips/palette/spectro-cert-manager/cert-manager-cainjector:v1.19.3-spectro-4.8.b"
  controllerImage: "harbor.docs.spectro.dev/spectro-images/us-docker.pkg.dev/palette-images-fips/palette/spectro-cert-manager/cert-manager-controller:v1.19.3-spectro-4.8.b"
  webhookImage: "harbor.docs.spectro.dev/spectro-images/us-docker.pkg.dev/palette-images-fips/palette/spectro-cert-manager/cert-manager-webhook:v1.19.3-spectro-4.8.b"
  amceResolverImage: "harbor.docs.spectro.dev/spectro-images/us-docker.pkg.dev/palette-images-fips/palette/spectro-cert-manager/cert-manager-acmesolver:v1.19.3-spectro-4.8.b"

Update the cert-manager chart using the following command.

helm upgrade --install cert-manager \
  ./extras/cert-manager/cert-manager-*.tgz \
  --namespace cert-manager \
  --create-namespace \
  --values ./extras/cert-manager/values.yaml

You should receive an output similar to the following.

Release "cert-manager" has been upgraded. Happy Helming!
NAME: cert-manager
LAST DEPLOYED: Thu Feb 22 19:42:33 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

Upgrade the Spectro Management CRDs chart.

helm upgrade --install spectro-mgmt-crds \
  extras/spectro-mgmt-crds/spectro-mgmt-crds-*.tgz \
  --values extras/spectro-mgmt-crds/values.yaml

You should receive an output similar to the following.

Release "spectro-mgmt-crds" has been upgraded. Happy Helming!
NAME: spectro-mgmt-crds
LAST DEPLOYED: Thu Feb 22 19:43:00 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

Prepare the Palette VerteX configuration file values.yaml. If you saved values.yaml used during the Palette VerteX installation, you can reuse it for the upgrade. Alternatively, follow the Kubernetes Installation Instructions to populate your values.yaml.

warning
Ensure that the values.yaml file is ready before proceeding. Specifically, make sure that the ociPackEcrRegistry and ociImageRegistry configurations include the parameters necessary to interact with your spectro-images and spectro-packs repositories.

(Self-hosted OCI registry only) If you use image swap for self-hosted OCI registries, upgrade the image-swap chart with the following command. Point to the vertex/values.yaml file from step 13.

helm upgrade --values vertex/values.yaml \
image-swap extras/image-swap/image-swap-*.tgz --install

You should receive an output similar to the following.

Release "image-swap" has been upgraded. Happy Helming!
NAME: image-swap
LAST DEPLOYED: Thu Feb 22 19:44:13 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

(Proxy environments only) If you are upgrading a Palette VerteX instance in an environment where a network proxy must be configured for Palette VerteX to access the internet, upgrade the reach-system chart with the following command. Point to the vertex/values.yaml file from step 13.
```
helm upgrade --values vertex/values.yaml \
reach-system extras/reach-system/reach-system-*.tgz --install
```
You should receive an output similar to the following.
```
Release "reach-system" has been upgraded. Happy Helming!
NAME: reach-system
LAST DEPLOYED: Thu Feb 22 19:47:10 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
```

Upgrade Palette VerteX with the following command.

helm upgrade --values vertex/values.yaml \
hubble vertex/spectro-mgmt-plane-*.tgz --install

You should receive an output similar to the following.

Release "hubble" has been upgraded. Happy Helming!
NAME: hubble
LAST DEPLOYED: Thu Feb 22 20:05:24 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

Use the following command to track the upgrade process.
```
kubectl get pods --all-namespaces --watch
```
tip
For a more user-friendly experience, consider using K9s or a similar tool to track the upgrade.

The upgrade usually takes up to five minutes. Palette VerteX is upgraded when the deployments in the namespaces cp-system, hubble-system, ingress-traefik, jet-system, and ui-system are in the Ready status.

Validate

In your terminal, issue the following command.

helm ls

You should receive an output with the version and other details of the currently deployed apps. Check the App Version column of cert-manager, image-swap, reach-system, and hubble to verify that they have the expected versions.

NAME        	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART                   	APP VERSION
cert-manager	default  	2       	2024-02-22 19:42:33.776829 +0100 CET	deployed	cert-manager-1.11.0     	1.11.0
image-swap  	default  	2       	2024-02-22 19:44:13.209592 +0100 CET	deployed	image-swap-v1.5.2-spectro-4.1.1	1.5.2
reach-system	default  	2       	2024-02-22 19:47:10.558061 +0100 CET	deployed	reach-system-4.2.0             	4.2.0
hubble      	default  	2       	2024-02-22 19:47:24.085305 +0100 CET	deployed	spectro-mgmt-plane-4.2.7	4.2.7

Issue the following command to verify that the Palette VerteX clusters work as expected.

kubectl get pods --all-namespaces --output custom-columns="NAMESPACE:metadata.namespace,NAME:metadata.name,STATUS:status.phase" \
| grep --extended-regexp '^(cp-system|hubble-system|ingress-traefik|jet-system|ui-system)\s'

The command should return a list of deployments in the cp-system, hubble-system, ingress-traefik, jet-system, and ui-system namespaces. All deployments should have the status Running.

cp-system        spectro-cp-ui-689984f88d-54wsw             Running
hubble-system    auth-85b748cbf4-6drkn                      Running
hubble-system    auth-85b748cbf4-dwhw2                      Running
hubble-system    cloud-fb74b8558-lqjq5                      Running
hubble-system    cloud-fb74b8558-zkfp5                      Running
hubble-system    configserver-685fcc5b6d-t8f8h              Running
hubble-system    event-68568f54c7-jzx5t                     Running
hubble-system    event-68568f54c7-w9rnh                     Running
hubble-system    foreq-6b689f54fb-vxjts                     Running
hubble-system    hashboard-897bc9884-pxpvn                  Running
hubble-system    hashboard-897bc9884-rmn69                  Running
hubble-system    hutil-6d7c478c96-td8q4                     Running
hubble-system    hutil-6d7c478c96-zjhk4                     Running
hubble-system    mgmt-85dbf6bf9c-jbggc                      Running
hubble-system    mongo-0                                    Running
hubble-system    mongo-1                                    Running
hubble-system    mongo-2                                    Running
hubble-system    msgbroker-6c9b9fbf8b-mcsn5                 Running
hubble-system    oci-proxy-7789cf9bd8-qcjkl                 Running
hubble-system    packsync-28205220-bmzcg                    Succeeded
hubble-system    spectrocluster-6c57f5775d-dcm2q            Running
hubble-system    spectrocluster-6c57f5775d-gmdt2            Running
hubble-system    spectrocluster-6c57f5775d-sxks5            Running
hubble-system    system-686d77b947-8949z                    Running
hubble-system    system-686d77b947-cgzx6                    Running
hubble-system    timeseries-7865bc9c56-5q87l                Running
hubble-system    timeseries-7865bc9c56-scncb                Running
hubble-system    timeseries-7865bc9c56-sxmgb                Running
hubble-system    user-5c9f6c6f4b-9dgqz                      Running
hubble-system    user-5c9f6c6f4b-hxkj6                      Running
ingress-traefik  traefik-ingress-controller-9dmzq           Running
ingress-traefik  traefik-ingress-controller-tpwtf           Running
ingress-traefik  traefik-ingress-controller-xz4jf           Running
jet-system       jet-6599b9856d-t9mr4                       Running
ui-system        spectro-ui-76ffdf67fb-rkgx8                Running

Upgrade Notes​

All Versions​

Specific Versions​

Prerequisites​

Upgrade​

Validate​

Upgrade Notes

All Versions

Specific Versions

Prerequisites

Upgrade

Validate