Skip to main content

Upgrade Palette VerteX Installed with Kubernetes

This guide takes you through the process of upgrading a self-hosted Palette VerteX instance installed with Helm on Kubernetes.

Upgrade Notes​

All Versions​

  • Before upgrading Palette VerteX to a new major version, you must first update it to the latest patch version of the latest minor version available. Refer to the Supported Upgrade Paths section for details.

  • If your setup includes a PCG, you must also allow the PCG to upgrade automatically before each major or minor Palette VerteX upgrade.

Specific Versions​

  • (pre-4.9.38 to 4.9.38+, proxy environments only) If you are upgrading Palette VerteX from a version earlier than 4.9.38 and your instance uses the reach-system chart for network proxy support, the first upgrade to reach-system-4.9.38-v4 or later requires a one-time migration. Earlier chart versions created the reach-controller-manager Deployment through a Helm pre-install hook, which did not record Helm ownership metadata on the Deployment. As a result, a plain helm upgrade cannot update it and, with Helm 3.14 or later, fails with an ownership-import error.

    To adopt the existing Deployment into the release, add the --take-ownership flag to the standard reach-system upgrade command. The --take-ownership flag requires Helm 3.17 or later.

    helm upgrade --values <values.yaml> \
    reach-system extras/reach-system/reach-system-*.tgz --install --take-ownership

    After the upgrade completes, remove the leftover hook annotations from the adopted Deployment.

    kubectl --namespace reach-system annotate deployment reach-controller-manager \
      helm.sh/hook- helm.sh/hook-weight- --overwrite

    Subsequent helm upgrade invocations of the reach-system chart do not require --take-ownership.

  • (pre-4.9.14 to 4.9.14+) If your self-hosted Palette VerteX environment is installed on a GKE cluster and you are upgrading Palette to version 4.9.14 or later, the principal that runs helm upgrade must have the following Cloud Identity and Access Management (IAM) permissions so the upgrade can prune leftover ingress-nginx Role-Based Access Control (RBAC) resources:

    • container.roles.delete
    • container.roleBindings.delete
    • container.clusterRoles.delete
    • container.clusterRoleBindings.delete

    If the upgrade principal is missing these permissions, the upgrade fails to remove the deprecated ingress-nginx RBAC objects.

  • (4.8.47 - 4.9.8 to 4.9.14+) The ingress.type and ingress.ingress.internal fields have been removed from the Palette Helm chart. Any references that remain in your override file are ignored; however, we recommend removing both fields for hygiene purposes.

  • (pre-4.8.47 to 4.9.14+) If you are upgrading from a version of Palette VerteX earlier than 4.8.47 to 4.9.14 or later, you must update your DNS records after upgrading. Beginning with version 4.8.47, Traefik replaced Ingress Nginx as the primary ingress controller. The upgrade creates a new service of type LoadBalancer in the ingress-traefik namespace with a new external IP address or hostname. If you do not update your DNS records, Palette VerteX will be unreachable at its custom domain.

    Use the following command to retrieve the new Traefik load balancer address.

    kubectl get service traefik-ingress-controller --namespace ingress-traefik \
    --output jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}'

    Update your DNS records for your Palette VerteX domain to resolve to the address returned by the previous command. The specific steps to update DNS records depend on your DNS provider. After updating DNS, verify that your domain resolves to the new address.

    nslookup <your-palette-domain>

  • (Any to 4.8.47 - 4.9.14) If upgrading an IP-based environment to any version between 4.8.47 - 4.9.14, Palette may replace custom Transport Security Layer (TLS) certificates with newly generated certificates. This results in Edge hosts and other clients that trusted the original, custom certificates to stop communicating with Palette. Before upgrading, we recommend backing up your existing certificates so that you can restore them through the system console. Refer to Scenario - Custom Certificate Replaced After Upgrade for more information.

  • (4.7.x to 4.8.x) If the MongoDB data for your clusters was first created on Palette VerteX 4.5.x through 4.6.9, the MongoDB Feature Compatibility Version (FCV) might be at 6.0, which causes the upgrade to fail. Your current Palette VerteX version does not indicate whether this applies to your cluster; you must check the FCV directly. You must raise the FCV to 7.0 before upgrading because Palette VerteX 4.8 uses MongoDB 8.0.

    To learn how to check the FCV version and upgrade to FCV 7.0 if necessary, refer to Scenario - MongoDB Feature Compatibility Version Mismatch after Palette Upgrade.

Prerequisites​

warning

Use Helm client v3.14.0 or later. Older Helm clients do not wait for pre-upgrade hook resources to finish terminating before recreating them. When you run helm upgrade against the hubble chart with a client earlier than v3.14.0, the hubble-system namespace can be inadvertently deleted mid-upgrade, which results in loss of MongoDB data — including tenants and cluster metadata — with no automatic recovery.

The fix for this hook behavior was released in Helm v3.14.0. Confirm your client version with helm version before you continue.

  • kubectl and helm available in your system. Helm must be v3.14.0 or later.

  • Access to the kubeconfig file of the target Kubernetes cluster. You must be able to interact with the cluster through kubectl and have sufficient permissions to upgrade Palette VerteX. We recommend using a role with the cluster-admin permissions.

  • unzip or a similar tool available in your system.

  • Access to the latest Palette VerteX Helm Chart. Refer to Access Palette VerteX for more details.

Upgrade​

info

Depending on your underlying infrastructure provider and Kubernetes distribution, you may need to modify these steps to match your environment.

  1. Open a terminal session and navigate to the directory where you downloaded the Palette VerteX install ZIP file provided by our support team. Unzip the file to a directory named vertex-install.

    unzip charts.zip -d vertex-install

  2. Navigate to the vertex-install directory.

    cd vertex-install

  3. Update the cert-manager chart using the following command.

    helm upgrade --install cert-manager \
      ./extras/cert-manager/cert-manager-*.tgz \
      --namespace cert-manager \
      --create-namespace \
      --values ./extras/cert-manager/values.yaml

    You should receive an output similar to the following.

    Release "cert-manager" has been upgraded. Happy Helming!
    NAME: cert-manager
    LAST DEPLOYED: Thu Feb 22 19:42:33 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None

  4. Upgrade the Spectro Management CRDs chart.

    helm upgrade --install spectro-mgmt-crds \
      extras/spectro-mgmt-crds/spectro-mgmt-crds-*.tgz \
      --values extras/spectro-mgmt-crds/values.yaml

    You should receive an output similar to the following.

    Release "spectro-mgmt-crds" has been upgraded. Happy Helming!
    NAME: spectro-mgmt-crds
    LAST DEPLOYED: Thu Feb 22 19:43:00 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None

  5. Prepare the Palette VerteX configuration file values.yaml. If you saved values.yaml used during the Palette VerteX installation, you can reuse it for the upgrade. Alternatively, follow the Kubernetes Installation Instructions to populate your values.yaml.

    warning

    Ensure that the values.yaml file is ready before proceeding. If you are using a self-hosted OCI registry, make sure that the ociImageRegistry.mirrorRegistries parameter in your values.yaml includes the necessary mirror links.

    Include /v2 in your endpoints if you are using a Harbor registry with a proxy cache project. Harbor proxy cache projects use /v2 as part of their internal URL routing for cached images. For all other registries, omit /v2, as the container runtime automatically appends /v2 when making API calls. Including /v2 for non-proxy-cache registries results in a doubled /v2/v2/ path, which causes image pull failures. For example:

  • Ensure that the values.yaml file is ready before proceeding. If you are using a self-hosted OCI registry, make sure that the ociImageRegistry.mirrorRegistries parameter in your values.yaml includes the necessary mirror links.

  • Include /v2 in your endpoints if you are using a Harbor registry with a proxy cache project. Harbor proxy cache projects use /v2 as part of their internal URL routing for cached images. For all other registries, omit /v2, as the container runtime automatically appends /v2 when making API calls.

    Including /v2 for non-proxy-cache registries results in a doubled /v2/v2/ path, which causes image pull failures. For example: docker.io::harbor.example.org/v2/proxy-cache-project/docker.io.

    :::

  1. (Self-hosted OCI registry only) If you use image swap for self-hosted OCI registries, upgrade the image-swap chart with the following command. Point to the vertex/values.yaml file from step 5.

    helm upgrade --values vertex/values.yaml \
    image-swap extras/image-swap/image-swap-*.tgz --install

    You should receive an output similar to the following.

    Release "image-swap" has been upgraded. Happy Helming!
    NAME: image-swap
    LAST DEPLOYED: Thu Feb 22 19:44:13 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None

  2. (Proxy environments only) If you are upgrading a Palette VerteX instance in an environment where a network proxy must be configured for Palette VerteX to access the internet, upgrade the reach-system chart with the following command. Point to the vertex/values.yaml file from step 5.

    helm upgrade --values vertex/values.yaml \
    reach-system extras/reach-system/reach-system-*.tgz --install

    You should receive an output similar to the following.

    Release "reach-system" has been upgraded. Happy Helming!
    NAME: reach-system
    LAST DEPLOYED: Thu Feb 22 19:47:10 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None

  3. Upgrade Palette VerteX with the following command.

    helm upgrade --values vertex/values.yaml \
    hubble vertex/spectro-mgmt-plane-*.tgz --install

    You should receive an output similar to the following.

    Release "hubble" has been upgraded. Happy Helming!
    NAME: hubble
    LAST DEPLOYED: Thu Feb 22 20:05:24 2024
    NAMESPACE: default
    STATUS: deployed
    REVISION: 2
    TEST SUITE: None

  4. Use the following command to track the upgrade process.

    kubectl get pods --all-namespaces --watch
    tip

    For a more user-friendly experience, consider using K9s or a similar tool to track the upgrade.

    The upgrade usually takes up to five minutes. Palette VerteX is upgraded when the deployments in the namespaces cp-system, hubble-system, ingress-traefik, jet-system, and ui-system are in the Ready status.

Validate​

  1. In your terminal, issue the following command.

    helm ls

    You should receive an output with the version and other details of the currently deployed apps. Check the App Version column of cert-manager, image-swap, reach-system, and hubble to verify that they have the expected versions.

    NAME        	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART                   	APP VERSION
    cert-manager	default  	2       	2024-02-22 19:42:33.776829 +0100 CET	deployed	cert-manager-1.11.0     	1.11.0
    image-swap  	default  	2       	2024-02-22 19:44:13.209592 +0100 CET	deployed	image-swap-v1.5.2-spectro-4.1.1	1.5.2
    reach-system	default  	2       	2024-02-22 19:47:10.558061 +0100 CET	deployed	reach-system-4.2.0             	4.2.0
    hubble      	default  	2       	2024-02-22 19:47:24.085305 +0100 CET	deployed	spectro-mgmt-plane-4.2.7	4.2.7

  2. Issue the following command to verify that the Palette VerteX clusters work as expected.

    kubectl get pods --all-namespaces --output custom-columns="NAMESPACE:metadata.namespace,NAME:metadata.name,STATUS:status.phase" \
    | grep --extended-regexp '^(cp-system|hubble-system|ingress-traefik|jet-system|ui-system)\s'

    The command should return a list of deployments in the cp-system, hubble-system, ingress-traefik, jet-system, and ui-system namespaces. All deployments should have the status Running.

    cp-system        spectro-cp-ui-689984f88d-54wsw             Running
    hubble-system    auth-85b748cbf4-6drkn                      Running
    hubble-system    auth-85b748cbf4-dwhw2                      Running
    hubble-system    cloud-fb74b8558-lqjq5                      Running
    hubble-system    cloud-fb74b8558-zkfp5                      Running
    hubble-system    configserver-685fcc5b6d-t8f8h              Running
    hubble-system    event-68568f54c7-jzx5t                     Running
    hubble-system    event-68568f54c7-w9rnh                     Running
    hubble-system    foreq-6b689f54fb-vxjts                     Running
    hubble-system    hashboard-897bc9884-pxpvn                  Running
    hubble-system    hashboard-897bc9884-rmn69                  Running
    hubble-system    hutil-6d7c478c96-td8q4                     Running
    hubble-system    hutil-6d7c478c96-zjhk4                     Running
    hubble-system    mgmt-85dbf6bf9c-jbggc                      Running
    hubble-system    mongo-0                                    Running
    hubble-system    mongo-1                                    Running
    hubble-system    mongo-2                                    Running
    hubble-system    msgbroker-6c9b9fbf8b-mcsn5                 Running
    hubble-system    oci-proxy-7789cf9bd8-qcjkl                 Running
    hubble-system    packsync-28205220-bmzcg                    Succeeded
    hubble-system    spectrocluster-6c57f5775d-dcm2q            Running
    hubble-system    spectrocluster-6c57f5775d-gmdt2            Running
    hubble-system    spectrocluster-6c57f5775d-sxks5            Running
    hubble-system    system-686d77b947-8949z                    Running
    hubble-system    system-686d77b947-cgzx6                    Running
    hubble-system    timeseries-7865bc9c56-5q87l                Running
    hubble-system    timeseries-7865bc9c56-scncb                Running
    hubble-system    timeseries-7865bc9c56-sxmgb                Running
    hubble-system    user-5c9f6c6f4b-9dgqz                      Running
    hubble-system    user-5c9f6c6f4b-hxkj6                      Running
    ingress-traefik  traefik-ingress-controller-9dmzq           Running
    ingress-traefik  traefik-ingress-controller-tpwtf           Running
    ingress-traefik  traefik-ingress-controller-xz4jf           Running
    jet-system       jet-6599b9856d-t9mr4                       Running
    ui-system        spectro-ui-76ffdf67fb-rkgx8                Running