Skip to main content

Palette Upgrade

We recommend you review the Release Notes and the Upgrade Notes before attempting to upgrade Palette. Use this information to address common issues that may occur during an upgrade.

Scenario - Palette Webhook Downgraded After Rolling Upgrade​

After upgrading a Palette or Palette VerteX deployment to 4.9.38, connected workload clusters may fail to reconcile because the palette-webhook deployment was downgraded during the rolling upgrade of palette-controller-manager in the cluster's namespace. The outgoing controller pod holds the leader lease briefly while the new controller pod starts. During that window, the outgoing pod can re-apply the older palette-webhook manifest baked into its own image, overwriting the newer manifest the incoming pod already applied. The older webhook does not serve the v1alpha1 to v1 conversion that Custom Resource Definitions (CRDs) in the cluster.spectrocloud.com API group now require, so every SpectroCluster reconcile fails and does not self-heal.

Symptoms of an affected cluster include:

  • The cluster appears grey on the Clusters page in the Palette UI.

  • Cluster profile updates do not apply to the cluster.

  • The palette-controller-manager logs record repeated reconcile failures similar to the following.

    Example error
    failed to set owner ref to cloud config: Internal error occurred: conversion webhook for
    cluster.spectrocloud.com/v1alpha1, Kind=AwsCloudConfig failed: the server could not find
    the requested resource

The trigger is the Palette agent upgrade on an existing cluster. New cluster deployments are not affected.

Debug Steps​

Recovery may require restarting both the palette-controller-manager deployment in the affected cluster's namespace and the palette-webhook deployment in the palette-system namespace. Complete the steps in order and check the webhook image after each restart to determine whether the second restart is needed.

  1. Establish a shell session with kubectl access to the affected workload cluster. Refer to Access Cluster with CLI for guidance.

  2. Identify the namespace of the affected cluster. The namespace follows the format cluster-<cluster-uid>, where <cluster-uid> is the identifier displayed on the cluster's Overview tab in the Palette UI.

  3. Compare the image on the palette-webhook deployment against the expected Palette version recorded in the palette-version-info-for-webhook ConfigMap.

    kubectl --namespace palette-system get deployment palette-webhook \
    --output jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
    kubectl --namespace palette-system get configmap palette-version-info-for-webhook \
    --output jsonpath='{.data.VERSION}{"\n"}'
    Example output on an affected cluster
    us-docker.pkg.dev/palette-images/palette/spectro-drive:4.9.11
    4.9.19

    If the two values do not agree, the webhook has been downgraded and the cluster is affected. In the example above, the webhook is still running the 4.9.11 image while the controller expects 4.9.19.

  4. Restart the palette-controller-manager deployment in the affected cluster's namespace. When the outgoing controller pod is gone, the surviving pod reapplies the correct palette-webhook manifest at startup, before it takes the leader lease.

    kubectl --namespace cluster-<cluster-uid> rollout restart deployment palette-controller-manager
    kubectl --namespace cluster-<cluster-uid> rollout status deployment palette-controller-manager \
    --timeout=600s
  5. Re-check the palette-webhook deployment image. When it matches the version recorded in palette-version-info-for-webhook, the cluster begins reconciling successfully.

    kubectl --namespace palette-system get deployment palette-webhook \
    --output jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
    Example output on a recovered cluster
    us-docker.pkg.dev/palette-images/palette/spectro-drive:4.9.19
  6. If the webhook image in step 5 still shows the older version, restart the palette-webhook deployment in the palette-system namespace. Some affected clusters need this second restart before the controller can move the webhook forward.

    kubectl --namespace palette-system rollout restart deployment palette-webhook
    kubectl --namespace palette-system rollout status deployment palette-webhook --timeout=600s

    After the rollout completes, run the check in step 5 again and confirm that the image now matches the expected version.

  7. Confirm that reconcile has recovered. Follow the controller logs and verify that the failed to set owner ref to cloud config error no longer appears on new reconcile attempts.

    kubectl --namespace cluster-<cluster-uid> logs deployment/palette-controller-manager --follow

Scenario - Custom Certificate Replaced After Upgrade​

If upgrading IP-based self-hosted Palette or Palette VerteX environments installed using the Palette Management Appliance, VerteX Management Appliance, or Helm charts to any version between 4.8.47 - 4.9.14, Palette may replace custom Transport Security Layer (TLS) certificates with newly generated certificates. This results in Edge hosts and other clients that trusted the original, custom certificates to stop communicating with Palette.

Before upgrading, we recommend backing up your existing certificates so that you can restore them through the system console.

Debug Steps​

  1. Log in to the system console.

  2. From the left main menu, select Enterprise Cluster.

  3. On the Overview tab, download the Kubernetes Config File.

  4. Open a terminal session in an environment that has network access to the cluster. Set the KUBECONFIG environment variable to the file path of the downloaded kubeconfig file.

    export KUBECONFIG=<path-to-kubeconfig>
  5. Verify that the default-ssl-certificate Secret is present. For Palette versions 4.8.47 and later, the Secret is in the ingress-traefik namespace; for versions earlier than 4.8.47, the Secret is in the ingress-nginx namespace.

    kubectl get secret default-ssl-certificate --namespace ingress-traefik
    Example output
    NAME                      TYPE     DATA   AGE
    default-ssl-certificate Opaque 3 60d
  6. Save a copy of your custom server certificate, private key, and Certificate Authority (CA) certificate.

    kubectl get secret default-ssl-certificate --namespace ingress-traefik --output yaml > default-ssl-certificate-backup.yaml
  7. Upgrade Palette using your standard procedure.

  8. After the upgrade completes, compare the current ca.crt, tls.crt, and tls.key values with your backup.

    kubectl get secret default-ssl-certificate --namespace ingress-traefik --output yaml
    Example output
    apiVersion: v1
    data:
    ca.crt: LS0tLS1CRUd... # output omitted for brevity
    tls.crt: LS0tLS1CRUd... # output omitted for brevity
    tls.key: LS0tLS1CRUd... # output omitted for brevity
    kind: Secret
    metadata:
    creationTimestamp: "2026-04-06T22:28:08Z"
    name: default-ssl-certificate
    namespace: ingress-traefik
    resourceVersion: "339401"
    uid: de363cf1-431a-497d-8ccb-0cff1b1ab609
    type: Opaque
  9. If the certificate values do not match, re-upload the server certificate, private key, and CA certificate through the system console. To do so, return to the system console in your web browser, and from the left main menu, select Administration.

  10. Select the System Address tab.

  11. Copy and paste the values from your backed up file into the appropriate fields.

    YAML FieldUI Field
    tls.crtCertificate
    tls.keyKey
    ca.crtCertificate Authority (Optional)
  12. Select Update to apply the certificate.

  13. Verify that connected Edge hosts and workload clusters resume reporting their status to Palette.

configserver Stuck on init-rootdomain-traefik After Upgrade to 4.9.14+​

After upgrading self-hosted Palette or Palette VerteX to version 4.9.14 or later, the configserver pod may remain in Init:CrashLoopBackOff for the init-rootdomain-traefik init container. This happens when leftover Nginx ingress controller pods on the same nodes hold host ports 80, 443, and 9090, which prevents the new Traefik DaemonSet pods from scheduling. The automatic Nginx cleanup script that runs at configserver startup cannot run because configserver itself never reaches Ready, so the leftover Nginx resources must be removed manually.

This scenario applies only to Enterprise installs of self-hosted Palette and Palette VerteX.

warning

Do not delete the default-ssl-certificate Secret in the ingress-nginx namespace, and do not delete the ingress-nginx namespace itself. The Enterprise cert-bridge continues to copy your uploaded Transport Layer Security (TLS) certificate from the default-ssl-certificate Secret into Traefik on every configserver restart. Deleting either resource breaks ingress TLS.

Debug Steps​

  1. Connect to the cluster using the cluster's kubeconfig file. Refer to the Access Cluster with CLI guide for additional guidance.

  2. Verify that the configserver pod is in the Init:CrashLoopBackOff state and inspect the init container logs.

    kubectl get pod --namespace hubble-system --selector component=configserver
    kubectl logs <configserver-pod> --namespace hubble-system --container init-rootdomain-traefik --previous | tail
    Example output
    NAME                            READY   STATUS                  RESTARTS   AGE
    configserver-685fcc5b6d-t8f8h 0/1 Init:CrashLoopBackOff 5 8m
    level=fatal msg="not able to get the host ip. got NotFound: Resource 'websecure' not found of type 'pod'"
  3. Confirm that the Traefik DaemonSet pods are stuck in Pending because the host ports are unavailable.

    kubectl get pods --namespace ingress-traefik
    Example output
    NAME                               READY   STATUS    RESTARTS   AGE
    traefik-ingress-controller-dkjs5 0/1 Pending 0 8m
    traefik-ingress-controller-nltlh 0/1 Pending 0 8m

    If the scheduler events for those pods show didn't have free ports for the requested pod ports, leftover Nginx pods are holding host ports 80, 443, and 9090 on the same nodes.

  4. Delete the cluster-scoped Nginx resources.

    kubectl delete validatingwebhookconfiguration ingress-nginx-admission --ignore-not-found
    kubectl delete clusterrole ingress-nginx ingress-nginx-admission --ignore-not-found
    kubectl delete clusterrolebinding ingress-nginx ingress-nginx-admission --ignore-not-found
    kubectl delete ingressclass nginx --ignore-not-found
    Example output
    validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted
    clusterrole.rbac.authorization.k8s.io "ingress-nginx" deleted
    clusterrole.rbac.authorization.k8s.io "ingress-nginx-admission" deleted
    clusterrolebinding.rbac.authorization.k8s.io "ingress-nginx" deleted
    clusterrolebinding.rbac.authorization.k8s.io "ingress-nginx-admission" deleted
    ingressclass.networking.k8s.io "nginx" deleted
  5. Delete the Nginx resources inside the ingress-nginx namespace. Do not delete the namespace itself or the default-ssl-certificate Secret.

    kubectl delete daemonset ingress-nginx-controller --namespace ingress-nginx --ignore-not-found
    kubectl delete service ingress-nginx-controller --namespace ingress-nginx --ignore-not-found
    kubectl delete service ingress-nginx-controller-admission --namespace ingress-nginx --ignore-not-found
    kubectl delete configmap ingress-nginx-controller --namespace ingress-nginx --ignore-not-found
    kubectl delete serviceaccount ingress-nginx ingress-nginx-admission --namespace ingress-nginx --ignore-not-found
    kubectl delete role ingress-nginx ingress-nginx-admission --namespace ingress-nginx --ignore-not-found
    kubectl delete rolebinding ingress-nginx ingress-nginx-admission --namespace ingress-nginx --ignore-not-found
    kubectl delete secret ingress-nginx-admission --namespace ingress-nginx --ignore-not-found
    kubectl delete lease ingress-nginx-leader --namespace ingress-nginx --ignore-not-found
    kubectl delete certificate ingress-nginx-admission --namespace ingress-nginx --ignore-not-found
    Example output
    daemonset.apps "ingress-nginx-controller" deleted
    service "ingress-nginx-controller" deleted
    service "ingress-nginx-controller-admission" deleted
    configmap "ingress-nginx-controller" deleted
    serviceaccount "ingress-nginx" deleted
    serviceaccount "ingress-nginx-admission" deleted
    role.rbac.authorization.k8s.io "ingress-nginx" deleted
    role.rbac.authorization.k8s.io "ingress-nginx-admission" deleted
    rolebinding.rbac.authorization.k8s.io "ingress-nginx" deleted
    rolebinding.rbac.authorization.k8s.io "ingress-nginx-admission" deleted
    secret "ingress-nginx-admission" deleted
    lease.coordination.k8s.io "ingress-nginx-leader" deleted
    certificate.cert-manager.io "ingress-nginx-admission" deleted
  6. Delete the legacy nginx-tls Secret and nginx Certificate from each application namespace.

    for ns in hubble-system ui-system cp-system; do
    kubectl delete certificate nginx --namespace "$ns" --ignore-not-found
    kubectl delete secret nginx-tls --namespace "$ns" --ignore-not-found
    done
    Example output
    certificate.cert-manager.io "nginx" deleted
    secret "nginx-tls" deleted
    certificate.cert-manager.io "nginx" deleted
    secret "nginx-tls" deleted
    certificate.cert-manager.io "nginx" deleted
    secret "nginx-tls" deleted
  7. Verify that the cluster-scoped Ingress Nginx resources are gone.

    kubectl get validatingwebhookconfiguration ingress-nginx-admission
    kubectl get clusterrole ingress-nginx ingress-nginx-admission
    kubectl get clusterrolebinding ingress-nginx ingress-nginx-admission
    kubectl get ingressclass nginx
    Example output
    Error from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "ingress-nginx-admission" not found
    Error from server (NotFound): clusterroles.rbac.authorization.k8s.io "ingress-nginx" not found
    Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "ingress-nginx" not found
    Error from server (NotFound): ingressclasses.networking.k8s.io "nginx" not found
  8. Verify that the ingress-nginx namespace and default-ssl-certificate Secret are preserved.

    kubectl get namespace ingress-nginx
    kubectl get secret default-ssl-certificate --namespace ingress-nginx
    Example output
    NAME            STATUS   AGE
    ingress-nginx Active 42d

    NAME TYPE DATA AGE
    default-ssl-certificate kubernetes.io/tls 2 42d
  9. Once the leftover Ingress Nginx pods are removed, the scheduler places the Traefik DaemonSet pods, configserver reaches Ready, and the remaining automatic cleanup completes on the next configserver start. Confirm that the Traefik pods are running and that configserver is Ready.

    kubectl get pods --namespace ingress-traefik
    kubectl get pod --namespace hubble-system --selector component=configserver
    Example output
    NAME                               READY   STATUS    RESTARTS   AGE
    traefik-ingress-controller-9dmzq 1/1 Running 0 3m
    traefik-ingress-controller-tpwtf 1/1 Running 0 3m
    traefik-ingress-controller-xz4jf 1/1 Running 0 3m

    NAME READY STATUS RESTARTS AGE
    configserver-685fcc5b6d-t8f8h 1/1 Running 0 2m

Self-Hosted Palette or Palette VerteX Upgrade Hangs​

Upgrading self-hosted Palette or Palette VerteX from version 4.6.x to 4.7.x can cause the upgrade to hang if any member of a MongoDB ReplicaSet is not fully synced and in a healthy state prior to the upgrade.

Debug Steps​

To verify the health status of each MongoDB ReplicaSet member, use the following procedure based on whether you are upgrading Palette or Palette VerteX.

  1. Log in to the Palette or Palette VerteX system console.

  2. From the left main menu, select Enterprise Cluster.

  3. On the Overview tab, download the Kubernetes Config File.

  4. Open a terminal window and set the environment variable KUBECONFIG to point to kubeconfig file you downloaded.

    Example command
    export KUBECONFIG=~/Downloads/spectro-mgmt-cluster.kubeconfig
  1. Issue the following command to query the ReplicaSet for its current primary host, extract the Pod name, and save its value as MONGO_PRIMARY.

    info

    The values for MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD do not need to be exported, as they are already defined within the MongoDB Pods.

    MONGO_PRIMARY=$(
    kubectl exec \
    --namespace hubble-system \
    mongo-1 \
    --container mongo \
    -- \
    mongosh \
    --username "$MONGO_INITDB_ROOT_USERNAME" \
    --password "$MONGO_INITDB_ROOT_PASSWORD" \
    admin \
    --quiet \
    --eval "print(JSON.stringify(rs.hello()))" \
    | jq --raw-output .primary \
    | awk -F. '{print $1}'
    )
  2. Issue the following command to connect to the primary Pod and print each ReplicaSet member’s host, state, and health status.

    kubectl exec \
    --namespace hubble-system \
    "${MONGO_PRIMARY}" \
    --container mongo \
    -- bash -c \
    'mongosh \
    --username "$MONGO_INITDB_ROOT_USERNAME" \
    --password "$MONGO_INITDB_ROOT_PASSWORD" \
    --host "$HOSTNAME" \
    admin \
    --quiet \
    --eval "rs.status().members.forEach(m => printjson({host:m.name,state:m.stateStr,health:m.health}))"'

    All healthy members should have a health status of 1. If the ReplicaSet members are healthy, proceed with upgrading self-hosted Palette or VerteX.

    Example output
    {
    host: 'mongo-1.mongo.hubble-system.svc.cluster.local:27017',
    state: 'PRIMARY',
    health: 1
    }
    {
    host: 'mongo-0.mongo.hubble-system.svc.cluster.local:27017',
    state: 'SECONDARY',
    health: 1
    }
    {
    host: 'mongo-2.mongo.hubble-system.svc.cluster.local:27017',
    state: 'SECONDARY',
    health: 1
    }

Ingress Errors​

If you receive the following error message when attempting to upgrade to Palette versions greater than Palette 3.4.X in a Kubernetes environment, use the debugging steps to address the issue.

Error: UPGRADE FAILED: failed to create resource: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "_" and path "/v1/oidc" is already defined in ingress default/hubble-auth-oidc-ingress-resource

Debug Steps​

  1. Connect to the cluster using the cluster's kubeconfig file. Refer to the Access Cluster with CLI for additional guidance.

  2. Identify all Ingress resources that belong to Hubble - an internal Palette component.

    kubectl get ingress --namespace default
  3. Remove each Ingress resource listed in the output that starts with the name Hubble. Use the following command to delete an Ingress resource. Replace REPLACE_ME with the name of the Ingress resource you are removing.

    kubectl delete ingress --namespace default <REPLACE_ME>
  4. Restart the upgrade process.

Mongo DNS ConfigMap Value is Incorrect​

In VMware vSphere VerteX installations, if you encounter an error during the upgrade process where the MongoDB DNS ConfigMap value is incorrect, use the following steps to resolve the issue.

Debug Steps​

  1. Open up a terminal session in an environment that has network access to the Kubernetes cluster. Refer to the Access Cluster with CLI for additional guidance.

  2. Verify that the pods in the hubble-system namespace are not starting correctly by issuing the following command.

    kubectl get pods --namespace=hubble-system
  3. Verify that the ConfigMap for the configserver in the hubble-system namespace contains the incorrect host value mongo-1.mongohubble-system.svc.cluster. Use the following command to describe the ConfigMap and search for the host value.

    kubectl describe configmap configserver --namespace hubble-system | grep host
     host: mongo-0.mongo.hubble-system.svc.cluster.local,mongo-1.mongohubble-system.svc.cluster.local,mongo-2.mongo.hubble-system.svc.cluster.local
    host: '0.0.0.0'
    host: '0.0.0.0'
    host: '0.0.0.0'
    host: '0.0.0.0'
    host: '0.0.0.0'
    host: '0.0.0.0'
    host: '0.0.0.0'
    host: '0.0.0.0'
    host: '0.0.0.0'
    host: '0.0.0.0'
    host: '0.0.0.0'
    host: '0.0.0.0'
  4. If the host value is incorrect, log in to the System Console. You can find guidance on how to access the System Console in the Access the System Console documentation.

  5. Navigate to the Main Menu and select Enterprise Cluster. From the System Profiles page, select the Spectro pack.

    A view of the Spectro pack in the System Profiles page

  6. In the YAML editor, locate the parameter databaseUrl and update the value mongo-1.mongohubble-system.svc.cluster.local to mongo-1.mongo.hubble-system.svc.cluster.local.

    Below is what the updated databaseUrl value should look like.

    databaseUrl: "mongo-0.mongo.hubble-system.svc.cluster.local,mongo-1.mongo.hubble-system.svc.cluster.local,mongo-2.mongo.hubble-system.svc.cluster.local"
  7. Click Save to apply the changes.

  8. Verify the system pods are starting correctly by issuing the following command.

    kubectl get pods --namespace=hubble-system
      NAME                                   READY   STATUS    RESTARTS   AGE
    auth-64b88d97dd-5z7ph 1/1 Running 0 31m
    auth-64b88d97dd-bchr7 1/1 Running 0 31m
    cloud-b8796c57d-5r7d9 1/1 Running 0 31m
    cloud-b8796c57d-xpbx7 1/1 Running 0 31m
    configserver-778bd7c4c9-mrtc6 1/1 Running 0 31m
    event-5869c6bd75-2n7jl 1/1 Running 0 31m
    event-5869c6bd75-xnvmj 1/1 Running 0 31m
    foreq-679c7b7f6b-2ts2v 1/1 Running 0 31m
    hashboard-9f865b6c8-c52bb 1/1 Running 0 31m
    hashboard-9f865b6c8-rw6p4 1/1 Running 0 31m
    hutil-54995bfd6b-sh4dt 1/1 Running 0 31m
    hutil-54995bfd6b-tlqbj 1/1 Running 0 31m
    memstore-7584fdd94f-479pj 1/1 Running 0 31m
    mgmt-68c8dbfd58-8gxsx 1/1 Running 0 31m
    mongo-0 2/2 Running 0 29m
    mongo-1 2/2 Running 0 30m
    mongo-2 2/2 Running 0 30m
    msgbroker-7d7655559b-zxxfq 1/1 Running 0 31m
    oci-proxy-6fdf95885f-qw58g 1/1 Running 0 31m
    reloader-reloader-845cfd7fdf-2rq5t 1/1 Running 0 31m
    spectrocluster-5c4cb4ff58-658w9 1/1 Running 0 31m
    spectrocluster-5c4cb4ff58-fn8g5 1/1 Running 0 31m
    spectrocluster-5c4cb4ff58-zvwfp 1/1 Running 0 31m
    spectrocluster-jobs-5b54bf6bcf-mtgh8 1/1 Running 0 31m
    system-6678d47874-464n6 1/1 Running 0 31m
    system-6678d47874-rgn55 1/1 Running 0 31m
    timeseries-6564699c7d-b6fnr 1/1 Running 0 31m
    timeseries-6564699c7d-hvv94 1/1 Running 0 31m
    timeseries-6564699c7d-jzmnl 1/1 Running 0 31m
    user-866c7f779d-drf9w 1/1 Running 0 31m
    user-866c7f779d-rm4hw 1/1 Running 0 31m