Palette Upgrade
We recommend you review the Release Notes and the Upgrade Notes before attempting to upgrade Palette. Use this information to address common issues that may occur during an upgrade.
Scenario - Palette Webhook Downgraded After Rolling Upgrade
After upgrading a Palette or Palette VerteX deployment to 4.9.38, connected workload clusters may fail to reconcile
because the palette-webhook deployment was downgraded during the rolling upgrade of palette-controller-manager in
the cluster's namespace. The outgoing controller pod holds the leader lease briefly while the new controller pod starts.
During that window, the outgoing pod can re-apply the older palette-webhook manifest baked into its own image,
overwriting the newer manifest the incoming pod already applied. The older webhook does not serve the v1alpha1 to v1
conversion that Custom Resource Definitions (CRDs) in the cluster.spectrocloud.com API group now require, so every
SpectroCluster reconcile fails and does not self-heal.
Symptoms of an affected cluster include:
-
The cluster appears grey on the Clusters page in the Palette UI.
-
Cluster profile updates do not apply to the cluster.
-
The
palette-controller-managerlogs record repeated reconcile failures similar to the following.Example errorfailed to set owner ref to cloud config: Internal error occurred: conversion webhook for
cluster.spectrocloud.com/v1alpha1, Kind=AwsCloudConfig failed: the server could not find
the requested resource
The trigger is the Palette agent upgrade on an existing cluster. New cluster deployments are not affected.
Debug Steps
Recovery may require restarting both the palette-controller-manager deployment in the affected cluster's namespace and
the palette-webhook deployment in the palette-system namespace. Complete the steps in order and check the webhook
image after each restart to determine whether the second restart is needed.
-
Establish a shell session with
kubectlaccess to the affected workload cluster. Refer to Access Cluster with CLI for guidance. -
Identify the namespace of the affected cluster. The namespace follows the format
cluster-<cluster-uid>, where<cluster-uid>is the identifier displayed on the cluster's Overview tab in the Palette UI. -
Compare the image on the
palette-webhookdeployment against the expected Palette version recorded in thepalette-version-info-for-webhookConfigMap.kubectl --namespace palette-system get deployment palette-webhook \
--output jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl --namespace palette-system get configmap palette-version-info-for-webhook \
--output jsonpath='{.data.VERSION}{"\n"}'Example output on an affected clusterus-docker.pkg.dev/palette-images/palette/spectro-drive:4.9.11
4.9.19If the two values do not agree, the webhook has been downgraded and the cluster is affected. In the example above, the webhook is still running the
4.9.11image while the controller expects4.9.19. -
Restart the
palette-controller-managerdeployment in the affected cluster's namespace. When the outgoing controller pod is gone, the surviving pod reapplies the correctpalette-webhookmanifest at startup, before it takes the leader lease.kubectl --namespace cluster-<cluster-uid> rollout restart deployment palette-controller-manager
kubectl --namespace cluster-<cluster-uid> rollout status deployment palette-controller-manager \
--timeout=600s -
Re-check the
palette-webhookdeployment image. When it matches the version recorded inpalette-version-info-for-webhook, the cluster begins reconciling successfully.kubectl --namespace palette-system get deployment palette-webhook \
--output jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'Example output on a recovered clusterus-docker.pkg.dev/palette-images/palette/spectro-drive:4.9.19 -
If the webhook image in step 5 still shows the older version, restart the
palette-webhookdeployment in thepalette-systemnamespace. Some affected clusters need this second restart before the controller can move the webhook forward.kubectl --namespace palette-system rollout restart deployment palette-webhook
kubectl --namespace palette-system rollout status deployment palette-webhook --timeout=600sAfter the rollout completes, run the check in step 5 again and confirm that the image now matches the expected version.
-
Confirm that reconcile has recovered. Follow the controller logs and verify that the
failed to set owner ref to cloud configerror no longer appears on new reconcile attempts.kubectl --namespace cluster-<cluster-uid> logs deployment/palette-controller-manager --follow
Scenario - Custom Certificate Replaced After Upgrade
If upgrading IP-based self-hosted Palette or Palette VerteX environments installed using the Palette Management Appliance, VerteX Management Appliance, or Helm charts to any version between 4.8.47 - 4.9.14, Palette may replace custom Transport Security Layer (TLS) certificates with newly generated certificates. This results in Edge hosts and other clients that trusted the original, custom certificates to stop communicating with Palette.
Before upgrading, we recommend backing up your existing certificates so that you can restore them through the system console.
Debug Steps
-
Log in to the system console.
-
From the left main menu, select Enterprise Cluster.
-
On the Overview tab, download the Kubernetes Config File.
-
Open a terminal session in an environment that has network access to the cluster. Set the
KUBECONFIGenvironment variable to the file path of the downloaded kubeconfig file.export KUBECONFIG=<path-to-kubeconfig> -
Verify that the
default-ssl-certificateSecret is present. For Palette versions 4.8.47 and later, the Secret is in theingress-traefiknamespace; for versions earlier than 4.8.47, the Secret is in theingress-nginxnamespace.- Traefik
- Ingress Nginx
kubectl get secret default-ssl-certificate --namespace ingress-traefikkubectl get secret default-ssl-certificate --namespace ingress-nginxExample outputNAME TYPE DATA AGE
default-ssl-certificate Opaque 3 60d -
Save a copy of your custom server certificate, private key, and Certificate Authority (CA) certificate.
- Traefik
- Ingress Nginx
kubectl get secret default-ssl-certificate --namespace ingress-traefik --output yaml > default-ssl-certificate-backup.yamlkubectl get secret default-ssl-certificate --namespace ingress-nginx --output yaml > default-ssl-certificate-backup.yaml -
Upgrade Palette using your standard procedure.
-
After the upgrade completes, compare the current
ca.crt,tls.crt, andtls.keyvalues with your backup.- Traefik
- Ingress Nginx
kubectl get secret default-ssl-certificate --namespace ingress-traefik --output yamlkubectl get secret default-ssl-certificate --namespace ingress-nginx --output yamlExample outputapiVersion: v1
data:
ca.crt: LS0tLS1CRUd... # output omitted for brevity
tls.crt: LS0tLS1CRUd... # output omitted for brevity
tls.key: LS0tLS1CRUd... # output omitted for brevity
kind: Secret
metadata:
creationTimestamp: "2026-04-06T22:28:08Z"
name: default-ssl-certificate
namespace: ingress-traefik
resourceVersion: "339401"
uid: de363cf1-431a-497d-8ccb-0cff1b1ab609
type: Opaque -
If the certificate values do not match, re-upload the server certificate, private key, and CA certificate through the system console. To do so, return to the system console in your web browser, and from the left main menu, select Administration.
-
Select the System Address tab.
-
Copy and paste the values from your backed up file into the appropriate fields.
YAML Field UI Field tls.crtCertificate tls.keyKey ca.crtCertificate Authority (Optional) -
Select Update to apply the certificate.
-
Verify that connected Edge hosts and workload clusters resume reporting their status to Palette.
configserver Stuck on init-rootdomain-traefik After Upgrade to 4.9.14+
After upgrading self-hosted Palette or
Palette VerteX to version 4.9.14 or later, the configserver pod may remain in
Init:CrashLoopBackOff for the init-rootdomain-traefik init container. This happens when leftover Nginx ingress
controller pods on the same nodes hold host ports 80, 443, and 9090, which prevents the new Traefik DaemonSet pods from
scheduling. The automatic Nginx cleanup script that runs at configserver startup cannot run because configserver
itself never reaches Ready, so the leftover Nginx resources must be removed manually.
This scenario applies only to Enterprise installs of self-hosted Palette and Palette VerteX.
Do not delete the default-ssl-certificate Secret in the ingress-nginx namespace, and do not delete the
ingress-nginx namespace itself. The Enterprise cert-bridge continues to copy your uploaded Transport Layer Security
(TLS) certificate from the default-ssl-certificate Secret into Traefik on every configserver restart. Deleting
either resource breaks ingress TLS.
Debug Steps
-
Connect to the cluster using the cluster's kubeconfig file. Refer to the Access Cluster with CLI guide for additional guidance.
-
Verify that the
configserverpod is in theInit:CrashLoopBackOffstate and inspect the init container logs.kubectl get pod --namespace hubble-system --selector component=configserver
kubectl logs <configserver-pod> --namespace hubble-system --container init-rootdomain-traefik --previous | tailExample outputNAME READY STATUS RESTARTS AGE
configserver-685fcc5b6d-t8f8h 0/1 Init:CrashLoopBackOff 5 8m
level=fatal msg="not able to get the host ip. got NotFound: Resource 'websecure' not found of type 'pod'" -
Confirm that the Traefik DaemonSet pods are stuck in
Pendingbecause the host ports are unavailable.kubectl get pods --namespace ingress-traefikExample outputNAME READY STATUS RESTARTS AGE
traefik-ingress-controller-dkjs5 0/1 Pending 0 8m
traefik-ingress-controller-nltlh 0/1 Pending 0 8mIf the scheduler events for those pods show
didn't have free ports for the requested pod ports, leftover Nginx pods are holding host ports 80, 443, and 9090 on the same nodes. -
Delete the cluster-scoped Nginx resources.
kubectl delete validatingwebhookconfiguration ingress-nginx-admission --ignore-not-found
kubectl delete clusterrole ingress-nginx ingress-nginx-admission --ignore-not-found
kubectl delete clusterrolebinding ingress-nginx ingress-nginx-admission --ignore-not-found
kubectl delete ingressclass nginx --ignore-not-foundExample outputvalidatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted
clusterrole.rbac.authorization.k8s.io "ingress-nginx" deleted
clusterrole.rbac.authorization.k8s.io "ingress-nginx-admission" deleted
clusterrolebinding.rbac.authorization.k8s.io "ingress-nginx" deleted
clusterrolebinding.rbac.authorization.k8s.io "ingress-nginx-admission" deleted
ingressclass.networking.k8s.io "nginx" deleted -
Delete the Nginx resources inside the
ingress-nginxnamespace. Do not delete the namespace itself or thedefault-ssl-certificateSecret.kubectl delete daemonset ingress-nginx-controller --namespace ingress-nginx --ignore-not-found
kubectl delete service ingress-nginx-controller --namespace ingress-nginx --ignore-not-found
kubectl delete service ingress-nginx-controller-admission --namespace ingress-nginx --ignore-not-found
kubectl delete configmap ingress-nginx-controller --namespace ingress-nginx --ignore-not-found
kubectl delete serviceaccount ingress-nginx ingress-nginx-admission --namespace ingress-nginx --ignore-not-found
kubectl delete role ingress-nginx ingress-nginx-admission --namespace ingress-nginx --ignore-not-found
kubectl delete rolebinding ingress-nginx ingress-nginx-admission --namespace ingress-nginx --ignore-not-found
kubectl delete secret ingress-nginx-admission --namespace ingress-nginx --ignore-not-found
kubectl delete lease ingress-nginx-leader --namespace ingress-nginx --ignore-not-found
kubectl delete certificate ingress-nginx-admission --namespace ingress-nginx --ignore-not-foundExample outputdaemonset.apps "ingress-nginx-controller" deleted
service "ingress-nginx-controller" deleted
service "ingress-nginx-controller-admission" deleted
configmap "ingress-nginx-controller" deleted
serviceaccount "ingress-nginx" deleted
serviceaccount "ingress-nginx-admission" deleted
role.rbac.authorization.k8s.io "ingress-nginx" deleted
role.rbac.authorization.k8s.io "ingress-nginx-admission" deleted
rolebinding.rbac.authorization.k8s.io "ingress-nginx" deleted
rolebinding.rbac.authorization.k8s.io "ingress-nginx-admission" deleted
secret "ingress-nginx-admission" deleted
lease.coordination.k8s.io "ingress-nginx-leader" deleted
certificate.cert-manager.io "ingress-nginx-admission" deleted -
Delete the legacy
nginx-tlsSecret andnginxCertificate from each application namespace.for ns in hubble-system ui-system cp-system; do
kubectl delete certificate nginx --namespace "$ns" --ignore-not-found
kubectl delete secret nginx-tls --namespace "$ns" --ignore-not-found
doneExample outputcertificate.cert-manager.io "nginx" deleted
secret "nginx-tls" deleted
certificate.cert-manager.io "nginx" deleted
secret "nginx-tls" deleted
certificate.cert-manager.io "nginx" deleted
secret "nginx-tls" deleted -
Verify that the cluster-scoped Ingress Nginx resources are gone.
kubectl get validatingwebhookconfiguration ingress-nginx-admission
kubectl get clusterrole ingress-nginx ingress-nginx-admission
kubectl get clusterrolebinding ingress-nginx ingress-nginx-admission
kubectl get ingressclass nginxExample outputError from server (NotFound): validatingwebhookconfigurations.admissionregistration.k8s.io "ingress-nginx-admission" not found
Error from server (NotFound): clusterroles.rbac.authorization.k8s.io "ingress-nginx" not found
Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io "ingress-nginx" not found
Error from server (NotFound): ingressclasses.networking.k8s.io "nginx" not found -
Verify that the
ingress-nginxnamespace anddefault-ssl-certificateSecret are preserved.kubectl get namespace ingress-nginx
kubectl get secret default-ssl-certificate --namespace ingress-nginxExample outputNAME STATUS AGE
ingress-nginx Active 42d
NAME TYPE DATA AGE
default-ssl-certificate kubernetes.io/tls 2 42d -
Once the leftover Ingress Nginx pods are removed, the scheduler places the Traefik DaemonSet pods,
configserverreaches Ready, and the remaining automatic cleanup completes on the nextconfigserverstart. Confirm that the Traefik pods are running and thatconfigserveris Ready.kubectl get pods --namespace ingress-traefik
kubectl get pod --namespace hubble-system --selector component=configserverExample outputNAME READY STATUS RESTARTS AGE
traefik-ingress-controller-9dmzq 1/1 Running 0 3m
traefik-ingress-controller-tpwtf 1/1 Running 0 3m
traefik-ingress-controller-xz4jf 1/1 Running 0 3m
NAME READY STATUS RESTARTS AGE
configserver-685fcc5b6d-t8f8h 1/1 Running 0 2m
Self-Hosted Palette or Palette VerteX Upgrade Hangs
Upgrading self-hosted Palette or Palette VerteX from version 4.6.x to 4.7.x can cause the upgrade to hang if any member of a MongoDB ReplicaSet is not fully synced and in a healthy state prior to the upgrade.
Debug Steps
To verify the health status of each MongoDB ReplicaSet member, use the following procedure based on whether you are upgrading Palette or Palette VerteX.
-
Log in to the Palette or Palette VerteX system console.
-
From the left main menu, select Enterprise Cluster.
-
On the Overview tab, download the Kubernetes Config File.
-
Open a terminal window and set the environment variable
KUBECONFIGto point to kubeconfig file you downloaded.Example commandexport KUBECONFIG=~/Downloads/spectro-mgmt-cluster.kubeconfig
- Palette
- Palette VerteX
-
Issue the following command to query the ReplicaSet for its current primary host, extract the Pod name, and save its value as
MONGO_PRIMARY.infoThe values for
MONGO_INITDB_ROOT_USERNAMEandMONGO_INITDB_ROOT_PASSWORDdo not need to be exported, as they are already defined within the MongoDB Pods.MONGO_PRIMARY=$(
kubectl exec \
--namespace hubble-system \
mongo-1 \
--container mongo \
-- \
mongosh \
--username "$MONGO_INITDB_ROOT_USERNAME" \
--password "$MONGO_INITDB_ROOT_PASSWORD" \
admin \
--quiet \
--eval "print(JSON.stringify(rs.hello()))" \
| jq --raw-output .primary \
| awk -F. '{print $1}'
) -
Issue the following command to connect to the primary Pod and print each ReplicaSet member’s host, state, and health status.
kubectl exec \
--namespace hubble-system \
"${MONGO_PRIMARY}" \
--container mongo \
-- bash -c \
'mongosh \
--username "$MONGO_INITDB_ROOT_USERNAME" \
--password "$MONGO_INITDB_ROOT_PASSWORD" \
--host "$HOSTNAME" \
admin \
--quiet \
--eval "rs.status().members.forEach(m => printjson({host:m.name,state:m.stateStr,health:m.health}))"'All healthy members should have a
healthstatus of1. If the ReplicaSet members are healthy, proceed with upgrading self-hosted Palette or VerteX.Example output{
host: 'mongo-1.mongo.hubble-system.svc.cluster.local:27017',
state: 'PRIMARY',
health: 1
}
{
host: 'mongo-0.mongo.hubble-system.svc.cluster.local:27017',
state: 'SECONDARY',
health: 1
}
{
host: 'mongo-2.mongo.hubble-system.svc.cluster.local:27017',
state: 'SECONDARY',
health: 1
}
-
Issue the following command to query the ReplicaSet for its current primary host, extract the Pod name, and save its value as
MONGO_PRIMARY.infoThe values for
MONGODB_INITDB_ROOT_USERNAMEandMONGODB_INITDB_ROOT_PASSWORDdo not need to be exported, as they are already defined within the MongoDB Pods.MONGO_PRIMARY=$(
kubectl exec \
--namespace hubble-system \
mongo-1 \
--container mongo \
-- bash -c \
'mongosh \
--username "$MONGODB_INITDB_ROOT_USERNAME" \
--password "$MONGODB_INITDB_ROOT_PASSWORD" \
--host "$HOSTNAME" \
--tls \
--tlsCAFile /var/mongodb/tls/ca.crt \
--tlsCertificateKeyFile /var/mongodb/tls/tls-combined.pem \
--tlsAllowInvalidHostnames \
admin \
--quiet \
--eval "print(JSON.stringify(rs.isMaster()))"' \
| jq --raw-output .primary \
| awk -F. '{print $1}'
) -
Issue the following command to connect to the primary Pod and print each ReplicaSet member’s host, state, and health status.
kubectl exec \
--namespace hubble-system \
"${MONGO_PRIMARY}" \
--container mongo \
-- bash -c \
'mongosh \
--username "$MONGODB_INITDB_ROOT_USERNAME" \
--password "$MONGODB_INITDB_ROOT_PASSWORD" \
--host "$HOSTNAME" \
--tls \
--tlsCAFile /var/mongodb/tls/ca.crt \
--tlsCertificateKeyFile /var/mongodb/tls/tls-combined.pem \
--tlsAllowInvalidHostnames \
admin \
--quiet \
--eval "rs.status().members.forEach(m => printjson({host:m.name,state:m.stateStr,health:m.health}))"'All healthy members should have a
healthstatus of1. If the ReplicaSet members are healthy, proceed with upgrading self-hosted Palette or VerteX.Example output{
host: 'mongo-1.mongo.hubble-system.svc.cluster.local:27017',
state: 'PRIMARY',
health: 1
}
{
host: 'mongo-0.mongo.hubble-system.svc.cluster.local:27017',
state: 'SECONDARY',
health: 1
}
{
host: 'mongo-2.mongo.hubble-system.svc.cluster.local:27017',
state: 'SECONDARY',
health: 1
}
Ingress Errors
If you receive the following error message when attempting to upgrade to Palette versions greater than Palette 3.4.X in a Kubernetes environment, use the debugging steps to address the issue.
Error: UPGRADE FAILED: failed to create resource: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "_" and path "/v1/oidc" is already defined in ingress default/hubble-auth-oidc-ingress-resource
Debug Steps
-
Connect to the cluster using the cluster's kubeconfig file. Refer to the Access Cluster with CLI for additional guidance.
-
Identify all Ingress resources that belong to Hubble - an internal Palette component.
kubectl get ingress --namespace default -
Remove each Ingress resource listed in the output that starts with the name Hubble. Use the following command to delete an Ingress resource. Replace
REPLACE_MEwith the name of the Ingress resource you are removing.kubectl delete ingress --namespace default <REPLACE_ME> -
Restart the upgrade process.
Mongo DNS ConfigMap Value is Incorrect
In VMware vSphere VerteX installations, if you encounter an error during the upgrade process where the MongoDB DNS ConfigMap value is incorrect, use the following steps to resolve the issue.
Debug Steps
-
Open up a terminal session in an environment that has network access to the Kubernetes cluster. Refer to the Access Cluster with CLI for additional guidance.
-
Verify that the pods in the
hubble-systemnamespace are not starting correctly by issuing the following command.kubectl get pods --namespace=hubble-system -
Verify that the ConfigMap for the
configserverin thehubble-systemnamespace contains the incorrect host valuemongo-1.mongohubble-system.svc.cluster. Use the following command to describe the ConfigMap and search for the host value.kubectl describe configmap configserver --namespace hubble-system | grep hosthost: mongo-0.mongo.hubble-system.svc.cluster.local,mongo-1.mongohubble-system.svc.cluster.local,mongo-2.mongo.hubble-system.svc.cluster.local
host: '0.0.0.0'
host: '0.0.0.0'
host: '0.0.0.0'
host: '0.0.0.0'
host: '0.0.0.0'
host: '0.0.0.0'
host: '0.0.0.0'
host: '0.0.0.0'
host: '0.0.0.0'
host: '0.0.0.0'
host: '0.0.0.0'
host: '0.0.0.0' -
If the host value is incorrect, log in to the System Console. You can find guidance on how to access the System Console in the Access the System Console documentation.
-
Navigate to the Main Menu and select Enterprise Cluster. From the System Profiles page, select the Spectro pack.

-
In the YAML editor, locate the parameter
databaseUrland update the valuemongo-1.mongohubble-system.svc.cluster.localtomongo-1.mongo.hubble-system.svc.cluster.local.Below is what the updated
databaseUrlvalue should look like.databaseUrl: "mongo-0.mongo.hubble-system.svc.cluster.local,mongo-1.mongo.hubble-system.svc.cluster.local,mongo-2.mongo.hubble-system.svc.cluster.local" -
Click Save to apply the changes.
-
Verify the system pods are starting correctly by issuing the following command.
kubectl get pods --namespace=hubble-systemNAME READY STATUS RESTARTS AGE
auth-64b88d97dd-5z7ph 1/1 Running 0 31m
auth-64b88d97dd-bchr7 1/1 Running 0 31m
cloud-b8796c57d-5r7d9 1/1 Running 0 31m
cloud-b8796c57d-xpbx7 1/1 Running 0 31m
configserver-778bd7c4c9-mrtc6 1/1 Running 0 31m
event-5869c6bd75-2n7jl 1/1 Running 0 31m
event-5869c6bd75-xnvmj 1/1 Running 0 31m
foreq-679c7b7f6b-2ts2v 1/1 Running 0 31m
hashboard-9f865b6c8-c52bb 1/1 Running 0 31m
hashboard-9f865b6c8-rw6p4 1/1 Running 0 31m
hutil-54995bfd6b-sh4dt 1/1 Running 0 31m
hutil-54995bfd6b-tlqbj 1/1 Running 0 31m
memstore-7584fdd94f-479pj 1/1 Running 0 31m
mgmt-68c8dbfd58-8gxsx 1/1 Running 0 31m
mongo-0 2/2 Running 0 29m
mongo-1 2/2 Running 0 30m
mongo-2 2/2 Running 0 30m
msgbroker-7d7655559b-zxxfq 1/1 Running 0 31m
oci-proxy-6fdf95885f-qw58g 1/1 Running 0 31m
reloader-reloader-845cfd7fdf-2rq5t 1/1 Running 0 31m
spectrocluster-5c4cb4ff58-658w9 1/1 Running 0 31m
spectrocluster-5c4cb4ff58-fn8g5 1/1 Running 0 31m
spectrocluster-5c4cb4ff58-zvwfp 1/1 Running 0 31m
spectrocluster-jobs-5b54bf6bcf-mtgh8 1/1 Running 0 31m
system-6678d47874-464n6 1/1 Running 0 31m
system-6678d47874-rgn55 1/1 Running 0 31m
timeseries-6564699c7d-b6fnr 1/1 Running 0 31m
timeseries-6564699c7d-hvv94 1/1 Running 0 31m
timeseries-6564699c7d-jzmnl 1/1 Running 0 31m
user-866c7f779d-drf9w 1/1 Running 0 31m
user-866c7f779d-rm4hw 1/1 Running 0 31m