Back Up and Restore the Palette Management Appliance

The following procedure describes how to back up a Palette Management Appliance installation and restore it to another Palette Management Appliance installation.

Use the following procedure to back up a Palette Management Appliance installation and restore it to another Palette Management Appliance installation. In this guide, the installation you back up from is the source cluster, and the installation you restore to is the destination cluster.

You can also use this procedure to back up a cluster and restore to the same cluster.

info

The utility archive referenced throughout this guide contains the scripts you need to complete the backup and restore. Download the scripts linked below to your Linux jump host before you begin.

Palette Backup and Restore Scripts

Supported Scenarios

The following combinations are supported.

Scenario	Supported
Source and destination are the same cluster.	Yes
Destination is at the same Palette version as source.	Yes
Destination is at a higher Palette version than source.	Yes
Destination is at a lower Palette version than source.	No
Source and destination are different editions.	No

For example, a source cluster at 4.7.x can be restored to a destination cluster at 4.7.x or 4.8.x, but not to a destination cluster at 4.6.x. Both clusters must be the same edition, either Palette or VerteX.

Prerequisites

A Linux jump host running Ubuntu 24.04 with network access to both the source and destination clusters. This host is used to run the backup and restore scripts, including the Zot registry sync, which requires Ubuntu 24.04.
The Palette Backup and Restore Scripts utility archive downloaded to your Linux jump host.

The source cluster and destination cluster meet the requirements listed in Supported Scenarios.
The destination cluster is already installed and healthy. Refer to the Management Appliance install guide for installation instructions.
The kubeconfig file for the source cluster and the kubeconfig file for the destination cluster.
Administrative access to the Palette system console for the source cluster.
The system address for the source cluster is configured with a Fully Qualified Domain Name (FQDN) instead of an IP address.

warning
If your source cluster uses an IP address instead of an FQDN for the system address, a config map on each workload cluster must be updated manually to point to the destination cluster's IP address. Contact Spectro Cloud Support for guidance.
Every Edge host used to launch a Connected Edge Native workload cluster from the source cluster references the Palette FQDN, and not an IP address, as the PaletteEndpoint value in its user data ISO. This is required so that the DNS cutover in Update DNS and Re-Register Workload Clusters reaches every edge host.
On the jump host, install the following packages. Skopeo mirrors the internal Zot registry, and jq parses the registry catalog response.
```
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install --yes skopeo jq apt-transport-https ca-certificates curl gnupg
```

Add the Kubernetes apt repository and install kubectl.

curl --fail --silent --show-error --location https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | \
  sudo gpg --dearmor --output /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install --yes kubectl

Add the MongoDB apt repository and install MongoDB Database Tools and the MongoDB Shell. The backup and restore scripts call mongodump and mongorestore from the Database Tools package. The Post-Restore Workarounds use mongosh from the MongoDB Shell package.

curl --fail --silent --show-error --location https://www.mongodb.org/static/pgp/server-8.0.asc | \
  sudo gpg --dearmor --output /etc/apt/keyrings/mongodb-server-8.0.gpg

echo "deb [signed-by=/etc/apt/keyrings/mongodb-server-8.0.gpg] https://repo.mongodb.org/apt/ubuntu noble/mongodb-org/8.0 multiverse" | \
  sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list

sudo apt-get update
sudo apt-get install --yes mongodb-database-tools mongodb-mongosh

Back Up the Source Cluster

Use the following steps to pause agent upgrades on workload clusters and take a database backup from the source cluster.

Pause Agent Upgrade on the Source Cluster

Pause the agent upgrade on all workload clusters before you take a backup. This helps ensure a controlled agent upgrade of workload clusters during the restore step if the destination cluster is at a newer version than the source cluster.

Log in to the Palette tenant console for the source cluster.
From the left main menu, select Tenant Settings.
Select Platform Settings.
Enable Pause Agent Upgrade.

Take a Database Backup

On the jump host, create a working directory and change into it.

mkdir --parents palette-backup-restore && cd palette-backup-restore

Place the utility archive downloaded in the Prerequisites into this directory, then extract it. The scripts extract into the current directory.
```
tar --extract --verbose --file=Palette-Backup-Restore-*.tar.gz
```
Copy the source cluster kubeconfig file into this directory and name it source.kubeconfig.
Run the backup script. The script creates the backup directory automatically. If you omit --backup-dir, the script generates a timestamped directory named mongo_hubble_backup_<YYYYMMDD-HHMMSS>.
```
./edge-backup.sh --kubeconfig source.kubeconfig --backup-dir source-backup
```
Confirm that the backup directory contains the expected artifacts.
```
tree --level 1 source-backup
```
The output shows backup_summary.txt, the database dump directories, the cleaned secrets, and the extracted MongoDB TLS material, similar to the following.
```
source-backup
├── backup_summary.txt
├── cleaned-secrets
├── hubble_archivedb
├── hubble_timeseriesdb
├── hubbledb
└── mongo-tls
```

Restore to the Destination Cluster

Use the following steps to synchronize the internal Zot registry with the source cluster and then restore the database backup to the destination cluster.

Synchronize the Zot Registry

Synchronize the internal Zot registry on the destination cluster with the internal Zot registry on the source cluster so that the destination has the packs and images required by the workload clusters.

info

If both clusters share a common external registry, you can skip this section.

On the jump host, open the zot-sync.sh script for editing.
```
vi ./zot-sync.sh
```

Update the four values at the top of the script to match your environment. The SRC and DST values can be the Virtual IP Address (VIP) or the IP address of any node in the corresponding cluster. The credentials must match the ones you set during installation.
```
SRC=192.0.2.10:30003
DST=198.51.100.10:30003
SRC_CREDS="<username>:<password>"
DST_CREDS="<username>:<password>"
```
Save the file and close the editor.
Run the sync script. By default, the script mirrors every repository. To limit the sync to a subset, pass a repository path prefix as the first argument. For example, if you launched the destination cluster edge hosts from the same ISO as the source, you only need to sync the packs uploaded after edge host creation, which live under spectro-content/spectro-packs/archive/.
```
./zot-sync.sh
```
```
./zot-sync.sh spectro-content/spectro-packs/archive/
```
Log in to the destination cluster Zot registry and confirm that the packs are synchronized and that the image checksums match the source registry.

Restore the Database Backup

Copy the destination cluster kubeconfig file into the working directory and name it dst.kubeconfig.

On the jump host, set the KUBECONFIG environment variable to point to the destination cluster.

export KUBECONFIG=dst.kubeconfig

Retrieve the cluster namespace for the destination cluster.
```
kubectl --kubeconfig dst.kubeconfig get ns | grep cluster-
```
The output contains a namespace in the format cluster-<uid>. Record the <uid> value for the next step.
Run the restore script.
- For a destination cluster installed with the Management Appliance or the Palette CLI, replace <uid> with the value from the previous step and run the following command.
```
./edge-restore.sh --uid <uid> --kubeconfig dst.kubeconfig --backup-dir source-backup
```
- For a destination cluster installed with the Helm chart, pass --is-helm and omit --uid. The script derives the namespace from the Helm release.
```
./edge-restore.sh --is-helm --kubeconfig dst.kubeconfig --backup-dir source-backup
```
Wait for the restore to complete.

Update DNS and Re-Register Workload Clusters

After the restore, workload clusters may temporarily show as unhealthy because they are still pointed at the source cluster. Update DNS so that workload clusters reach the destination cluster.

On your DNS server, update the record for the Palette FQDN to point to the destination cluster VIP.
In the Palette system console for the destination cluster, go to Administration > System Address. Confirm the FQDN and TLS certificate settings match the destination cluster.

tip
If you need to issue a new self-signed TLS certificate for the destination cluster VIP, the utility archive includes a helper script. Run ./certs-gen.sh <VIP> from the working directory to generate a CA, server certificate, and matching key. The script writes base64-encoded values to out/cert.b64, out/key.b64, and out/ca.b64, ready to paste into the System Address fields.
For each Connected Edge Native workload cluster, SSH into every edge host and reload the Stylus services so that they pick up the new destination cluster. The daemon-reload re-reads the unit files, and the two restarts drop any in-memory connection state that still points at the source cluster.
```
systemctl daemon-reload
systemctl restart stylus-operator
systemctl restart stylus-agent
```
Wait for workload clusters to reconnect and report a Healthy status.

Resume Agent Upgrade in Stages

Enable agent upgrades in stages so that you can validate the restore against a small number of workload clusters before applying it broadly.

Log in to the Palette tenant console for the destination cluster.
From the left main menu, select Tenant Settings.
Select Platform Settings.
Disable Pause Agent Upgrade so that per-cluster Agent Upgrade toggles take effect.
Select one or two workload clusters to use as test clusters.
In the cluster settings for each test cluster, enable Agent Upgrade.
Wait for the agent upgrade to complete. On the cluster Overview tab, confirm the following.
- The Cluster Management Agent version shows the new version.
- For Edge clusters, on the Edge Hosts tab, the Stylus Agent version on each edge host shows the new version. Controller mode clusters do not have this tab.
Deploy a new test cluster from the destination cluster to further validate that the environment is stable.
Once the test clusters are healthy and running the new agent, enable Agent Upgrade for the remaining workload clusters.

Post-Restore Workarounds

Depending on how you originally installed the source cluster, and which workload cluster types are re-pointed to the destination cluster, you may need to run further manual steps after the restore. Review the following scenarios and apply the workarounds that fit your environment.

Several of the workarounds finish by clearing the cache in the Hubble pods on the destination cluster. Use the procedure in Clear the Hubble Pod Cache whenever it is referenced.

Clear the Hubble Pod Cache

The utility archive includes helper scripts that shut down and start up the Hubble system deployments for a cluster namespace. Running them in sequence forces the Hubble pods to reload their in-memory state.

Shut down the Hubble system deployments. Replace <uid> with the cluster UID retrieved in Restore the Database Backup.
```
./k8s-shutdown-edge.sh --uid <uid> --kubeconfig dst.kubeconfig
```
Wait one to two minutes for the pods to fully terminate.

Start the Hubble system deployments back up.

./k8s-startup-edge.sh --uid <uid> --kubeconfig dst.kubeconfig

Confirm the Hubble pods return to a Running status before continuing.

kubectl --kubeconfig dst.kubeconfig get pods --namespace hubble-system

If workload clusters continue to report Resource Not Found errors against a Cluster UID after the restore, the Hubble pods on the destination cluster may still be caching stale data. Repeat this procedure to clear the cache.

CLI-Installed Enterprise Cluster Source

If the source cluster was installed using the Palette CLI in Enterprise Cluster mode, the installerMode field in the systemconfig document in the hubbledb.config collection must be updated to self-hosted on the destination cluster. This document is immutable through the Palette API, so the update must be applied with a direct MongoDB write.

Contact Spectro Cloud Support for assistance before applying this workaround.

Use the following signals to confirm this workaround applies to your destination cluster.

The destination cluster system console displays a System Cluster section, which is only present on CLI-installed Enterprise Cluster environments.
The response from <VIP>/v1/auth/org contains "appEnv": "enterprise-cli".

Apply the workaround with the utility script, or with a direct MongoDB write. The utility script is the recommended path because it handles the port-forward, credential lookup, and document update for you.

Apply with the utility script

On the jump host, run the set-installer-mode.sh script from the working directory. The script sets installerMode to self-hosted in the systemconfig document for you.
```
./set-installer-mode.sh --kubeconfig dst.kubeconfig
```
Clear the Hubble pod cache using the procedure in Clear the Hubble Pod Cache.

Apply directly in MongoDB

If you prefer to apply the change by hand, use the following steps.

On the jump host, port-forward the MongoDB service on the destination cluster.

kubectl --kubeconfig dst.kubeconfig port-forward --namespace hubble-system svc/mongo 27017:27017

In a new terminal, retrieve the MongoDB root password.

kubectl --kubeconfig dst.kubeconfig get secret spectromongosecret --namespace hubble-system \
  --output jsonpath='{.data.mongoRootPassword}' | base64 --decode

Connect to MongoDB with mongosh. Replace <password> with the value from the previous step.

mongosh "mongodb://root:<password>@127.0.0.1:27017/hubbledb?authSource=admin&directConnection=true"

Locate the systemconfig document and record its _id value.
```
db.config.find({ kind: "systemconfig" })
```

Update the document to set installerMode to self-hosted. Replace <doc_id> with the _id from the previous step.

db.config.updateOne(
  { kind: "systemconfig", _id: "<doc_id>" },
  { $set: { "spec.installerMode": "self-hosted", "status.updateTime": new Date() } }
)

Confirm the update.

db.config.find({ kind: "systemconfig" })

Exit mongosh and stop the port-forward.
Clear the Hubble pod cache using the procedure in Clear the Hubble Pod Cache.

Validate the workaround with the following checks.

The System Cluster section is no longer shown in the destination system console. Compare it against a system console from a fresh appliance environment if needed.
The response from <VIP>/v1/auth/org no longer contains "appEnv": "enterprise-cli".

Default Pack Registry Conflict

If the source cluster used an external registry, such as Amazon Elastic Container Registry (ECR), as the default pack registry, the destination cluster inherits that record after the restore, replacing the internal Zot registry that the destination cluster was installed with. The Palette UI blocks editing the default record to point to a Zot registry because of the registry type conflict, so the reconciliation must be applied directly in MongoDB.

In the destination cluster system console, go to Tenant Settings > Registries > Pack Registries and create a new pack registry that points at the destination Zot registry. Give it a distinct name, for example dst-zot.

On the jump host, port-forward the MongoDB service and connect with mongosh, following steps 1 through 3 of the Apply directly in MongoDB procedure in the CLI-Installed Enterprise Cluster Source workaround.

Confirm both the current default record and the new record. Replace <default> with the name of the current default record.
```
db.registries.findOne({ "metadata.name": "<default>" })
db.registries.findOne({ "metadata.name": "dst-zot" })
```

Mark the new dst-zot record as the default in metadata so that the spec copy in the next step passes UI validation.

db.registries.updateOne(
  { "metadata.name": "dst-zot" },
  {
    $set: {
      "metadata.annotations.isDefault": "true",
      "spec.registryMeta.isDefault": true,
      "spec.registryMeta.containsSpectroManifest": true
    }
  }
)

Copy the dst-zot spec and status into the existing default record. Replace <default> with the same name used in step 3.

const zot = db.registries.findOne({ "metadata.name": "dst-zot" });

db.registries.updateOne(
  { "metadata.name": "<default>" },
  {
    $set: {
      spec: zot.spec,
      status: zot.status
    }
  }
);

Confirm both records reflect the intended Zot values.

db.registries.findOne({ "metadata.name": "<default>" })
db.registries.findOne({ "metadata.name": "dst-zot" })

In the Palette system console, confirm that the default pack registry details now show the destination Zot registry, then sync the registry. After the sync completes, review cluster profiles for any errors or warnings.

Remove the temporary record.

db.registries.deleteOne({ "metadata.name": "dst-zot" })

Exit mongosh and stop the port-forward.
Clear the Hubble pod cache using the procedure in Clear the Hubble Pod Cache.

Private Cloud Gateway Workload Clusters

Workload clusters deployed through a Private Cloud Gateway (PCG) may continue to reconcile against stale controller state on the destination cluster, even when the source cluster used an FQDN for the system address. Restart the affected controllers on the destination cluster so that they shed the stale caches.

warning

Confirm the exact deployment names and namespaces with Spectro Cloud engineering before applying this workaround in production. The list below is based on field observations and may need to be adjusted for your PCG topology and enabled infrastructure providers.

Restart the following deployments on the destination cluster.

cluster-management-agent
palette-controller-manager
jet
spectro-cloud-driver
The CAPI controller and each active CAPI infrastructure provider, for example capv-controller-manager for VMware.

For each deployment, run the following command. Replace <name> with the deployment name and <ns> with its namespace.

kubectl --kubeconfig dst.kubeconfig rollout restart deployment <name> --namespace <ns>

Wait for each rollout to report all replicas ready, then verify that PCG workload clusters return to a Healthy status.

Supported Scenarios​

Prerequisites​

Back Up the Source Cluster​

Pause Agent Upgrade on the Source Cluster​

Take a Database Backup​

Restore to the Destination Cluster​

Synchronize the Zot Registry​

Restore the Database Backup​

Update DNS and Re-Register Workload Clusters​

Resume Agent Upgrade in Stages​

Post-Restore Workarounds​

Clear the Hubble Pod Cache​

CLI-Installed Enterprise Cluster Source​

Default Pack Registry Conflict​

Private Cloud Gateway Workload Clusters​