Edge Cluster Upgrade Behavior
When you update an active Edge cluster's profile, Palette will upgrade the active cluster to the latest version of the profile. Depending on the nature of the change, implementing an upgrade might involve repaving a cluster, rebooting a cluster, restarting services, or doing nothing. For more information about cluster repaves, refer to Repave Behavior and Configurations. For more information about how to update a cluster profile, refer to Update a Cluster.
- If a cluster's Kubernetes service is down, updates to the cluster's profile will not get applied to the cluster. You must fix the issue impacting the Kubernetes service first. Once the Kubernetes service is back to being operational, the Palette agent will apply the updates from the cluster profile to the cluster.
Known Issues
- For RKE2 clusters, updates to the
stages.*section in the Operating System (OS) and the Kubernetes pack of the cluster profile will trigger a repave instead of a reboot. The only exception is changes to thestages.reconcile.*section, which will behave as expected and trigger a configuration reload.
A/B Partitioning in Upgrades
Edge clusters created with Kairos-based provider images use A/B system partitions to handle upgrades. In an A/B partitions system, the Edge host retains two bootable partitions: an active partition and a passive partition. In a normal startup sequence, the bootloader will load the active image. In the event that the active image is not bootable, the bootloader will load the passive image.
During upgrades, the image that the cluster is upgrading to becomes a transitional image. When the upgrade is successful, the transitional image becomes the new active image, while the old active image becomes the new passive image. The old passive image is then discarded.
Upgrade Behaviors
A cluster could respond to an upgrade in several ways. The table below lists the potential upgrade behaviors you could encounter.
| Upgrade Behavior | Description |
|---|---|
| Repave | Repaves all nodes in the cluster, starting with the control plane. |
| Reboot | Reboots all nodes in the cluster, starting with the control plane. |
| Service restart | Restarts specific services in all nodes in the cluster, starting with the control plane. |
| Configuration reload | Updates the configurations without needing to restart services or the node. This is only triggered when you update stage.reconcile.* in the Operating System (OS) layer. |
| No operations (No-op) | Acknowledges the update request but does nothing. |
Upgrade Behavior by Change Location
Updates in the OS or Kubernetes layers can trigger different upgrade behaviors depending on the exact values changed.
OS Layer
| Repave | Reboot | Service Restart | No-op |
|---|---|---|---|
- options.system.uri | - stages.rootfs.* - stages.initramfs.*- stages.boot.*- stages.fs.*- stages.network.*- stages.after-install.*- stages.after-install-chroot.*- stages.after-upgrade.*- stages.after-upgrade-chroot.*- stages.after-reset.*- stages.after-reset-chroot.*- stages.before-install.*- stages.before-upgrade.*- stages.before-reset.* | None. | - pack.*- providerCredentials.*- options.system.registry- options.system.repo- options.system.k8sDistribution- options.system.osName- options.system.peVersion- options.system.customTag- options.system.osVersion |
Changes to any other parameters that are used by the options.system.uri parameter will also trigger a cluster repave.
For example, if your options.system.uri parameter is
{{ .spectro.pack.edge-native-byoi.options.system.registry }}/{{ .spectro.pack.edge-native-byoi.options.system.repo }}:{{ .spectro.pack.edge-native-byoi.options.system.k8sDistribution }},
changes to .spectro.pack.edge-native-byoi.options.system.registry will trigger a cluster repave because it changes the
options.system.uri parameter.
Kubernetes Layer
- PXK-E
- K3s
- RKE2
| Repave | Reboot | Service Restart | No-op |
|---|---|---|---|
| None. | - stages.rootfs.*- stages.initramfs.*- stages.boot.*- stages.fs.*- stages.network.*- stages.after-install.*- stages.after-install-chroot.*- stages.after-upgrade.*- stages.after-upgrade-chroot.*- stages.after-reset.*- stages.after-reset-chroot.*- stages.before-install.*- stages.before-upgrade.*- stages.before-reset.* | - cluster.* | - clientConfig.* - pack.* |
| Repave | Reboot | Service Restart | No-op |
|---|---|---|---|
| None. | - stages.rootfs.*- stages.initramfs.*- stages.boot.*- stages.fs.*- stages.network.*- stages.after-install.*- stages.after-install-chroot.*- stages.after-upgrade.*- stages.after-upgrade-chroot.*- stages.after-reset.*- stages.after-reset-chroot.*- stages.before-install.*- stages.before-upgrade.*- stages.before-reset.* | - cluster.* | - clientConfig.* - pack.* |
| Repave | Reboot | Service Restart | No-op |
|---|---|---|---|
| None. | - stages.rootfs.*- stages.initramfs.*- stages.boot.*- stages.fs.*- stages.network.*- stages.after-install.*- stages.after-install-chroot.*- stages.after-upgrade.*- stages.after-upgrade-chroot.*- stages.after-reset.*- stages.after-reset-chroot.*- stages.before-install.*- stages.before-upgrade.*- stages.before-reset.* | - cluster.* | - clientConfig.* - pack.* |
Network Layer
Changes made to the Container Network Interface (CNI) pack typically do not result in cluster repave or reboot, and can be applied by restarting the related networking services.
Do not change to a different CNI pack after provisioning a cluster. You can make changes to the existing CNI pack, but if you want to use a different CNI pack altogether, we recommend you create another cluster.
Storage Layer
Changes made to the storage layer typically do not result in cluster repave or reboot and can be applied by restarting the related storage services.
Do not change to a different storage pack after provisioning a cluster. You can make changes to the existing storage pack, but if you want to use a different storage pack altogether, we recommend you create another cluster.
Decoupled Control Plane and Worker Node Upgrades
Edge Native clusters that use Palette eXtended Kubernetes Edge (PXK-E) or Canonical Kubernetes support upgrading the control plane independently from worker pools. You can enable the Skip worker node update (Optional) toggle on individual worker pools to defer their Kubernetes upgrade while the control plane advances. This decouples worker upgrades from control plane upgrades, which lets you move the control plane forward, for example to apply a security patch, without repaving worker nodes. It also reduces the number of worker upgrades needed to cross several Kubernetes minor versions.
Decoupled control plane and worker node upgrades are supported for PXK-E and Canonical Kubernetes only. K3s and RKE2 clusters are not supported.
Support also depends on how the cluster is managed.
| Cluster management | Support |
|---|---|
| Connected (centrally managed) | Supported. Configure the toggle in Palette. |
| Locally managed, airgapped | Supported. Configure the toggle in Local UI or through the Local UI API. |
When a cluster profile update bumps the Kubernetes version, the control plane and any worker pools that do not have Skip worker node update enabled are upgraded. Worker pools with the toggle enabled are skipped and stay at their current Kubernetes version. Control plane nodes are always upgraded to the Kubernetes and OS versions declared in the cluster profile, and are upgraded one at a time.
A skipped worker pool can still be updated on Day-2 for other node configuration changes, such as taints and labels. Its Kubernetes and OS versions stay unchanged.
The Kubernetes N-3 minor version skew is enforced. If a profile update would push a worker pool more than three minor versions behind the control plane, the update is rejected and the profile is not applied to the cluster. Disable the toggle on the affected pools first.
Scale-up is not permitted while the toggle is enabled. Scale-up requests on a pool with the toggle enabled are rejected, whether initiated manually or by the cluster autoscaler, because a new node cannot honor the pool's pinned Kubernetes version. To expand capacity, create a new worker pool and add Edge hosts to it instead.
Upgrade a Skipped Worker Pool
To sync the Kubernetes version of a skipped worker pool with the current Kubernetes control plane version, disable the Skip worker node update toggle on that pool.
Disabling Skip worker node update triggers a repave of the worker pool. Ensure you are ready to repave before disabling the toggle.
For configuration details, refer to Skip Worker Node Update. For step-by-step instructions to trigger the upgrade, refer to Trigger Worker Node Upgrade.