Known Issues
Throughout the development lifecycle of Palette, known issues may arise that affect the user experience. Use this page to review and stay informed about the status of known issues in Palette. As issues are resolved, this page is updated.
Active Known Issues
The following table lists all known issues that are currently active and affecting users.
| Description | Workaround | Publish Date | Product Component |
|---|---|---|---|
| If the Palette management plane briefly loses access to its database, the Palette agent on a workload cluster can delete the namespaces that Palette created through cluster Role-Based Access Control (RBAC) settings or a workspace. After the management plane regains access to its database, the agent recreates the namespaces empty on its next reconciliation, which runs every 2 minutes. The deletion removes all resources in the namespaces, including workloads, Helm releases, ConfigMaps, and Secrets. Recreating the namespaces does not restore them. The agent does not delete namespaces that existed on the cluster before you added them in Palette. The management plane can lose access to its database during Palette upgrades or whenever the MongoDB replica set loses its primary, even briefly. On self-hosted Palette and Palette VerteX, for example, a MongoDB crash, a failover, or overlapping node maintenance can cause this. | On each workload cluster, remove the label that marks a namespace as created by Palette. The Palette agent does not delete a namespace without this label and does not add the label back. As a result, removing the namespace from the cluster RBAC or workspace settings no longer deletes it from the cluster. For the steps, refer to Troubleshooting - Cluster Deployment. | November 6, 2026 | Clusters, Workspaces |
On Edge clusters running Palette agent version 4.10.3 or later, each provider image download and extraction attempt has a fixed time limit of 45 minutes, or 6 hours in Palette agent version 4.10.10 and later. On connections with limited bandwidth, downloading and extracting a large provider image can exceed the limit, and the attempt fails with context deadline exceeded. The Palette agent does not resume partially completed downloads, so each new attempt starts from the beginning. Agent upgrades can fail repeatedly, and the affected cluster can remain in the Provisioning state. | To avoid the time limit, make the provider image available locally on the Edge host instead of downloading it over a bandwidth-constrained connection. Use palette content build to package the provider image into a content bundle, transfer the bundle to the affected Edge host, and extract it into the local bundle store. The Palette agent then uses the local image instead of downloading it. For new Edge hosts, you can also include the content bundle in the Edge Installer ISO. Alternatively, increase the network bandwidth available to the Edge host so that the download and extraction complete within the time limit. For more information, refer to Troubleshooting - Edge. | October 23, 2026 | Edge |
| If the Palette agent on a Palette Virtual Cluster upgrades before the Palette agent on its host cluster, the virtual cluster's agent upgrades components that it shares with the host cluster. This can happen when Pause Agent Upgrades is enabled on the host cluster but not on the virtual cluster. The host cluster can no longer scale, repave, or replace its nodes, and its virtual clusters stop applying cluster profile changes. Workloads continue to run. | Upgrade the Palette agent on a host cluster before the agents on its virtual clusters, and keep Pause Agent Upgrades enabled on the virtual clusters until the host cluster's agent upgrade completes. Refer to Agent Upgrades for Virtual Clusters for the steps. If a host cluster is already affected, disable Pause Agent Upgrades on the host cluster. After its agent upgrade completes, the host cluster recovers, and its virtual clusters can finish their agent upgrades. | October 15, 2026 | Virtual Clusters |
On Edge clusters running Linux kernel 6.19 or later, including Hadron 7.1.3 and Ubuntu 7.0.0-generic, shared Read-Write-Many (RWX) volumes served by an older NFS server fail to open with a remote I/O error. The Longhorn CSI pack at versions 1.8.x through 1.11.2, the Piraeus RWX driver, and Rook CephNFS are affected because Rook CephNFS serves shares through the NFS Ganesha image. The in-cluster Zot registry enters a CrashLoopBackOff state because it cannot read its shared volume, image pulls from the registry VIP fail with connection refused or ImagePullBackOff, and Kubernetes upgrades hang because the shared volume is broken at the NFS layer. Block, hostPath, Rook RBD and CephFS, Portworx, Trident, and HPE volumes are not affected, and pre-6.19 kernels such as stock Ubuntu 6.8 and RHEL 9 do not trigger the issue. | Upgrade the Longhorn CSI pack to version 1.12.0. Version 1.11.3 also contains the fix but requires Kubernetes v1.34 or later. No fixed Piraeus RWX pack is available yet, so avoid Piraeus shared RWX volumes on kernel 6.19 or later. Read-Write-Once (RWO) and DRBD volumes are unaffected. For an interim node-level workaround, refer to Troubleshooting - Edge. | September 6, 2026 | Edge, Packs |
On Edge clusters deployed in airgap mode at Kubernetes v1.35.x or later, on all three Edge Kubernetes distributions (Palette Optimized K3s, Palette Optimized RKE2, and Palette eXtended Kubernetes Edge (PXK-E)), pods may intermittently fail to start with an ImagePullBackOff status even though the image is already present on the node. The palette-webhook and palette-lite-controller-manager pods are the most commonly affected. Kubernetes v1.35 enables the KubeletEnsureSecretPulledImages feature gate, which can record an image loaded from the Palette content bundle without credential information and force Kubelet to attempt a new pull from the image's original public registry. On K3s and RKE2, pack version 1.35.6 includes an Airgap preset that resolves the issue, but the preset is not enabled by default, so clusters on 1.35.6 remain affected until you enable it, and pack versions 1.35.2 and 1.35.3 do not include it at all. On PXK-E, no released pack version includes the preset, so the setting must be applied as a values override, which additionally requires setting the Kubelet --config-dir argument. | On K3s or RKE2 pack version 1.35.6 or later, set the Airgap preset to Enable on the Kubernetes layer of your cluster profile before you provision or upgrade the cluster. On pack versions 1.35.2 and 1.35.3, apply the equivalent Kubelet configuration as a values override instead. On PXK-E, apply the values override on any pack version. For both approaches, and for the recovery steps for a cluster that is already affected, refer to Troubleshooting - Edge. | August 12, 2026 | Edge, Packs |
During Day-1 cluster provisioning with an addon pack that uses Helm 4.0, helm history may temporarily show multiple failed revisions before a successful revision. Helm 4.0 applies a chart's manifests in an order that can submit custom resources before their validating webhook pods are ready, causing the early revisions to fail. This is a known Helm 4.0 behavior tied to in-chart manifest install order, not a product defect. The MetalLB (Helm) pack is currently affected, where the IPAddressPool custom resource fails validation until the MetalLB webhook pod is ready. | No workaround is required. The pack install completes on its own once the validating webhook is ready, and the final cluster state is healthy. The failed revisions remain visible in helm history for observability but do not affect the deployed cluster. | July 30, 2026 | Packs, Clusters |
Edge clusters that use an in-cluster Zot or Harbor primary registry serve a TLS certificate with a one-year validity period. On zot pack version 0.1.89 or earlier, and on harbor pack version 1.19.0 or earlier, this certificate does not renew automatically and must be rotated manually before each 365-day expiration. When the certificate expires, registry connectivity breaks. Clusters can no longer pull images, and workload deployments, scale-out operations, node replacements, and cluster repaves fail until the certificate is replaced. At remote or unattended sites, recovery often requires on-site intervention during an unplanned outage. Clusters approaching one year since deployment, or since their last manual certificate rotation, are affected first. | Upgrade to zot pack version 0.1.89-rev2 or harbor pack version 1.19.0-rev1. These versions manage the certificate lifecycle through cert-manager, which renews the certificate automatically 15 days before it expires. For the upgrade steps, refer to Enable Automatic TLS Certificate Rotation on the Zot Primary Registry and Enable Automatic TLS Certificate Rotation on the Harbor Primary Registry. We recommend upgrading during a scheduled maintenance window, prioritizing the clusters closest to their one-year certificate anniversary. The upgrade involves a pack version change and a pod restart, so validate it on a non-production cluster first. If you need help identifying affected clusters, contact support@spectrocloud.com. | July 30, 2026 | Edge, Packs |
When migrating VMs to a Virtual Machine Orchestrator (VMO) cluster backed by block-based storage such as LINSTOR/DRBD, migrations can fail at the ConvertGuest phase for Linux guests with an nbdkit minimum block-size error. This issue affects VMO clusters deployed with the PaletteAI VM Launchpad appliance (formerly Launchpad for VMs). | Refer to Scenario - VM Migration Fails During Guest Conversion on Block-Based Storage for the workaround. To restore live migration for migrated VMs, contact your Spectro Cloud support representative to obtain the updated forklift-virt-v2v content bundle. | July 3, 2026 | Virtual Machine Orchestrator, PaletteAI VM Launchpad |
| During an interactive installation using the Edge TUI, the disk checks screen does not refresh after a disk wipe. If you select a disk to wipe and continue, the disk is wiped successfully, but selecting ESC to return to the checks screen displays stale data that still lists the previous partitions. | The disk is wiped as expected even though the checks screen displays stale partition data, and you can continue the installation without restarting. To display the refreshed disk status, exit and restart the interactive installer. | July 4, 2026 | Edge |
| During an interactive installation using the Edge TUI, selecting all disks to be wiped does not always fully erase pre-existing data. Intermittently, raw block data written to a non-target disk before installation remains readable at the block level after the wipe and reinstall complete. This issue does not occur on the first installation attempt. | No workaround available. | July 4, 2026 | Edge |
Cluster nodes fail to become Ready on Kubernetes v1.35.x on OSes that default to cgroup v1 because Kubelet fails to start. Affected OSes include Ubuntu 20.04 and earlier, RHEL/CentOS 7–8, SLES 15 SP3 and earlier, Debian 10 and earlier, and Amazon Linux 2. | For all Kubernetes v1.35.x clusters, enable cgroup v2 instead of cgroup v1, or configure Kubelet with failCgroupV1: false where supported. For a detailed workaround for Edge clusters, refer to Troubleshooting - Edge. | May 3, 2026 | Clusters |
| Existing EKS clusters with hybrid nodes cannot upgrade the EKS control plane to version 1.35, because the bundled Nodeadm pack supports Kubernetes only up to 1.31, exceeding the maximum skew of three minor versions between nodes and control plane. | Keep the EKS control plane within the supported skew range for the bundled Nodeadm pack. For example, Nodeadm 1.31 supports up to EKS version 1.34. | May 3, 2026 | Clusters, Packs |
| Zot registry content may appear missing after storage changes or Helm upgrades, even though the original content bundle still exists on the node. | Force a re-sync by deleting the /usr/local/spectrocloud/.local-sync-state file. For Palette and VerteX Management Appliance, delete this file on all Management Appliance nodes in the cluster. stylus-operator automatically detects the change and re-syncs the content to Zot. Confirm that the content appears in the Local UI Content tab. | April 5, 2026 | Edge |
| The cluster's Virtual Machines tab or the Virtual Machine Orchestrator (VMO) Graphical UI (GUI) may fail to load for self-hosted Palette and Palette VerteX installations. This issue applies to all installation methods but does not apply to self-hosted environments that use an IP address instead of a domain name. | Refer to Scenario - VMO Loading Errors in Self-Hosted Palette for the workaround. | April 5, 2026 | Self-Hosted, Virtual Machine Orchestrator |
| Clusters configured with Palette eXtended Kubernetes Edge (PXK-E) pack version 1.33.6 and an OIDC provider fail to deploy successfully. | Upgrade to newer Palette eXtended Kubernetes Edge (PXK-E) pack versions. Alternatively, if remaining on PXK-E version 1.33.6, refer to Scenario - Configure OIDC for Palette eXtended Kubernetes Edge (PXK-E) 1.33.6 for the workaround. | February 6, 2026 | Packs, Clusters |
The spectrocloud_registry_oci resource or data source continuously detects changes and repeatedly reconciles when created or imported using the Crossplane provider with is_synchronization enabled. The Terraform plan operation and audit logs also incorrectly report changes. This affects Spectro Cloud Terraform provider and Spectro Cloud Crossplane provider version 0.27.0 or later. | No workaround available. | January 30, 2026 | Terraform, Crossplane |
Palette Management Appliance and VerteX Management Appliance installations may become stuck due to ErrImagePull errors on the Calico pods. This affects all Palette versions later than 4.8.0. | Restart the node on which the pod is scheduled and wait for the pod to recover. | January 27, 2026 | Clusters, Packs |
AWS EKS clusters using Kubernetes (EKS) version 1.33 with Palette as an Identity Provider (IdP) fail to deploy. This is due to IdentityProviderConfigName not being automatically injected into the values.yaml file. | In the managedControlPlane.oidcIdentityProvider section of your cluster profile's Kubernetes layer, add the value identityProviderConfigName: "eks-oidc". | December 30, 2025 | Packs |
Velero backups may fail with errors Timed out awaiting reconciliation of volumesnapshot hipster-app/velero-data-redis-cart-2-zctck. This occurs because the CloudStack CSI Helm chart is missing the required csi-snapshotter sidecar container in the controller deployment, preventing CSI snapshot operations from executing. As a result, all CSI-based snapshots and any Velero backups that rely on them time out and fail. | No workaround available. | December 17, 2025 | Packs |
After upgrading to Palette Management Appliance or VerteX Management Appliance 4.8.10, the new stylus version is not reflected in Local UI, which continues to display the previous value of 4.7.16. | No workaround available. | December 12, 2025 | Edge |
| Due to strict schema adherence enforcement, Helm charts with parameters that do not exist in the chart schema fail to install on Palette 4.7.15 or later. | Remove parameters that do not exist in the chart schema from the pack values. Alternatively, add the missing parameters to the chart schema or remove the chart schema file entirely. | September 20, 2025 | Packs |
After an OS image upgrade in appliance mode, an Edge host may fail to boot into the expected active system image and instead boot into the passive partition as a fallback due to an upgrade failure. When this happens, the Edge host does not automatically rejoin the cluster. The kernel command line (/proc/cmdline) includes the upgrade_failure flag and confirms the system root is set to LABEL=COS_PASSIVE. | Recover the Edge host manually using one of the following methods: - Reboot the host and select Palette eXtended Kubernetes – Edge at the GRand Unified Bootloader (GRUB) menu to boot the active image. - Establish an SSH connection to the host and run /usr/bin/grub2-editenv /oem/grubenv set next_entry=cos && reboot. This command updates GRUB to use the boot entry labeled cos (the active image) and reboots the host. | September 1, 2025 | Edge |
| In self-hosted Palette and Vertex Management Appliance environments, uploading the same pack as both a FIPS and non-FIPS version to the same registry overwrites the original pack. For example, if you have a non-FIPS version of the byoi-2.1.0 pack in your Zot registry and you upload the FIPS version of byoi-2.1.0, the new version will overwrite the existing one. This results in a SHA mismatch between the pack stored in the registry and the pack referenced in the cluster profile, which can lead to cluster creation failures. | Upload either a FIPS or non-FIPS version of a pack to your registry. Do not upload both to the same registry. | September 1, 2025 | Clusters, Self-Hosted |
Container runtime may fail to run with the message Failed to run CRI service error=failed to recover state: failed to get metadata for stored sandbox after a node is upgraded to 1.29.14. This is related to an upstream issue with containerd. | Remove the container runtime folder with rm -rf /var/lib/containerd. Then restart containerd and Kubelet using systemctl restart containerd && systemctl restart kublet. | August 17, 2025 | Edge |
Due to an upstream issue with a Go library and CLIs for working with container registries, unintended or non-graceful reboots during content push operations to registries can cause consistency issues. This leads to content sync in locally managed clusters throwing the content-length: 0 error. | Refer to Troubleshooting - Edge for the workaround. | August 17, 2025 | Edge |
| Controller mode MAAS deployments using Canonical Kubernetes versions earlier than 1.35 with the Canonical Kubernetes pack automatically install the Cilium CNI. This happens because of a known issue with the Canonical Kubernetes Cluster API (CAPI) bootstrap provider and cannot be disabled. However, Palette still requires users to explicitly configure a CNI in the cluster profile. | Select the Cilium CNI (Canonical Kubernetes) pack when creating a cluster profile to fulfill the CNI requirement. Palette recognizes this selection and allows cluster creation to proceed, even though Cilium is installed by the bootstrap process. Canonical Kubernetes 1.35 and later support the first-class Cilium pack, which you can configure and manage directly in the cluster profile. | August 17, 2025 | Clusters, Packs |
Canonical Kubernetes clusters on MAAS fail to provision on machines whose primary interface is a br0 bridge. Cilium cannot detect the direct routing device, and the agent exits with daemon creation failed: failed to detect devices: unable to determine direct routing device. Use --direct-routing-device to specify it. Canonical Kubernetes supports several Cilium annotations, but a limitation in k8s-snap prevents Palette from applying the fields that the MAAS br0 network configuration requires. | Provision Canonical Kubernetes clusters on MAAS machines whose primary interface connects directly to the MAAS-managed subnet or VLAN, instead of machines with a br0 bridge interface. Refer to the Cilium pack documentation for more information. | August 17, 2025 | Clusters, Packs |
| When deploying an Edge RKE2 cluster on Rocky Linux, a worker node may fail to join the cluster if TCP port 9345 is not open on the control plane node. This port is required for communication between the RKE2 agent and the control plane. | Verify if the port is open by running firewall-cmd --list-all on the control plane node. If 9345/tcp is not listed in the output, open it with firewall-cmd --zone=public --add-port=9345/tcp --permanent and apply the change using firewall-cmd --reload. | July 21, 2025 | Edge |
When using the Palette/VerteX Management Appliance, clicking on the Zot service link in Local UI results in a new tab displaying Client sent an HTTP request to an HTTPS server. | Change the prefix of the URL in your web browser to https:// instead of http://. | July 21, 2025 | Clusters, Packs |
| Cloning a virtual machine using KubeVirt 1.5 or later may hang if volume snapshots are not configured. | Ensure that you configure a VolumeSnapshotClass in the charts.virtual-machine-orchestrator.snapshot-controller.volumeSnapshotClass resource in the Virtual Machine Orchestrator pack. | July 19, 2025 | Virtual Machine Orchestrator |
Edge K3s clusters may fail kube-bench tests even when they are expected to pass. These failures do not indicate security issues, but rather stem from how the tests are executed. | No workaround available. | July 11, 2025 | Edge |
| Palette eXtended Kubernetes - Edge (PXK-E) clusters running Kubernetes v1.32.x or later on RHEL or Rocky Linux 8.x may experience failure during Kubernetes initialization due to unsupported kernel version. | Use RHEL or Rocky Linux 9.x as the base OS or update the kernel version to 4.19 or later in the 4.x series, or to any 5.x or 6.x version. Refer to Troubleshooting - Edge for debug steps. | June 23, 2025 | Edge |
Palette eXtended Kubernetes - Edge (PXK-E) control plane nodes in VerteX clusters may experience failure of the kube-vip component after reboot. | Refer to Troubleshooting - Edge for debug steps. | June 23, 2025 | Edge |
| The Pause Agent Upgrades configuration is not applied to Edge hosts that are not part of a cluster. Edge hosts that are part of a cluster are not affected. | No workaround. | June 23, 2025 | Edge |
| Palette eXtended Kubernetes (PXK) and Palette eXtended Kubernetes - Edge (PXK-E) versions 1.30.10, 1.31.6, and 1.32.2 or older do not support TLS 1.3 or applications that require TLS 1.3 encrypted communications. | Use PXK and PXK-E versions 1.30.11, 1.31.7, and 1.32.3 or later instead. | June 5, 2025 | Edge |
| Clusters with Pause Agent Upgrades enabled may be stuck in the Deleting state. Cluster resources will not be deleted without manual intervention. | Disable the Pause Agent Upgrades setting and trigger the cluster deletion. | May 31, 2025 | Clusters |
| Edge clusters using K3s version 1.32.1 or 1.32.2 may fail to provision due to an upstream issue. Refer to the K3s issue page for more information. | No workaround available. | May 31, 2025 | Edge |
For clusters deployed with Palette eXtended Kubernetes - Edge (PXK-E) and agent mode using the FIPS installation package, adding a custom stylus.path to the user-data file causes cluster creation to fail as it cannot find kubelet. | No workaround available. | May 31, 2025 | Edge |
During a Kubernetes upgrade, the Cilium pod may get stuck in the Init:CrashLoopBackoff state due to nsenter permission issues. | Refer to Troubleshooting - Edge for debug steps. | May 31, 2025 | Edge |
Appliance Studio does not validate the value of each field in .arg or user-data files. | No workaround available. | May 31, 2025 | Edge |
| Palette virtual clusters provisioned with EKS clusters as host clusters in the cluster group and using the Calico CNI are stuck in the Provisioning state due to Cert Manager not being reachable. This stems from an upstream limitation between Cert Manager on EKS and custom CNIs. | No workaround available. | May 21, 2025 | Edge |
| Remote shell sessions executing in the Chrome and Microsoft Edge browsers time out after approximately five minutes of inactivity. | Start remote shell sessions in the Firefox browser instead. Firefox supports a 12 hour inactivity timeout. | May 5, 2025 | Edge |
| When you enable remote shell on an Edge host, the remote shell configuration may become stuck in the Configuring state. | Disable remote shell in the UI, and wait for one minute before enabling it again. | April 19, 2025 | Edge |
| Disconnected Edge clusters using PXK-E version 1.29.14 or 1.30.10 will sometimes go into the unknown state after a reboot. | Use the command kubectl delete pod kube-vip-<node-name> --namespace kube-system to delete the Kubernetes VIP pod and let it be re-created automatically. Replace node-name with the name of the host node. | March 15, 2025 | Edge |
| MAAS and VMware vSphere clusters fail to provision on existing self-hosted Palette and VerteX environments deployed with Palette 4.2.13 or later. These installations have an incorrectly configured default image endpoint, which causes image resolution to fail. New self-hosted installations are not affected. | Refer to Troubleshooting for a workaround for non-airgap environments. For airgap environments, ensure that the images are downloaded to your environment. Refer to the Additional OVAs page for further details. | February 16, 2025 | Self-Hosted, Clusters |
For clusters deployed with Virtual Machine Orchestrator (VMO), namespaces on the Virtual Machine tab cannot be viewed by users with any spectro-vm cluster role. | Add the spectro-namespace-list cluster role to users who need to view virtual machines and virtual machine namespaces. Refer to the Add Roles and Role Bindings guide for instructions on how to add roles for VMO clusters. | February 5, 2025 | Virtual Machine Orchestrator |
For clusters deployed with Palette eXtended Kubernetes - Edge (PXK-E) and agent mode, the contents of the /opt/cni/bin folder are not set correctly, causing cluster deployment issues because the cluster network cannot come up. | Refer to Troubleshooting for a workaround. | January 30, 2025 | Palette agent |
For clusters deployed with Palette eXtended Kubernetes - Edge (PXK-E) and agent mode on Palette agent version 4.5.14, adding a custom stylus.path to the user-data file causes cluster creation to fail as it cannot find kubeadm. | Review the Edge Troubleshooting section for workarounds. Refer to Identify the Latest Palette Agent Version for guidance in retrieving your Palette agent version number. | January 19, 2025 | Edge |
| For clusters deployed with Palette eXtended Kubernetes - Edge (PXK-E) and agent mode, upgrades to higher Kubernetes versions are not supported with Palette agent version 4.5.12 or earlier. | No workaround available. Upgrades to higher Kubernetes versions are only supported from Palette agent version 4.5.14 and above for clusters deployed with PXK-E and agent mode. Refer to Identify the Latest Palette Agent Version for guidance in retrieving your Palette agent version number. | January 19, 2025 | Edge |
| Transferring the management of a local Edge cluster to central management by Palette or VerteX is not supported for multi-node clusters. | No workaround is available. | January 19, 2025 | Edge |
| When using the VM Migration Assistant to migrate VMs to your VMO cluster, migration plans can enter an Unknown state if more VMs are selected for migration than the Max concurrent virtual machine migrations setting allows. | Review the Virtual Machine Orchestrator (VMO) Troubleshooting section for workarounds. | January 19, 2025 | Virtual Machine Orchestrator |
| If you have manually configured the metrics server in your Edge airgap cluster using a manifest, upgrading to 4.5.15 may cause an additional metrics server pod to be created in your cluster. | Remove the manifest layer that adds the manifest server from your cluster profile and apply the update on your cluster. | December 15, 2024 | Edge |
| When deploying an Edge cluster using content bundles built from cluster profiles with PXK-E as the Kubernetes layer, some images in the Kubernetes layer fail to load into containerd. This issue occurs due to image signature problems, resulting in deployment failure. | Remove the packs.content.images from the Kubernetes layer in the pack configuration before building the content bundle. These components are already included in the provider image and do not need to be included in the content bundle. | December 13, 2024 | Edge |
| Hosts provisioned in agent mode do not display host information in the console after using the Palette Terminal User Interface to complete host setup. | Local UI is still available and will display host information. Refer to Access Local UI to learn how to access Local UI. | December 12, 2024 | Edge |
| In a multi-node Edge cluster, the reset action on a cluster node does not update the node status on the leader node's linking screen. | Scale down the cluster and free up the follower node before resetting the node. | December 12, 2024 | Edge |
| For Edge airgap clusters, manifests attached to packs are not applied during cluster deployment. | Add the manifest as a layer directly instead of attaching it to a pack. For more information, refer to Add a Manifest. | November 15, 2024 | Edge |
| In some cases, the differential editor incorrectly reports YAML differences for customizations not created by you. The issue is more common when items in a list or array are removed. Clicking the Keep button when non-user-generated customization is the focus causes the button to become unresponsive after the first usage. | Skip differential highlights not created by you. Click the arrow button to skip and proceed. | November 11, 2024 | Cluster Profiles |
| Palette fails to provision virtual clusters on airgapped and proxy Edge cluster groups. This error is caused by Palette incorrectly defaulting to fetch charts from an external repository, which is unreachable from these environments. | No workaround. | November 9, 2024 | Virtual Clusters |
| The resource limits on Palette Virtual Clusters are too low and may cause the Palette agent to experience resource exhaustion. As a result, Palette pods required for Palette operations may experience Out-Of-Memory (OOM) errors. | Refer to the Apply Host Cluster Resource Limits to Virtual Cluster guide for workaround steps. | November 4, 2024 | Virtual Clusters |
| Palette incorrectly modifies the indentation of the Prometheus Operator pack after it is configured as a cluster profile layer. The modified indentation does not cause errors, but you may observe changes to the pack values.yaml. | No workaround available. | October 30, 2024 | Cluster Profiles, Pack |
| Palette does not correctly configure multiple search domains when provided during the self-hosted installation. The configuration file resolve.conf ends up containing incorrect values. | Connect remotely to each node in the Palette self-hosted instance and edit the resolution.conf configuration file. | October 17, 2024 | Self-Hosted, PCG |
| Upgrading the RKE2 version from 1.29 to 1.30 fails due to an upstream issue with RKE2 and Cilium. | Refer to the Troubleshooting section for the workaround. | October 12, 2024 | Edge |
Clusters deployed with Microk8s cannot accept kubectl commands if the Spectro Proxy pack is added to the cluster's cluster profile. This is because Microk8s does not support certSANs, which causes the Kubernetes API server to reject Spectro Proxy certificates. Refer to the GitHub Issue for more information. This applies to existing MicroK8s clusters, because the MicroK8s pack is no longer available in Palette and Palette VerteX. | Use the CLI flag --insecure-skip-tls-verify with kubectl commands or use the admin kubeconfig file to access the cluster API, as it does not use the Spectro Proxy server. | October 4, 2024 | Clusters, Pack |
| Deploying new Nutanix clusters fails for self-hosted Palette or VerteX users on version 4.4.18 or newer. | No workaround is available. | September 26, 2024 | Clusters |
| OCI Helm registries added to Palette or VerteX before support for OCI Helm registries hosted in AWS ECR was available in Palette have an invalid API payload that is causing cluster imports to fail if the OCI Helm Registry is referenced in the cluster profile. | Log in to Palette as a tenant administrator and navigate to the left Main Menu . Select Registries and click on the OCI Registries tab. For each OCI registry of the Helm type, click on the three-dot Menu at the end of the row. Select Edit. To fix the invalid API payload, click on Confirm. Palette will automatically add the correct provider type behind the scenes to address the issue. | September 25, 2024 | Helm Registries |
| Using the Flannel Container Network Interface (CSI) pack together with a Red Hat Enterprise Linux (RHEL)-based provider image may lead to a pod becoming stuck during deployment. This is caused by an upstream issue with Flannel that was discovered in a K3s GitHub issue. Refer to the K3s issue page for more information. | No workaround is available | September 14, 2024 | Edge |
| Persistent Volume Claims (PVCs) metadata do not use a unique identifier for self-hosted Palette clusters. This causes incorrect Cloud Native Storage (CNS) mappings in vSphere, potentially leading to issues during node operations and cluster upgrades. | Refer to the Troubleshooting section for guidance. | September 13, 2024 | Self-hosted |
| Third-party binaries downloaded and used by the Palette CLI may become stale and incompatible with the CLI. | Refer to the Incompatible Stale Palette CLI Binaries troubleshooting guide for workaround guidance. | September 11, 2024 | CLI |
An issue with Edge hosts using Trusted Boot and encrypted drives occurs when TRIM is not enabled. As a result, Solid-State Drive and Nonvolatile Memory Express drives experience degraded performance and potentially cause cluster failures. This issue stems from Kairos not passing through the --allow-discards flag to the systemd-cryptsetup attach command. | Check out the Degraded Performance on Disk Drives troubleshooting guide for guidance on workaround. | September 4, 2024 | Edge |
| The AWS CSI pack has a Pod Disruption Budget (PDB) that allows for a maximum of one unavailable pod. This behavior causes an issue for single-node clusters as well as clusters with a single control plane node and a single worker node where the control plane lacks worker capability. OS patch updates may attempt to evict the CSI controller without success, resulting in the node remaining in the un-schedulable state. | If OS patching is enabled, allow the control plane nodes to have worker capability. For single-node clusters, turn off the OS patching feature. | September 4, 2024 | Cluster, Packs |
| On AWS IaaS Microk8s clusters, OS patching can get stuck and fail. This applies to existing MicroK8s clusters, because the MicroK8s pack is no longer available in Palette and Palette VerteX. | Refer to the Troubleshooting section for debug steps. | August 17, 2024 | Palette |
When upgrading a self-hosted Palette instance from 4.3 to 4.4 the MongoDB pod may be stuck with the following error: ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible. | Delete the PVC, PV and the pod manually. All resources will be recreated with the correct configuration. | August 17, 2024 | Self-Hosted Palette |
| For existing clusters that have added a new machine and all new clusters, pods may be stuck in the draining process and require manual intervention to drain the pod. | Manually delete the pod if it is stuck in the draining process. | August 17, 2024 | Palette |
Clusters with the Virtual Machine Orchestrator (VMO) pack may experience VMs getting stuck in a continuous migration loop, as indicated by a Migrating or Migration VM status. | Review the Virtual Machine Orchestrator (VMO) Troubleshooting section for workarounds. | August 1, 2024 | Virtual Machine Orchestrator |
Palette CLI users who authenticated with the login command and specified a Palette endpoint that does not contain the tenant name are encountering issues with expired JWT tokens. | Re-authenticate using your tenant URL, for example, https://my-org.console.spectrocloud.com. If the issue persists after re-authenticating, remove the ~/.palette/palette.yaml file that is auto-generated by the Palette CLI. Re-authenticate with the login command if other commands require it. | July 25, 2024 | CLI |
| Adding new cloud providers, such as Nutanix, is currently unavailable. Private Cloud Gateway (PCG) deployments in new Nutanix environments fail to complete the installation. As a result, adding a new Nutanix environment to launch new host clusters is unavailable. This does not impact existing Nutanix deployments with a PCG deployed. | No workarounds are available. | July 20, 2024 | Clusters, Self-Hosted, PCG |
| Single-node Private Cloud Gateway (PCG) clusters are experiencing an issue upgrading to 4.4.11. The vSphere CSI controller pod fails to start because there are no matching affinity rules. | Check out the vSphere Controller Pod Fails to Start in Single Node PCG Cluster guide for workaround steps. | July 20, 2024 | PCG |
| When provisioning an Edge cluster, it is possible that some OS user credentials will be lost once the cluster is active. This is because the cloud-init stages from different sources merge during the deployment process, and sometimes, the same stages without distinct names overwrite each other. | Give each of your cloud-init stages in the OS pack and in the Edge installer user-data file a unique name. For more information about cloud-init stages and examples of cloud-init stages with names, refer to Cloud-init Stages. | July 17, 2024 | Edge |
When you use a content bundle to provision a new cluster without using the local Harbor registry, it's possible for the images to be pulled from external networks instead of from the content bundle, consuming network bandwidth. If your Edge host has no connection to external networks or if it cannot locate the image on a remote registry, some pods may enter the ImagePullBackOff state at first, but eventually the pods will be created using images from the content bundle. | For connected clusters, you can make sure that the remote images are not reachable by the Edge host, which will stop the Palette agent from downloading the image and consuming bandwidth, and eventually the cluster will be created using images from the content bundle. For airgap clusters, the ImagePullBackOff error will eventually resolve on its own and there is no action to take. | July 11, 2024 | Edge |
| When you add a new VMware vSphere Edge host to an Edge cluster, the IP address may fail to be assigned to the Edge host after a reboot. | Review the Edge Troubleshooting section for workarounds. | July 9, 2024 | Edge |
| When you install Palette Edge using an Edge Installer ISO with a RHEL 8 operating system on a VM with insufficient video memory, the QR code in the registration screen does not display correctly. | Increase the video memory of your VM to 8 MB or higher. The steps to do this vary depending on the platform you use to deploy your VM. In vSphere, you can right click on the VM, click Edit Settings and adjust the video card memory in the Video card tab. | July 9, 2024 | Edge |
Custom Certificate Authority (CA) is not supported for accessing AKS clusters. Using a custom CA prevents the spectro-proxy pack from working correctly with AKS clusters. | No workaround is available. | July 9, 2024 | Packs, Clusters |
| Manifests attached to an Infrastructure Pack, such as OS, Kubernetes, Network, or Storage, are not applied to the Edge cluster. This issue does not impact the infrastructure pack's YAML definition, which is applied to the cluster. | Specify custom configurations through an add-on pack or a custom manifest pack applied after the infrastructure packs. | Jul 9, 2024 | Edge, Packs |
| Clusters using Cilium and deployed to VMware environments with the VXLAN tunnel protocol may encounter an I/O timeout error. This issue is caused by the VXMNET3 adapter, which is dropping network traffic and resulting in VXLAN traffic being dropped. You can learn more about this issue in the Cilium's GitHub issue #21801. | Review the Cilium Troubleshooting section for workarounds. | June 27, 2024 | Packs, Clusters, Edge |
| Sonobuoy scans fail to generate reports on airgapped Palette Edge clusters. | No workaround is available. | June 24, 2024 | Edge |
During the platform upgrade from Palette 4.3 to 4.4, Virtual Clusters may encounter a scenario where the pod palette-controller-manager is not upgraded to the newer version of Palette. The virtual cluster will continue to be operational, and this does not impact its functionality. | Refer to the Controller Manager Pod Not Upgraded troubleshooting guide. | June 15, 2024 | Virtual Clusters |
Edge hosts with FIPS-compliant Red Hat Enterprise Linux (RHEL) and Ubuntu operating systems may encounter the error where the systemd-resolved.service service enters the failed state. This prevents the nameserver from being configured, which will result in cluster deployment failure. | Refer to TroubleShooting for a workaround. | June 15, 2024 | Edge |
| The GKE cluster's Kubernetes pods are failing to start because the Kubernetes patch version is unavailable. This is encountered during pod restarts or node scaling operations. | Deploy a new cluster and use a GKE cluster profile that does not contain a Kubernetes pack layer with a patch version. Migrate the workloads from the existing cluster to the new cluster. This is a breaking change introduced in Palette 4.4.0 | June 15, 2024 | Packs, Clusters |
| OIDC integration with Virtual Clusters is not functional. All other operations related to Virtual Clusters are operational. | No workaround is available. | Jun 11, 2024 | Virtual Clusters |
| Deploying self-hosted Palette or VerteX to a vSphere environment fails if vCenter has standalone hosts directly under a data center. Persistent Volume (PV) provisioning fails due to an upstream issue with the vSphere Container Storage Interface (CSI) for all versions before v3.2.0. Palette and VerteX use the vSphere CSI version 3.1.2 internally. The issue may also occur in workload clusters deployed on vSphere using the same vSphere CSI for storage volume provisioning. | If you encounter the following error message when deploying self-hosted Palette or VerteX: 'ProvisioningFailed failed to provision volume with StorageClass "spectro-storage-class". Error: failed to fetch hosts from entity ComputeResource:domain-xyz then use the following workaround. Remove standalone hosts directly under the data center from vCenter and allow the volume provisioning to complete. After the volume is provisioned, you can add the standalone hosts back. You can also use a service account that does not have access to the standalone hosts as the user that deployed Palette. | May 21, 2024 | Self-Hosted |
| Conducting cluster node scaling operations on a cluster undergoing a backup can lead to issues and potential unresponsiveness. | To avoid this, ensure no backup operations are in progress before scaling nodes or performing other cluster operations that change the cluster state | April 14, 2024 | Clusters |
Palette automatically creates an AWS security group for worker nodes using the format <cluster-name>-node. If a security group with the same name already exists in the VPC, the cluster creation process fails. | To avoid this, ensure that no security group with the same name exists in the VPC before creating a cluster. | April 14, 2024 | Clusters |
| When deploying a multi-node AWS EKS cluster with the Container Network Interface (CNI) Calico , the cluster deployment fails. | A workaround is to use the AWS VPC CNI in the interim while the issue is resolved. | April 14, 2024 | Packs, Clusters |
| If a Kubernetes cluster deployed onto VMware is deleted, and later re-created with the same name, the cluster creation process fails. The issue is caused by existing resources remaining inside the PCG, or the System PCG, that are not cleaned up during the cluster deletion process. | Refer to the VMware Resources Remain After Cluster Deletion troubleshooting guide for resolution steps. | April 14, 2024 | Clusters |
| Day-2 operations related to infrastructure changes, such as modifying the node size and count, when using MicroK8s are not taking effect. This applies to existing MicroK8s clusters, because the MicroK8s pack is no longer available in Palette and Palette VerteX. | No workaround is available. | April 14, 2024 | Packs, Clusters |
| If a cluster that uses the Rook-Ceph pack experiences network issues, it's possible for the file mount to become and remain unavailable even after the network is restored. | This a known issue disclosed in the Rook GitHub repository. To resolve this issue, refer to Rook-Ceph pack documentation. | April 14, 2024 | Packs, Edge |
| Edge clusters on Edge hosts with ARM64 processors may experience instability issues that cause cluster failures. | ARM64 support is limited to a specific set of Edge devices. Currently, NVIDIA Jetson devices are supported. | April 14, 2024 | Edge |
| During the cluster provisioning process of new edge clusters, the Palette webhook pods may not always deploy successfully, causing the cluster to be stuck in the provisioning phase. This issue does not impact deployed clusters. | Review the Palette Webhook Pods Fail to Start troubleshooting guide for resolution steps. | April 14, 2024 | Edge |
Resolved Known Issues
The following table lists all known issues that have been resolved and are no longer affecting users. Refer to the table for information on the fix version and the date the issue was resolved.
| Description | Publish Date | Product Component | Fix Version |
|---|---|---|---|
When you hot-plug a disk to a running VM and select Create New Blank Disk, the Virtual Machine Orchestrator (VMO) UI creates the DataVolume with the ReadWriteMany access mode and the Block volume mode. You cannot change either setting. If the StorageClass cannot provision Block volumes, for example because its StorageProfile allows only the Filesystem volume mode, the DataVolume never becomes ready. The VM cannot use the disk, even though VMO displays a message that the volume is attached. This issue affects both the VMO pack and PaletteAI VM Launchpad. For the workaround on earlier PaletteAI VM Launchpad and VMO pack versions, refer to Scenario - Hot-Plugged Blank Disk Never Becomes Ready. | October 12, 2026 | Virtual Machine Orchestrator, PaletteAI VM Launchpad | 4.10.25 |
On Edge clusters running Palette cluster management agent version 4.9.0 or later, the agent could remove a stale node record that still referenced an Edge host in use, which caused Palette to release and reset that host. This interrupted workloads and could result in loss of data stored on the host. Starting with version 4.10.27, Palette rejects a second node record for an Edge host that already has a running one, and the agent no longer removes the node record of a running node. Palette also removes existing duplicate node records automatically after the upgrade. | October 15, 2026 | Edge | 4.10.27 |
| When you add an agent mode Edge host to a new cluster after a reset, kube-vip does not start, the control plane virtual IP (VIP) address of the cluster is not claimed, and cluster provisioning does not complete. A reset occurs, for example, when you delete a cluster, remove a host from its node pool, or reset the host. Refer to Agent Mode Edge Hosts Do Not Claim the Control Plane VIP after a Reset to restore the kube-vip image archive on a host that you reset on a Palette version earlier than 4.10.25. | October 9, 2026 | Edge | 4.10.25 |
| On the non-FIPS Portworx variant of PaletteAI VM Launchpad, the appliance image does not include the NFS server components that Portworx requires for shared Read-Write-Many (RWX) volumes. As a result, VMs cannot attach volumes from a storage class that has Shared (RWX) enabled, and VM live migration fails. | September 25, 2026 | PaletteAI VM Launchpad | 4.10.20 |
| When you move a PXK-E Edge cluster to systemd extensions, custom systemd drop-in files for the containerd and Kubelet services stop applying, and no error is reported. For example, image pulls from a private registry that uses a Kubelet credential provider fail, or a containerd HTTP proxy stops applying. Refer to Custom systemd Drop-In Files Do Not Apply After Migration to systemd Extensions to restore the files on a cluster that moved before version 4.10.25. | October 9, 2026 | Edge | 4.10.25 |
GKE has deprecated the No channel release channel option and no longer allows you to create clusters that are not enrolled in a release channel. Previously, Palette did not set a default release channel, so new GKE cluster deployments failed during provisioning with an InvalidArgument error stating that not enrolling clusters in a release channel is only allowed for existing customers. Starting with 4.10.25, Palette enrolls new GKE clusters in the Regular release channel by default. The Regular channel supports Kubernetes 1.34 and later; to deploy an earlier version, set a supported release channel at the cluster level with a CAPI override. Existing GKE clusters that run without a release channel continue to operate, as GKE supports them until GKE removes the option in June 2027. | October 9, 2026 | Clusters | 4.10.25 |
The order of the packs of a cluster profile that was created with the API or Terraform can change unexpectedly when the profile is opened and saved in the Palette UI. The Palette UI sorts the packs according to their install priority for display. If this order differs from the order in Terraform and you click Save changes in the UI, then Palette persists the displayed order, overwriting the stored order. For profiles managed as code, this later appears as unexpected terraform plan drift, and applying it can mark attached clusters as having updates available and the affected packs unhealthy. | September 6, 2026 | Cluster Profiles, Packs | 4.10.13 |
Edge hosts that rely on multipath storage configurations may experience storage accessibility issues due to a missing multipath-tools package on Edge installer and provider images built with CanvOS version 4.8.8 and later. This issue is caused by the underlying Kairos base image (v3.5.9) used by CanvOS, where a system cleanup step inadvertently removes the multipath-tools package. For the workaround on earlier CanvOS versions, refer to Scenario - Missing multipath-tools Package on Edge Images. | September 6, 2026 | Edge | 4.10.13 |
| On Palette 4.9.14 and later, clusters imported with the Generic cloud type do not deploy add-on cluster profiles. The cluster imports successfully and continues to report health, cost, and event data, so there is no visible signal of the failure. Clusters imported as AWS IaaS, Azure IaaS, or GCP IaaS are not affected. | August 6, 2026 | Clusters | 4.9.41 |
After upgrading to Palette 4.9.38, connected workload clusters can permanently fail to reconcile. A race during the rolling upgrade of palette-controller-manager reverts the palette-webhook deployment to an older image that does not serve the v1alpha1 to v1 conversion the cluster.spectrocloud.com Custom Resource Definitions (CRDs) now require. SpectroCluster reconcile then fails with conversion webhook for cluster.spectrocloud.com/v1alpha1 … failed: the server could not find the requested resource, the cluster appears grey on the Clusters page in the Palette UI, and cluster profile updates do not apply. Only Palette agent upgrades on existing clusters are affected. For recovery steps, refer to Scenario - Palette Webhook Downgraded After Rolling Upgrade. | August 6, 2026 | Clusters, Self-Hosted | 4.9.41 |
GCP clusters intermittently take far longer to provision than expected. This is due to the control plane being repeatedly repaved due to a timing dependency between kubeadm init and the GCP load balancer. For the workaround on earlier Palette versions, refer to Scenario - GCP IaaS Control Plane Repaved Repeatedly. | July 30, 2026 | Clusters | 4.9.38 |
| During PaletteAI VM Launchpad cluster installation, you must enter a complex password in all three Keycloak fields. The installation fails if any password does not meet the complexity requirements. | June 11, 2026 | PaletteAI VM Launchpad | 4.9.22 |
| Headlamp does not support the Custom or Inherit from Tenant OIDC Identity Provider options configured in the Palette eXtended Kubernetes pack of your cluster profile. | May 3, 2026 | Packs | 4.9.22 |
MAAS clusters deployed or repaved on Palette versions 4.9.14 or 4.9.16 lose the default ubuntu user. The SSH key injection feature introduced in 4.9.14 removed the default user due to a known issue in cloud-init, preventing SSH access to nodes using the default user. To regain SSH access, users should configure SSH keys for the cluster through Palette, creating the spectro user on all nodes. Nodes already affected will not recover the default user automatically after upgrading to Palette version 4.9.18 and must be repaved to restore the default user. Self-hosted Palette and dedicated SaaS customers must skip versions 4.9.14 and 4.9.16 to avoid being impacted by this issue. | June 11, 2026 | Clusters | 4.9.18 |
| Azure IaaS clusters using Palette eXtended Kubernetes (PXK) version 1.32.13 or earlier get stuck when upgrading to a PXK version in the 1.33.x series. | May 3, 2026 | Clusters, Packs | 4.9.6 |
When upgrading a self-hosted Palette instance from 4.8.35 to 4.8.37, MongoDB replica pods may crash with a CrashLoopBackOff error. This may be due to MongoDB replication lag or a cluster split-brain state, resulting in a stuck upgrade. Refer to Scenario - MongoDB Replica Pods Crash during Palette Upgrade for the workaround. | March 12, 2026 | Self-Hosted Palette | 4.9.5 |
| Palette API calls fail with Error 500 when response bodies exceed the configured size. Refer to Scenario - Errors in API Calls After Traefik Migration for the workaround. | April 5, 2026 | Clusters, Self-Hosted | 4.8.52 |
| Edge hosts configured with K3s, Traefik and kube-vip may experience unexpected IP takeover behavior and loss of etcd quorum due to a kube-vip issue. Ensure that the following configurations are provided in the K3s pack values: - traefik and servicelb remain listed in the cluster.config.disable field. - cluster.kubevipArgs.svc_enable remains set to false. These configurations are default pack values. | April 5, 2026 | Edge, Packs | 4.8.51 |
EKS clusters deployed with the AWS Cluster Autoscaler pack fail to upgrade the Kubernetes version of their worker node groups or MachinePools. Refer to Scenario - EKS Worker Nodes Configured with Autoscaler Fail to Upgrade for the workaround. | February 21, 2026 | Packs, Clusters | 4.8.37 |
When global DNS values are configured through user data, Local UI displays the Edge host's local stub resolver address from /etc/resolv.conf, 127.0.0.53, instead of the upstream DNS server addresses. DNS resolution is not affected. | March 5, 2026 | Edge | 4.8.37 |
| Virtual Machine Orchestrator Day-2 operations, such as VM start, stop, restart, and migrate, do not complete when triggered using the Palette API on environments using 4.8.33. | February 24, 2026 | Virtual Machine Orchestrator | 4.8.35 |
EKS clusters configured with mismatched Classless Inter-Domain Routing (CIDR) values and NodePool customization using AWS Launch Templates have nodes with incorrect kubelet CIDR and clusterDNS values. Create a new node and restart the pods scheduled on affected nodes so that they are rescheduled onto a healthy node. | January 30, 2026 | Clusters | 4.8.25 |
| Edge hosts using encrypted persistent partitions may fail to complete a reset. A limitation in Kairos version 3.5.3 prevents these partitions from being reinitialized during the reset workflow, resulting in the operation stopping prematurely. This was resolved with Kairos version 3.5.9. Refer to Troubleshooting - Edge for the workaround for Kairos version 3.5.3. | December 17, 2025 | Edge | 4.8.21 |
On Edge Native clusters, restores of security-hardened applications (such as Argo CD) may fail with the affected application pods stuck in the Init:CreateContainerConfigError state, showing Error: container has runAsNonRoot and image has non-numeric user (cnb), cannot verify user is non-root on the restore-wait init container. Refer to Troubleshooting - Edge for the workaround. | November 14, 2025 | Edge, Packs | 4.8.21 |
Pods with emptyDir volumes that are backed up using Velero 1.9, restored using Velero 1.15, and backed up and restored again with Velero 1.15 are stuck in the init state when performing a second restore. This was caused by a known upstream issue and has been resolved with Velero 1.17. For older Palette versions affected by the upstream issue, delete stuck pods or restart affected deployments. | May 31, 2025 | Clusters | 4.8.21 |
On Edge clusters whose hosts run Ubuntu 24.04 with a Unified Kernel Image (UKI), CoreDNS pods may enter the CrashLoopBackOff state with logs showing [FATAL] plugin/loop: Loop (127.0.0.1:<ephemeral-port> -> :53) detected for zone ".". This happens because /etc/resolv.conf is symlinked to /run/systemd/resolve/stub-resolv.conf, which lacks real DNS server entries. As a result, CoreDNS forwards DNS queries to itself, creating a recursive loop. Refer to Troubleshooting - Edge for the workaround. | October 7, 2025 | Edge | 4.8.21 |
| If you configure static IP on a host using the TUI, the cluster that is formed by the host cannot enable network overlay. | July 31, 2025 | Edge | 4.8.21 |
Agent mode Edge cluster nodes configured with Agent version 4.8.1 may lose pre-existing systemd-networkd configurations (VLANs, bonds, and bridges) after a reboot. This occurs because the cleanup logic intended to remove stale VLAN configurations inadvertently deletes valid, pre-existing configurations not created through user data. | December 5, 2025 | Edge | 4.8.12 |
| Users cannot remove proxy values for connected Edge hosts in Local UI due to a validation error. Proxy values can still be added and updated. | October 19, 2025 | Edge | 4.8.6 |
| Users cannot create Ubuntu 20.04 FIPS ISO images using CanvOS version 4.7.16. | November 14, 2025 | Edge | 4.7.30 |
| Cluster deployments may fail after upgrading the Cluster API Provider AWS (CAPA) if the Instance Metadata Service Version 2 (IMDSv2) is configured with Metadata version set to V2 only (token required) in the Amazon Elastic Compute Cloud (EC2) settings. | September 1, 2025 | Clusters | 4.7.21 |
Cluster profile variables are incorrectly applied to the charts.virtual-machine-orchestrator.appConfig.clusterInfo.consoleBaseAddress field used to provide direct access to Virtual Machine Dashboard. | September 20, 2025 | Virtual Machine Orchestrator | 4.7.20 |
Network overlay cluster nodes may display erroneous failed to add static FDB entry after cleanup...Stdout already set, output logs after upgrading the Palette agent to version 4.7.9. Cluster functionality is not affected. | September 20, 2025 | Edge | 4.7.20 |
| On Azure IaaS clusters created using a Palette version prior to 4.6.32, scaling worker node pools does not attach newly created nodes to an outbound load balancer after upgrading to Palette version 4.6.32 or later and the cluster's Palette Agent version to 4.6.7 or later. This impacts outbound connectivity and may also disassociate existing NAT gateways from the worker node pool subnet, resulting in a loss of egress connectivity. Upgrade to Palette version 4.7.20 or later before attempting to perform Day-2 scaling or repave operations. | September 1, 2025 | Clusters, Self-Hosted | 4.7.20 |
| HTTP-Proxies configured in Local UI prior to cluster creation are not applied correctly. | August 17, 2025 | Edge | 4.7.15 |
After deleting an Edge cluster via the Palette UI, users cannot establish an SSH connection with their user-data credentials to the Edge host on which the cluster was previously deployed. This affects Edge hosts using Palette agent versions 4.6.21 - 4.7.9. To restore SSH access, connect to the Edge host via remote shell with a temporary username and password and add your user's password to the user-data file. | August 8, 2025 | Edge, Clusters | 4.7.15 |
| Virtual Machine Orchestrator (VMO) 4.7.1 cannot be uninstalled due to a missing image. | September 1, 2025 | Virtual Machine Orchestrator | 4.7.13 |
| Agent mode clusters may initiate cluster repaves intermittently. | July 21, 2025 | Edge | 4.7.13 |
For locally deployed clusters, adding a custom stylus.path to the user-data file causes cluster creation to fail, as it prevents the Palette agent from locating the user-data file. | July 21, 2025 | Edge | 4.7.13 |
| Masked cluster profile variable values are displayed as plain text in Edge Management API responses. | July 24, 2025 | Clusters | 4.6.44 |
When deploying a workload cluster with packs using namespaceLabels, the associated Pods get stuck if the cluster is deployed via self-hosted Palette or Palette VerteX, or if the palette-agent ConfigMap specifies data.feature.workloads: disable. Force-apply privileged labels to the affected namespace. Refer to the Packs - Troubleshooting guide for additional information. | July 19, 2025 | Clusters | 4.7.13 |
| When rebooting nodes in a Canonical Edge cluster deployed in a proxied environment, the nodes may fail to come back online. | May 31, 2025 | Edge | 4.7.3 |
| Due to CAPZ upgrades in version 4.6.32, Azure US Government clusters (both IaaS and AKS) cannot be provisioned, and Day-2 operations cannot be performed on existing Azure US Government clusters. | May 31, 2025 | Clusters | 4.6.36 |
| Newly created AWS EKS clusters that have configured node pool customizations fail to provision. | May 31, 2025 | Clusters | 4.6.33 |
| Configuring node pool customizations on existing AWS EKS clusters causes errors. | May 31, 2025 | Clusters | 4.6.33 |
| Palette eXtended Kubernetes (PXK) versions 1.31.x and 1.32.x do not support GCP. Clusters with this configuration fail to deploy. | May 31, 2025 | Clusters | 4.6.32 |
| Palette eXtended Kubernetes (PXK) versions 1.31.x and 1.32.x do not support AWS. Clusters with this configuration fail to deploy. | May 31, 2025 | Clusters | 4.6.32 |
When upgrading an airgapped Edge cluster to version 4.6.24, some pods may get stuck in the ImagePullBackOff state. | May 5, 2025 | Edge | 4.6.32 |
| Deletion of GCP clusters fails for clusters associated with cloud with cloud accounts that have been created on older Palette versions. | May 5, 2025 | Clusters | 4.6.25 |
| Software Bill of Materials (SBOM) scans are failing to execute successfully for all clusters. | May 5, 2025 | Clusters | 4.6.24 |
Registry Mapping Rules provided with stylus.externalRegistries.registryMappingRules are not respected for connected edge clusters. | April 19, 2025 | Edge | 4.6.23 |
For clusters deployed with agent mode on Palette agent version 4.5.14, having a custom stylus.path defined in the user-data file causes Kubernetes upgrades to fail. This affects all Kubernetes packs. | March 15, 2025 | Edge | 4.6.12 |
| If an Edge host operating a cluster in connected mode loses connection to Palette, the cluster will not auto-renew its Public Key Infrastructure (PKI) certificates. When it re-establishes the connection to Palette, the Edge host will renew the certificates if the existing certificates have less than 30 days before they expire. | March 15, 2025 | Edge | 4.6.12 |
| Cluster Removal of Palette and VerteX clusters will get stuck if you attempt to delete a cluster with expired cloud account credentials. This may lead to cloud resources not being correctly removed and incurring unexpected cloud costs. | March 15, 2025 | Clusters | 4.6.12 |
| Palette eXtended Kubernetes - Edge (PXK-E) version 1.31 is not yet supported for Edge clusters. | February 24, 2025 | Edge | 4.6.12 |
| K3s certificate renewals for Two-node Edge clusters are not managed by Palette. Two-node clusters will use a default certificate list. | February 17, 2025 | Edge | 4.6.8 |
| Clusters that are currently deployed and attempting to upgrade from Kubernetes version 1.31.1 to newer versions are getting stuck in the upgrade state. This issue also affects Day-2 operations related to changes in the Kubernetes layer YAML and the resize of control plane nodes for clusters using Kubernetes version 1.31.1. The root of the problem is an internal conversion error stemming from an internal type change introduced in Kubernetes version 1.31.x. | January 19, 2025 | Clusters, Packs | 4.6.6 |
Clusters using Microk8s and conducting backup and restore operations using Velero with restic are encountering restic pods going into the crashloopbackoff state. This issue stems from an upstream problem in the Velero project. You can learn more about it in the GitHub issue 4035. Refer to the MicroK8s Additional Details section for troubleshooting workaround steps. | October 12, 2024 | Clusters | 4.6.6 |
| Clusters configured with OpenID Connect (OIDC) at the Kubernetes layer encounter issues when authenticating with the non-admin Kubeconfig file. Kubeconfig files using OIDC to authenticate will not work if the SSL certificate is set at the OIDC provider level. | June 21, 2024 | Clusters | 4.6.6 |
| Deployed clusters using a Palette version before 4.4.20 will not have accurate Kubernetes certificate expiration information displayed on the Palette UI certificate page. Starting with Palette version 4.4.20, the certificate information displayed on the certificate management page will be correct. | October 31, 2024 | Clusters | 4.4.20 |
A change in the Edge Cluster API endpoint affects Terraform and API workflows for Edge cluster creation or modification. The field spec.cloudConfig.controlPlaneEndpoint.type no longer accepts IP addresses. The accepted values are now VIP, External, and DDNS. As a result, API and Terraform workflows for Edge cluster creation or modification that use the IP address are currently unavailable. | September 18, 2024 | Edge | 4.4.19 |
| In a VMware environment, self-hosted Palette instances do not receive a unique cluster ID when deployed, which can cause issues during a node repave event, such as a Kubernetes version upgrade. Specifically, Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) will experience start problems due to the lack of a unique cluster ID. | April 14, 2024 | Self-Hosted | 4.4.14 |
| The Palette agent fails to pull Helm charts from private registries when initializing Edge clusters. | July 25, 2024 | Edge | 4.4.12 |
| The VerteX enterprise cluster is unable to complete backup operations. | June 6, 2024 | VerteX | 4.4.11 |
| An issue prevents RKE2 and Palette eXtended Kubernetes (PXK) on version 1.29.4 from operating correctly with Canonical MAAS. | July 20, 2024 | Packs, Clusters | 4.4.11 |
| Deploying self-hosted Palette or VerteX to a vSphere environment fails if vCenter has standalone hosts directly under a data center. Persistent Volume (PV) provisioning fails due to an upstream issue with the vSphere Container Storage Interface (CSI) for all versions before v3.2.0. Palette and VerteX use the vSphere CSI version 3.1.2 internally. The issue may also occur in workload clusters deployed on vSphere using the same vSphere CSI for storage volume provisioning. | July 20, 2024 | Self-Hosted | 4.4.11 |
When you upgrade VerteX from version 4.3.x to 4.4.x, a few system pods may remain unhealthy and experience CrashLoopBackOff errors. This issue only impacts VMware vSphere-based installations and occurs because the internal Mongo DNS is incorrectly configured in the configserver ConfigMap. | June 29, 2024 | Self-Hosted, VerteX | 4.4.7 |
| Clusters are failing to start on AWS EKS when using the AWS VPC CNI pack and Kubernetes versions 1.28. | February 26, 2024 | Packs, Clusters | 4.2.13 |
| The Kubernetes Dashboard fails to load when added to a cluster profile and deployed. | February 26, 2024 | Packs | 4.2.13 |
| Clusters using MicroK8s are failing to launch pods due to a mismatch in node affinity labels. | February 26, 2024 | Packs, Clusters | 4.2.13 |
| MAAS clusters are failing to deploy when the default image endpoint is not set in an airgap environment. | February 26, 2024 | Clusters | 4.2.13 |