Skip to main content
Version: latest

Release Notes

Feb 16, 2024 - Release 4.2.12

Bug Fix: IaaS Cluster Repaves Causing Cluster Downtime

Affected services

IaaS clusters in Palette 4.2.x prior to 4.2.12, including Palette SaaS, self-hosted Palette/VerteX, as well as dedicated instances. Affected cluster types include the following:

  • AWS IaaS (not EKS)
  • Azure IaaS (not AKS)
  • Google IaaS (not GKE)
  • MAAS
  • vSphere
  • OpenStack

Managed Kubernetes clusters on EKS, GKE and AKS are not affected. Edge clusters are not affected.

Issue Summary

We identified an issue related to cluster repaves in Palette 4.2.x. During a cluster upgrade that required a repave, the Palette Agent deployed within the clusters would delete all the worker nodes within a worker pool before provisioning new worker nodes. This results in the worker pool being down during an upgrade. All workloads within the pool will be offline during the upgrade.

If the cluster is configured to enable updating worker pools in parallel, then this can result in all services on the cluster becoming unavailable.

Customer Guidance

This issue has been addressed in Palette 4.2.12 and its corresponding Palette Agent version 4.2.4. Use the following steps to identify whether your cluster uses an affected agent version.

  1. Log in to Palette.
  2. From the left Main Menu, click on Clusters. Select your cluster to access the cluster details page.
  3. At the bottom of the cluster details page, the Palette agent version used by your cluster is displayed. If your Agent version is any of the following versions, your cluster is still susceptible to this issue: 4.2.0, 4.2.1, 4.2.2, 4.2.3.
warning

Ensure that you do not initiate any cluster repaves as long as you are using an affected agent version. Changes in the OS or the Kubernetes layer would initiate an cluster repave attempt. When you get the cluster repave notification, reject the repave.

If you are not using an affected agent version, no action is required on your part. If you plan to upgrade to 4.2.x in the future, ensure you upgrade to a version of Palette that's 4.2.12 or later.

If you are using an affected agent version, first make sure that your Palette instance version is 4.2.12 or newer. Once you have confirmed your Palette version, unpause Agent upgrades for your cluster if they are paused. To learn how to toggle agent upgrades, refer to Pause Platform Upgrades. In 5 - 10 minutes, the Palette agent will upgrade to a new version that includes the bug fix. If the agent does not upgrade for an extended period of time, contact support@spectrocloud.com.

February 3, 2024 - Release 4.2.9

Bug Fixes

  • Fixed an issue that caused errors when creating pods after certificate renewals.
  • Resolved image pull errors from the AWS ECR registry.

January 25, 2024 - Release 4.2.7

Bug Fixes

  • Fixed an issue that caused MinIO S3 URL setting to be missing in backup location settings.
  • Fixed an issue that prohibited updating Helm packs in cluster profiles.
  • Fixed an issue that caused certain OCI registries created before the Palette 4.2 upgrade to be unlisted.
  • Fixed an issue that caused HTTP 400 errors when visiting the Kubernetes Dashboard. The issue was caused by an internal cookie size limit that was insufficient for the Kubernetes Dashboard.

January 9, 2024 - Release 4.2.4

Bug Fixes

  • An invalid toggle User Interface option that appeared in the Edge cluster creation process when defining node groups has been removed.

January 6, 2024 - Release 4.2.0

Palette 4.2.0 is a release that includes new features and various improvements. New features include support for Nutanix clusters, automatic SSL certificate renewal, and enhanced cluster repave control and mitigation. Improvements include support for MicroK8S on MAAS clusters, several network enhancements for Edge deployments, a new differential editor that helps you identify cluster profile changes, and support for a local image registry for Edge clusters. Check out the notes below to learn more about the new features and improvements.

Palette

Features

  • Palette now supports the cloud provider, Nutanix, as a Technical Preview feature. You can deploy Kubernetes clusters on Nutanix using Palette. Technical Preview features are subject to change as we continue to improve the integration. Refer to the Nutanix resource to learn more about deploying Nutanix clusters with Palette.

  • Automatic SSL certificate renewal is now supported for clusters deployed through Palette. In the past, this was a manual action that had to be performed by the user, which also caused node repaves. Palette will now automatically renew the certificate 30 days before the expiration date without triggering a node repave. This feature is available in all supported infrastructure providers except for Edge. For more information, refer to the Certificate Management resource.

  • Enhanced cluster repave control and mitigation. In the Palette 4.1 release, repave notification warnings become available through the User Console (UI). In this release, cluster administrators, project administrators, and tenant administrators must acknowledge the repave notification and decide whether to proceed with the action. This feature helps prevent accidental node upgrades that may cause downtime and provides a way to mitigate repaves by allowing administrators to cancel the action that will trigger a repave.

  • A Pack's README file is displayed during the cluster profile creation and editing process. You can find additional information about a pack in the Packs List page.

  • Palette CLI now supports integration with Validator, an open-source framework that you can use to validate your self-hosted Palette, VerteX, or workload cluster environment. Validator performs Day 0-2 validation and configuration drift detection in a composable manner across various systems. Use the palette validator command to verify your environment before installing a self-hosted instance of Palette or VerteX. You can also use Validator to verify the environment requirements for deploying a cluster. For more information, refer to the Validator CLI reference.

  • Support for passkeys is now available for the self-hosted Palette admin user. When accessing the system console, you can now use passkeys to authenticate to the admin user account. For more information, refer to the System Console Credentials resource.

  • You can start a local Palette documentation server by using the Palette CLI's docs command. This feature is useful when you want to access Palette documentation offline. For more information, refer to the Docs command page.

Improvements

  • MicroK8S is now available for MAAS clusters. Create a cluster profile with MicroK8S as the Kubernetes pack to deploy a MAAS cluster with MicroK8S.

  • An improved differential editor is now available. The new editor provides a side-by-side comparison of the changes that will be applied to the cluster profile. The editor also identifies the YAML customizations you have added and guides you through carrying over the customizations to the new version of the YAML. The ability to undo changes and accept all changes is also available.

  • When updating a deployed cluster profile or an active cluster's profile, the new differential editor is available to help you identify the changes that will be applied to the cluster profile.

  • Private Cloud Gateway (PCG) deployments now use Kubernetes version 1.26. Previously, the default Kubernetes version was 1.24. Use the latest version of the Palette CLI to install PCG clusters. Existing Private Cloud Gateway deployments will require a manual reconciliation of the cluster profile to update the Kubernetes version to 1.26. Make sure you carry over any customizations the current cluster profile may have, such as pod CIDR and service CIDR before updating the cluster profile with the new Kubernetes version. Refer to the Update a Cluster Profile guide to learn more on reconciling a cluster profile pack layer change.

Known Issues

  • The ability to change the underlying node type of a node pool is not available for Google Cloud Platform GKE clusters.

  • Clusters launched in VMware vSphere with the Container Network Interface (CNI) Cilium, lose node-to-node connectivity when the vSphere adapter is configured to use VMXNET3. This is a known issue with Cilium and VMXNET3. Refer to the GitHub issue discussion to learn more about this issue.

  • Enabling passkeys in a self-hosted Palette instance will cause JSON Web Tokens (JWT) returned by the system API endpoint /v1/auth/syslogin to be invalid. Refer to the Passkeys and API Access resource for more information on accessing the system API when passkeys are enabled. This issue does not affect the regular Palette API used by clusters and users.

Edge

Breaking Changes

  • Edge hostnames are not allowed to have special characters. Validation has been added to prevent issues arising from using special characters in host names. Edge hostnames must comply with RFC1035, refer to the Edge Installer Configuration and review the name parameter for more information.

Features

  • Overlay support for DHCP is now available as a Tech Preview feature. Edge clusters can now establish an VxLAN overlay network during cluster creation, and Edge hosts can self-discover the overlay network within a single ethernet broadcast domain. Clusters using this feature will remain operational when the host IP addresses change unexpectedly. Check out the Enable Overlay Network resource for more information.

  • Local registry support is now available as a Tech Preview feature. You can deploy a self-hosted Harbor registry on your Edge cluster and use the registry to store images for your workloads and initialize a cluster's other edge host nodes. Using a local registry can help you reduce the amount of data transferred over the network, cache images locally, and provide a backup for when internet access is unavailable.

  • Edge Kubernetes network interface management support. You can now specify the network interface for your edge hosts versus relying on the default interface selected by Kubernetes. This feature is useful when you have multiple network interfaces on your edge hosts and want to use a specific interface for your workloads or if you are using the new overlay support for DHCP. Check out the Create Cluster Definition resource for more information on how to specify the network interface for your edge hosts during cluster deployment.

Improvements

  • New Edge clusters can now retrieve provider images from authenticated registries. Previously, only public registries were supported for non-airgapped clusters. Now, you can use authenticated registries to store your provider images and retrieve them during cluster deployment. For more information, refer to the Deploy Cluster with a Private Registry guide.

  • Extended kube-vip customization is now available for new Edge clusters. You can now specify additional kube-vip configuration parameters as part of the Kubernetes pack layer configuration. To learn more about the available kube-vip configuration parameters, refer to the Publish Cluster Services with Kube-vip resource.

Known Issues

  • The following known issues apply to the VxLAN network overlay feature:

    • When adding multiple nodes to an existing cluster with overlay enabled, failure to add one node will block the addition of the other nodes.

    • When deleting an Edge host from a cluster with overlay enabled, ensure the node doesn't have the palette-webhook pod on it, or the node will be stuck in the deleting state. You can use the command kubectl get pods --all-namespaces --output wide to identify which node the pod palette-webhook is on. Reach out to our support team support@spectrocloud.com if you need to remove a node with the palette-webhook pod on it.

  • In a multi-node cluster with PXK-E as the Kubernetes distribution, you cannot change the Network Interface Card (NIC). When you add an Edge host to such a cluster, leave the NIC field as its default value.

  • The following known issues apply to Harbor Edge Native Config when deployed with the Longhorn Container Storage Interface (CSI) driver:

    • The Harbor job service pod is in a Terminating and ContainerCreating state in an Edge Native High Availability (HA) cluster after a Day-2 operation.

    • The Harbor database pod might fail to start due to file permission issues. This is a known issue in the Harbor GitHub repository.

    • A cluster may get stuck in the provisioning state. If this happens, remove the cluster and try again.

Palette Dev Engine (PDE)

Improvements

  • The default deployed Kubernetes version for new virtual clusters is now v1.26.

Virtual Machine Orchestrator (VMO)

Features

  • You can now deploy virtual machines using VMO on an Edge cluster. Edge clusters are useful when deploying Kubernetes clusters in remote locations. Refer to the Create a VMO Profile to learn how to create an Edge cluster profile for VMO.

VerteX

Features

  • Azure Government Cloud support is now available for VerteX. You can now deploy Azure IaaS clusters on Azure Government accounts. The following Azure regions are available: US Gov Arizona, US Gov Texas, and US Gov Virginia. For more information, refer to the Supported Platforms resource.

  • Canonical MAAS support is now available for VerteX. You can now deploy Canonical MAAS clusters with VerteX. Refer to the MAAS resource for more information on deploying MAAS clusters.

  • Support for passkeys is now available for the admin user. When accessing the system console, you can now use passkeys to authenticate the admin user account. For more information, refer to the System Console Credentials resource.

Improvements

  • To better support airgap installs and customers in internet-restricted environments. You can now access Palette documentation offline by using the Palette documentation container. For more information, refer to the Offline Documentation page.

Known Issues

  • Enabling passkeys in a VerteX instance will cause JSON Web Tokens (JWT) returned by the system API endpoint /v1/auth/syslogin to be invalid. Refer to the Passkeys and API Access resource for more information on accessing the system API when passkeys are enabled. This issue does not affect the regular VerteX API used by clusters and users.

Terraform

Breaking Changes

  • The parameter cluster_context is now a required attribute for the resource spectrocloud_application.

  • The resource spectrocloud_cluster_edge_native is deprecating the following arguments; ssh_key, and host_uids.

Features

Docs and Education

  • The Deploy a Custom Pack tutorial has been updated to include instructions on deploying a custom pack with a custom OCI Pack registry.

  • The Palette Offline Documentation container image is now cryptographically signed. You can verify the authenticity of the container image by using the Cosign CLI and the public key. Refer to the Offline Documentation page for more information.

Packs

Kubernetes

PackNew Version
Kubernetes Azure AKS1.28.2
Kubernetes Amazon EKS1.28.2
Kubernetes Cox Edge1.28.2
Kubernetes Cox Edge1.27.6
Kubernetes Cox Edge1.26.9
Kubernetes Cox Edge1.25.14
Kubernetes Google GKE1.27.6
Kubernetes Google GKE1.26.9
Kubernetes Google GKE1.25.14
K3s1.28.2
K3s1.27.7
K3s1.26.10
K3s1.25.15
Palette eXtended Kubernetes - Edge1.28.2
Palette eXtended Kubernetes - Edge1.27.7
Palette eXtended Kubernetes - Edge1.26.10
Palette eXtended Kubernetes - Edge1.25.15
Palette eXtended Kubernetes1.28.3
Palette eXtended Kubernetes1.27.7
Palette eXtended Kubernetes1.26.10
Palette eXtended Kubernetes1.25.15
RKE21.28.2
RKE21.27.8
RKE21.26.11
RKE2 - Edge1.28.4
RKE2 - Edge1.27.7
RKE2 - Edge1.26.10
RKE2 - Edge1.25.15

CNI

PackNew Version
AWS VPC CNI1.15.1
Calico CNI3.26.3
Ciliium OSS1.14.3
Flannel CNI0.23.0

CSI

PackNew Version
Azure Disk CSI Driver1.29.1
AWS EBS CSI1.24.0
Longhorn CSI1.5.3
Nutanix CSI2.6.6
Portworx CSI3.0.4
Rook Ceph CSI1.12.7

Add-on Packs

PackNew Version
External Secrets Operator0.9.7
Flux22.10.2
Harbor Edge Native Config1.0.0
Istio1.17.2
Kong Ingress2.32.0
MetalLB0.13.11
Nginx Ingress1.9.4
Nvidia GPU Operator23.9.1
Open Policy Agent3.13.2
Prometheus Operator51.0.3
Reloader1.0.43
Imageswap1.5.3

FIPS Packs

PackNew Version
Azure CSI Driver1.28.3
Palette eXtended Kubernetes1.28.3
Palette eXtended Kubernetes1.27.7
Palette eXtended Kubernetes1.26.10
Palette eXtended Kubernetes1.25.15
Palette eXtended Kubernetes - Edge1.27.2
Palette eXtended Kubernetes - Edge1.26.4
Palette eXtended Kubernetes - Edge1.25.9
RKE21.28.6
RKE21.27.8
RKE21.26.11
RKE2 - Edge1.27.2
RKE2 - Edge1.26.4
RKE2 - Edge1.25.2

Pack Notes

  • A Pack's README file is displayed during the cluster profile creation and editing process. You can find additional information about a pack in the Packs List page.
  • ArgoCD is now a verified pack, starting with version 5.46.8.
  • Spot.io is now a verified pack, starting with version 1.0.117.
  • Edge cluster nodes deployed in a single node configuration using RKE2 version 1.26.X must upgrade to the latest minor version of 1.26.10 before upgrading to 1.27.7.
  • The prior issue related to Edge clusters deployed in a single node configuration using RKE2 version 1.26.X has been resolved starting with version 1.26.10.

Deprecations and Removals