Node Pools

A node pool is a group of nodes within a cluster that all have the same configuration. You can use node pools for different workloads. For example, you can create a node pool for your production workloads and another for your development workloads. You can update node pools for active clusters or create a new one for the cluster.

warning

Exercise caution when modifying node pools. We recommend creating a backup before you make a change in the event a configuration change causes an issue.

Repave Behavior and Configuration

In Kubernetes, the term "repave" refers to the process of replacing a node with a new node. Repaving is a common practice in Kubernetes to ensure that nodes are deployed with the latest version of the operating system and Kubernetes. Repaving is also used to replace nodes that are unhealthy or have failed. You can configure the repave time interval for a node pool.

Different types of repaving operations may occur, depending on what causes them:

info

This section only applies to non-Edge clusters. To learn about the upgrade behavior of Edge clusters, refer to the Edge Cluster Upgrade Behavior page.

• Control plane repave: This takes place when certain changes are made to the Kubernetes configuration, such as changing the apiServer specification. This type of repave also occurs when there are changes in the hardware specifications of the control plane nodes, such as during a node scale-up operation or when changing from one instance type to another. Control plane nodes are repaved sequentially.

• Worker node pool repave: This happens when changes to a node pool's specifications cause the the existing nodes to become incompatible with the pool's specified criteria. For instance, changing the hardware specifications of a worker pool. Nodes within the affected pool are sequentially replaced with new nodes that meet the updated specifications.

• Full cluster repave: This occurs if changes are made to the Operating System (OS) layer or if modifications to the Kubernetes layer impact all nodes, such as when upgrading to a different Kubernetes version. All nodes across all pools are sequentially repaved starting with the control plane.

You can customize the repave time interval for all node pools except the control plane node pool. The default repave time interval is zero seconds, meaning the old worker node is considered for deletion as soon as a new worker node is initialized. Because the new worker node can take some time to become fully operational, you can extend the repave time interval to preserve the old worker node until the new one is ready. If you need to modify the repave time interval post-cluster creation, follow the Change a Node Pool guide.

info

To ensure that repave operations do not interfere with your existing workloads, Palette requires explicit cluster-level approval to initiate them. Refer to the Approve Cluster Repave guide to learn how to approve cluster repaves.

Pod Drainage Toleration

Palette will not remove pods with the toleration key node.kubernetes.io/unschedulable and the effect NoSchedule during node draining operations. You can add this toleration to your pods to prevent them from being removed during a repave operation.

tolerations:
  - key: node.kubernetes.io/unschedulable
    effect: NoSchedule
    operator: Exists

Adding the node.kubernetes.io/unschedulable toleration to your pods can be useful in scenarios where you have pods that behave as DeamonSets and should not be removed during a repave operation. You can use this toleration to ensure that these pods are not removed during a repave operation.

You can also specify other tolerations during the cluster creation process. For more information on adding tolerations, refer to the Taints and Tolerations guide.

Dedicated Node Pool for System Pods

For EKS and GKE clusters, you can dedicate a node pool for system pods, both Palette and non-Palette. This can be helpful if you want to schedule system pods on a single node pool while reserving other pools for non-system workloads.

Palette ensures that non-system workloads are not scheduled on the dedicated node pool by applying a system taint to the pool, thus repelling other workloads. Palette system pods automatically tolerate this taint and can therefore be scheduled on the dedicated node pool. If you want a non-Palette pod to be scheduled on the dedicated node pool, you must add a toleration to the pod that matches the pool's node.spectrocloud.com/dedicated system taint.

Field	Value
key	node.spectrocloud.com/dedicated
operator	Equal
value	true
effect	NoExecute

Dedicating a node pool for system pods is not enough to guarantee that system pods are not scheduled on other node pools; it only ensures that non-system pods are not scheduled on the dedicated node pool. To prevent system pods from being scheduled on node pools that you want to reserve for other workloads, you must apply a custom taint to those node pools. You must then add the matching toleration to your non-system pods so they can be scheduled on the node pools with the custom taint. Because the system pods do not have the toleration that matches the custom taint, they are only scheduled on the dedicated node pool.

To dedicate a node pool for system pods, select the Dedicate node pool for system pods option when configuring the pool.

Node Pool Configuration Settings

The following tables contain the configuration settings for node pools. Depending on the type of node pool, some of the settings may not be available.

Control Plane Node Pool

Property	Description
Node pool name	A descriptive name for the node pool.
Number of nodes in the pool	Number of nodes to be provisioned for the node pool. This number can be 1, 3, or 5. For Edge control plane pools, you can temporarily run four nodes during a control plane host replacement.
Allow worker capability	Select this option to allow workloads to be provisioned on control plane nodes.
Additional Labels	Optional labels apply placement constraints on a pod. For example, you can add a label to make a node eligible to receive the workload. To learn more, refer to the Node Labels. For Edge Native clusters, you can also apply labels to a specific node in the pool. Refer to Node-level Taints and Labels for Edge Native.
Additional Annotations (Optional)	Additional Kubernetes annotations to assign to each control plane node.
Override Machine Health Check	Override the default Cluster API Machine Health Check (MHC) settings that Palette applies to this node pool. This option is disabled by default and is available only on Palette eXtended Kubernetes (PXK) infrastructure clusters that use Cluster API. It does not apply to EKS, AKS, or GKE clusters. When enabled, the Configure Machine Health Check button appears. When this option is disabled, the node pool uses the default Palette MHC settings.
Configure Machine Health Check	Available only when Override Machine Health Check is enabled. Select this option to define the maxUnhealthy, nodeStartupTimeout, and unhealthyNodeConditions values for the node pool. Palette honors per-pool overrides only when Cluster Auto Remediation is enabled at the tenant or project level. If the MHC configuration is invalid, Palette records an error in the cluster event stream, ignores the override, and applies the default MHC. Any changes made after cluster deployment trigger a cluster repave.
Taints	Applies taints to the node pool so that pods are scheduled to the pool only if they tolerate the taint. To learn more, refer to the Taints and Tolerations guide. For Edge Native clusters, you can also apply taints to a specific node in the pool. Refer to Node-level Taints and Labels for Edge Native.
Dedicate node pool for system pods	Reserves the node pool for Palette system pods. Available only on EKS and GKE clusters. When enabled, custom taints cannot be added to this pool. To learn more, refer to Dedicated node pool for system pods.
Availability Zones	The Availability Zones from which to select available servers for deployment. If you select multiple zones, Palette will deploy servers evenly across them as long as sufficient servers are available to do so.
Disk Size	Give the required storage size.

Worker Node Pool

Property	Description
Node pool name	A descriptive name for the worker pool.
Enable Autoscaler	Scale the worker pool horizontally based on its per-node workload counts. The Minimum size specifies the lower bound of nodes in the pool, and the Maximum size specifies the upper bound. Setting both parameters to the same value results in a static node count. Public cloud (AWS, Azure, and GCP) IaaS clusters and private data center (VMware vSphere, Apache CloudStack, and MAAS) clusters use the Cluster API autoscaler tool; Azure AKS uses the Azure autoscaler tool, which is based on the Cluster API autoscaler. NOTE: If autoscaling is enabled when deploying VMware vSphere clusters using an IP Address Management (IPAM) node pool with static placement configured, the Maximum size determines the number of IP addresses automatically reserved for worker nodes. During day-2 operations, even if autoscaler is disabled or the Maximum size of the worker pool is reduced, the original number of IP addresses remains allocated. To release the IP addresses, you must delete the worker node pool.
Number of nodes in the pool	Number of nodes to be provisioned for the node pool. This field is hidden if Enable Autoscaler is toggled on.
Node repave interval	The time interval in seconds between repaves. The default value is 0 seconds.
Skip worker node update (Optional)	Enable this option to skip updating worker nodes during a cluster upgrade and update the control plane only. The Kubernetes version difference between the control plane and worker nodes must not exceed the N-3 minor version skew supported by Kubernetes. For more information, including supported cluster types, behavior during profile updates, and how to trigger the deferred worker upgrade, refer to Skip Worker Node Update.
Additional Labels	Optional labels apply placement constraints on a pod. For example, you can add a label to make a node eligible to receive the workload. To learn more, refer to the Node Labels guide. For Edge Native clusters, you can also apply labels to a specific node in the pool. Refer to Node-level Taints and Labels for Edge Native.
Additional Annotations (Optional)	Additional Kubernetes annotations to assign to each worker node.
Override Kubeadm Configuration	Adjust kubelet arguments for kubeadm or pre-kubeadm commands to meet specific operational or environment requirements for your worker nodes. This option is disabled by default. When enabled, the Configure Kubeadm button appears.
Configure Kubeadm	Available only when Override Kubeadm Configuration is enabled. Select this option to override kubeadmconfig.kubeletExtraArgs and kubeadmconfig.preKubeadmConfig commands configured in the Kubernetes layer of your cluster profile. Any changes made post-cluster deployment will trigger a cluster repave.
Override Machine Health Check	Override the default Cluster API Machine Health Check (MHC) settings that Palette applies to this node pool. This option is disabled by default and is available only on Palette eXtended Kubernetes (PXK) infrastructure clusters that use Cluster API. It does not apply to EKS, AKS, or GKE clusters. When enabled, the Configure Machine Health Check button appears. When this option is disabled, the node pool uses the default Palette MHC settings.
Configure Machine Health Check	Available only when Override Machine Health Check is enabled. Select this option to define the maxUnhealthy, nodeStartupTimeout, and unhealthyNodeConditions values for the node pool. Palette honors per-pool overrides only when Cluster Auto Remediation is enabled at the tenant or project level. If the MHC configuration is invalid, Palette records an error in the cluster event stream, ignores the override, and applies the default MHC. Any changes made after cluster deployment trigger a cluster repave.
Taints	Applies taints to the node pool so that pods are scheduled to the pool only if they tolerate the taint. To learn more, refer to the Taints and Tolerations guide. For Edge Native clusters, you can also apply taints to a specific node in the pool. Refer to Node-level Taints and Labels for Edge Native.
Dedicate node pool for system pods	Reserves the node pool for system pods. Available only on EKS and GKE clusters. To learn more, refer to Dedicated node pool for system pods.
Rolling update	Select Expand First, Contract First, or Custom to determine the order in which nodes are added to or removed from the worker node pool. - Expand First - Adds new nodes before removing old nodes. - Contract First - Removes old nodes before adding new nodes. - Custom - Set either an explicit numerical value or a percentage for Max Surge and Max Unavailable. Max Surge and Max Unavailable cannot both be set to 0. The Custom option is only supported for IaaS clusters.
Instance Option	AWS options for compute capacity. On Demand gives you full control over the instance lifecycle without long-term commitment. Spot allows the use of spare EC2 capacity at a discount but which can be reclaimed if needed.
Instance Type	The compute size.
Availability Zones	The Availability Zones from which to select available servers for deployment. If you select multiple zones, Palette will deploy servers evenly across them as long as sufficient servers are available to do so. If you select public subnets, ensure those subnets have automatic public IP addresses assigned. Otherwise, node deployment errors will occur. Automatic public IP address assignment is typically handled by the infrastructure provider Palette is deploying a cluster to. Discuss this with your network team for additional guidance.
Disk Size	Give the required storage size.

warning

Some features may not be available for all infrastructure providers. Review each infrastructure provider's node pool configuration settings to learn more.

Node-level Taints and Labels for Edge Native

For connected Edge Native clusters, you can apply taints and labels to a single node inside a node pool, in addition to the pool-level taints and labels that apply to every node. Node-level taints and labels are useful when nodes in the same pool need to play different roles.

A common example is a three-node control plane where two nodes run application workloads and a third, lightweight node participates only in etcd quorum. The lightweight node is sometimes called a witness node. To keep workloads off the witness, apply a node-level taint to that node only.

You set node-level taints and labels in the Palette UI on each host row inside the Edge Hosts section of an Edge Native node pool. The controls are available both during cluster creation and when you edit a node pool on an existing cluster.

Do not apply node-level taints with the NoSchedule or NoExecute effect to every node in a cluster. Palette

relies on system pods that carry out cluster operations and report cluster status back to Palette; these pods tolerate the control-plane taint and a small set of named taints, but do not tolerate a custom taint that you apply. If no node accepts them, they stay Pending, cluster operations fail, and the cluster does not report ready. Leave at least one node untainted, or apply only the PreferNoSchedule effect on the affected nodes. :::

How Pool-level and Node-level Values Interact

• Pool-level taints and labels apply to every node in the pool. A node-level taint or label applies only to the node it is set on.
• If a pool-level label and a node-level label share the same key, the node-level value takes effect on that node.
• A node cannot have more than one taint with the same key and effect combination. If a pool-level taint and a node-level taint share both the key and the effect, the node-level taint replaces the pool-level taint on that node. Otherwise, both the pool-level and the node-level taint apply.
• You cannot apply reserved label keys, such as those in the kubernetes.io or k8s.io namespaces, at the node level.
• Node-level taints and labels do not persist when the underlying Edge host is replaced. After a host replacement, reapply the node-level taints and labels you want on the new host.

Supported Distributions

Palette applies node-level taints and labels on the following Edge Native Kubernetes distributions:

• Palette eXtended Kubernetes - Edge (PXK-E)
• RKE2
• K3s
• Canonical Kubernetes

Other Kubernetes distributions ignore node-level taints and labels. The values are accepted in the Palette UI but do not reach the node.

Remove Node-level Taints and Labels

When you remove a node-level taint or label, Palette removes the value from the node on its next reconcile. Palette can only remove a value that it previously recorded, and it records a value on the first reconcile that happens while the node is reachable and the value is declared. If the node is never reachable for a reconcile while the value is declared, the value remains on the node after you remove the declaration.

If a taint remains on a node after you remove its declaration, remove it from the node manually with kubectl.

kubectl taint nodes <node-name> <key>:<effect>-

Tolerations for Custom Taints

Custom taints require matching tolerations on your cluster workloads. Container Network Interface (CNI) pods include a catch-all toleration and tolerate any custom taint. Container Storage Interface (CSI) pods do not automatically tolerate custom taints. You must add a toleration for your node-level taint to the CSI pack values in the cluster profile, at the path documented for each CSI pack, or storage pods fail to schedule on tainted nodes and the cluster's storage layer stops working.

Skip Worker Node Update

The Skip worker node update (Optional) toggle lets you upgrade the control plane of a cluster to a newer Kubernetes version while deferring the upgrade of one or more worker pools. This is useful for scenarios where you want to apply a security patch or initiate an LTS upgrade without affecting or repaving worker nodes. You can then schedule the worker pool upgrade for a time that is convenient for you, such as during a scheduled maintenance window.

Skipping worker node updates is supported for the following cluster types:

• AWS IaaS clusters
• Azure IaaS clusters
• GCP IaaS clusters
• MAAS clusters
• VMware vSphere clusters
• Apache CloudStack clusters
• Connected (centrally managed) Edge Native clusters
• Locally managed Edge Native clusters in an airgapped environment. Configure these clusters through Local UI or the Local UI API rather than through Palette. Refer to Edge Cluster Upgrade Behavior.

For Edge Native clusters, both connected and locally managed, only Palette eXtended Kubernetes Edge (PXK-E) and Canonical Kubernetes are supported. K3s and RKE2 clusters are not supported.

Cluster Profile Upgrade Behavior

When a cluster profile update bumps the Kubernetes version, Palette upgrades the control plane and any worker pools that do not have Skip worker node update enabled. Worker pools with the option enabled are skipped and stay at their current Kubernetes version.

info

Palette enforces the Kubernetes N-3 minor version skew. If enabling Skip worker node update on a pool would cause it to fall more than three minor versions behind the control plane, Palette blocks the update.

Scaling Behavior

Scaling behavior for a worker pool with Skip worker node update enabled differs by cluster type.

For AWS IaaS, Azure IaaS, GCP IaaS, MAAS, VMware vSphere, and Apache CloudStack clusters, scale-up is permitted. New nodes added manually or by the cluster autoscaler join using the worker pool's current Kubernetes version, not the control plane version. Scale-down is not restricted.

For Edge Native clusters, both connected and locally managed, scale-up is not permitted while the toggle is enabled. Scale-up requests on a pool with the toggle enabled are rejected, whether initiated manually or by the cluster autoscaler, because a new node cannot honor the pool's pinned Kubernetes version. To expand capacity, create a new worker pool and add Edge hosts to it instead.

Upgrade a Skipped Worker Pool

To sync the Kubernetes version of a skipped worker pool with the current Kubernetes control plane version, disable the Skip worker node update toggle on that pool.

danger

Disabling Skip worker node update triggers a repave of the worker pool. Ensure you are ready to repave before disabling the toggle.

For step-by-step instructions, refer to Trigger Worker Node Upgrade. For Edge Native-specific upgrade behavior, refer to Edge Cluster Upgrade Behavior.

Create a New Node Pool

Prerequisites

• An active cluster in Palette.

• The cluster.update permission that is required to edit clusters.

Create Node Pool

You can create a new node pool for an active cluster. To create a new node pool follow the steps below.

1. Log in to Palette.

2. Navigate to the left Main Menu and click on Clusters.

3. Click on the row of the cluster you want to edit the node pool.

4. Click on the Nodes tab.

5. Click on New Node Pool.

6. Fill out the input fields in the Add node pool page. Refer to the Node Pool Configuration Settings tables for more information on each field.

   If available for your environment, click Copy from Control Plane Pool to reuse the control plane pool's configuration in the new worker node pool. The copied fields remain editable.

7. Click on Confirm to create the new node pool.

Validate

After you create a new node pool, you can validate the node pool by following the steps below.

1. Log in to Palette.

2. Navigate to the left Main Menu and click on Clusters.

3. Click on the row of the cluster you added the new node pool.

4. Click on the Nodes tab.

5. Ensure the new node pool is listed in the Node Pools section and that all compute instances are in the healthy status.

Change a Node Pool

You can modify node pools for existing clusters. This includes changing the node pool's taints, adjusting the node repavement interval, modifying the number of compute instances in the node pool, activating or disabling autoscaler, and more. To make changes to an active cluster's node pool, follow the steps below.

warning

If autoscaler is enabled, you cannot adjust the number of nodes in the pool manually.

Prerequisites

• An active cluster in Palette.

• The cluster.update permission that is required to edit clusters.

Edit Node Pool

1. Log in to Palette.

2. Navigate to the left Main Menu and click on Clusters.

3. Select a cluster to edit the node pool.

4. Click on the Nodes tab.

5. The nodes details page is where you can review the existing node pools and their configuration. You can also add a new node pool from this page. Click on the Edit button to make changes to the node pool.

6. Make changes as needed. Refer to the Node Pool Configuration Settings tables for more information on each field.

7. Click on Confirm to update the node pool.

Validate

After you have modified a new node pool, you can validate the node pool by following the steps below.

1. Log in to Palette.

2. Navigate to the left Main Menu and click on Clusters.

3. Select the cluster with the new node pool.

4. Click on the Nodes tab.

5. Ensure the new node pool is listed in the Node Pools section and that all compute instances are in the healthy status.

Delete a Node Pool

You can delete a worker node pool from an active cluster using the following steps.

warning

The deletion of a node pool drains and removes its worker nodes. Before proceeding, ensure that your remaining node pools have enough resources to take on the workloads from the pool being deleted, to prevent any disruption.

Prerequisites

• An active cluster in Palette with at least one worker node pool.

• The cluster.update permission, which is required to edit clusters.

Delete Node Pool

1. Log in to Palette.

2. From the left Main Menu, click Clusters.

3. Select the cluster with the node pool you want to delete.

4. Click the Nodes tab.

5. Locate the node pool you want to delete and click the Delete button.

6. Click Delete again to confirm the deletion. This action will drain and delete all nodes in the node pool.

Validate

Use the following steps to confirm that the node pool was deleted successfully.

1. Log in to Palette.

2. From the left Main Menu, click on Clusters.

3. Select the cluster that had the deleted node pool.

4. Click the Nodes tab.

5. Verify that the deleted node pool is no longer listed in the Node Pools section and that all the remaining compute instances are in a healthy state.

Approve Cluster Repave

Prerequisites

• An active cluster in Palette.

• The cluster.update permission that is required to edit clusters.

• Pending cluster repave operations.

Approve Cluster Repave

1. Log in to Palette.

2. Navigate to the left Main Menu and click on Clusters.

3. Select a cluster with pending repave operations.

4. Click Updates.

   

5. Review and, optionally, override changes that the repave introduces.

   

6. Click Confirm updates.

Validation

1. Log in to Palette.

2. Navigate to the left Main Menu and click on Clusters.

3. Click on the Nodes tab.

4. Verify that all compute instances are in the healthy status and reflect the applied repave changes.