Skip to main content

Pause Agent Upgrades

Palette supports the Pause Agent Upgrades feature to exclude a cluster or a group of clusters from having their Palette agent automatically upgraded when Palette is upgraded. This feature only pauses upgrades for Palette agents, not updates to the clusters themselves.

If you specify a Palette agent image in the operating system pack of your cluster profile using the stylusPackage parameter, the Edge host installs that agent version when Palette applies the cluster profile. The stylusPackage parameter alone does not keep the agent on that version. Unless you pause agent upgrades, Palette continues to upgrade the agent automatically even if you set stylusPackage.

Pause Agent Upgrade Scopes​

Agent upgrades can be paused and resumed in the following scopes:

  • Pause agent upgrades for a single cluster
  • Pause agent upgrades for all clusters within a project
  • Pause agent upgrades for all clusters within a tenant

When determining if the agent upgrades for one cluster is paused or not, you only need to look at the setting for the cluster itself. Agent upgrade settings are always applied based on individual cluster settings. Project and tenant agent upgrade settings are not inherited - instead cluster level settings are set to match each time project and tenant level settings are changed.

Pausing or resuming agent upgrades at a higher-level scope will automatically pause or resume agent upgrades in the lower-level scopes. For example, if you pause agent upgrades at the tenant level, then agent upgrades will be paused for all projects within that tenant, and all clusters within those projects. Similarly, if you resume upgrades at the project level, then all clusters within that project will have their agent upgrades resumed.

This is a one-time change that happens at the moment when you pause or resume upgrades in the project or tenant scope, and it does not mandate that the same setting be kept at the lower scopes. If you pause or resume agent upgrades in a lower-level scope, it will override the setting from the higher-level scope. For example, even if all agent upgrades are paused at the tenant level, you can override the tenant-level pause by resuming upgrades in a specific project or a specific cluster.

warning

Overrides of agent upgrade settings are not permanent. When the pause agent settings at the project or tenant scope change, the agent upgrade setting in the cluster or project scopes will always be set to match the higher-level scope setting regardless. If you want to override the project or tenant level agent upgrade setting, you must change the agent upgrade setting in the lower scope after the change in the higher scope.

The following table lists some example upgrade configurations and whether the Palette agent will be upgrades in those settings. Note that only the settings at the cluster level determines whether the Palette agent will be upgraded.

TenantProjectClusterOutcome
Upgrades pausedUpgrades pausedUpgrades enabledPalette agent will upgrade automatically.
Upgrades enabledUpgrades enabledUpgrade pausedPalette agent upgrades are paused.

Agent Upgrades for PCG and Edge Hosts​

Aside from clusters, you can also pause the agent upgrades on Private Cloud Gateways (PCG) and Edge hosts that are registered with Palette but are not part of a cluster.

Since PCGs are scoped to tenants, you can pause the agent upgrades on a PCG by pausing agent upgrades on the tenant to which the PCG is associated. You can also pause or resume upgrades for a PCG in the PCG details page through Cluster Settings. Similar to clusters, pausing and resuming upgrades at the tenant level will pause or resume agent upgrades for all PCGs in the tenant. Pausing and resuming upgrades for a PCG individually will override the tenant-level setting.

Edge hosts that are part of a cluster have their agent upgrades managed by the settings of their cluster. Edge hosts that are not part of a cluster have their agent upgrades managed at the project and tenant level. Similar to clusters, pausing or resuming agent upgrades at the tenant level will automatically pause or resume agent upgrades for all projects within that tenant. However, you can override the tenant level setting by manually changing the upgrade setting at the project level.

The following is a table showing the scopes at which you can pause agent upgrades for different objects. The same relationship between the scopes applies: Changing the setting in a higher scope will trigger a one-time change to the lower scopes, and changing the setting at the lower scope will override the setting in the higher scope.

Individual Cluster/PCGProjectTenant
Cluster✅✅✅
PCG✅❌✅
Idle Edge hosts❌✅✅

Agent Upgrades for Virtual Clusters​

A Palette Virtual Cluster runs on a host cluster in its cluster group. The Palette agent on a virtual cluster installs some components into the host cluster that the host cluster's own Palette agent also manages. If the virtual cluster's agent upgrades first, it upgrades those shared components before the host cluster's agent does. The host cluster can then no longer scale, repave, or replace its nodes, and its virtual clusters stop applying cluster profile changes. Workloads continue to run.

warning

Do not let the Palette agent on a virtual cluster upgrade before the Palette agent on its host cluster. If you cannot upgrade the Palette agent on the host cluster yet, keep Pause Agent Upgrades enabled on its virtual clusters.

For the most control over the order, change the setting on each cluster individually. If you resume agent upgrades at the project or tenant scope instead, the host cluster and its virtual clusters all upgrade, in no set order. To find the setting on a cluster, refer to the Single Cluster tab in Enablement.

Use the following order to upgrade the Palette agents on a host cluster and its virtual clusters.

  1. Enable Pause Agent Upgrades on each virtual cluster that runs on the host cluster, if it is not already enabled.

  2. On the host cluster's Overview tab, note the version in the Agent version field.

  3. Disable Pause Agent Upgrades on the host cluster.

  4. Wait until the Agent version field on the host cluster's Overview tab shows a newer version. The host cluster's agent upgrade is then complete.

  5. Disable Pause Agent Upgrades on the virtual clusters.

If a host cluster is already affected, disable Pause Agent Upgrades on the host cluster. After its agent upgrade completes, the host cluster recovers, and its virtual clusters can finish their agent upgrades.

Limitations​

  • Do not pause platform upgrades for clusters if all of the following conditions apply:

    • Your cluster is an Edge cluster.
    • Your Palette/VerteX instance is on version 4.5.x or later.
    • Your agent version is older than 4.5.0.
    • Your Edge cluster has the local Harbor registry enabled.

    Using an agent version older than 4.5.0 with Palette/VerteX versions 4.5.x or later may lead to image download issues, issues with pushing images to the local registry, and cluster deployment issues. Ensure that you enable platform upgrades so your agent can be upgraded to be compatible with Palette/VerteX.

Agent Version Drift Warning​

While agent upgrades are paused, your Palette or VerteX instance continues to advance. If the management plane moves more than two major versions ahead of a cluster's agents, that cluster falls outside the supported N-2 compatibility window and a warning banner appears on the cluster.

The banner advises you to disable Pause Agent Upgrades so the agents can update to the latest version. Leaving a cluster outside the compatibility window might make it unhealthy, so treat the banner as an action to take rather than a notice to dismiss. Agents resume upgrading as soon as you disable the setting, which returns the cluster to a supported state without any further action.

For background on why cluster-side components are supported two major versions behind the management plane, refer to CRD API Versions.

Prerequisites​

  • Cluster admin permissions or Tenant admin permissions when pausing upgrades for all clusters within tenant scope.

  • An existing cluster. Follow this tutorial to get started.

Enablement​

Use the tabs below to view the instructions for enabling the Pause Agent Upgrades feature for a single cluster, all clusters within the project scope, or all within the tenant scope.

  1. Log in to Palette.

  2. Navigate to the left Main Menu and click on Clusters.

  3. Select your cluster.

  4. Select Settings > Cluster Settings.

  5. Toggle the Pause Agent Upgrades button to pause cluster upgrades.

  6. A pop-up box will ask you to confirm the action. Click OK.

Validate​

  1. Log in to Palette.

  2. Navigate to the left Main Menu and click on Clusters.

  3. Select your cluster.

  4. From the cluster details page, click Settings > Cluster Settings.

  5. The Pause Agent Upgrades toggle button is checked.