Configure the OS Upgrade Timeout
When you upgrade the OS of an existing Edge cluster, each upgrade attempt on an Edge host has a time limit. The default limit is 45 minutes. In appliance mode, an attempt includes the download of the new provider image. In agent mode, it includes the download of the Kubernetes image. Edge hosts with slow network connections might need more time. If an attempt does not finish in time, the upgrade is retried on that Edge host later.
To change the limit, set the system.upgradeTimeoutMinutes parameter in the OS layer of the cluster profile. The
parameter applies to both appliance mode and agent mode deployments.
How the Upgrade Timeout Works
-
The limit applies to each upgrade attempt on an Edge host, not to the upgrade of the whole cluster.
-
The limit applies to OS upgrades of control plane and worker nodes in existing clusters. It does not apply when you create a cluster.
-
In agent mode, Palette does not upgrade the OS of the Edge host. The limit applies to the upgrades that Palette runs on the Edge host when you update the OS or Kubernetes layer of the cluster profile. It does not apply to custom OS upgrades that you run on agent mode hosts.
-
When an attempt times out, the upgrade is retried on that Edge host after 1 hour. The wait doubles after each attempt that times out, up to 24 hours, and the retries continue until an attempt succeeds.
-
Palette does not show that an attempt timed out or that a retry is pending. While an Edge host waits for its next attempt, Local UI displays an OS upgrade failed banner on the Edge Host page and on the Overview tab of the Cluster page. Local UI also labels the OS of the Edge host Upgrade failed, but it does not show when the next attempt starts. The banner clears when an attempt succeeds.
-
The limit is read when an OS upgrade starts. A new value applies to the next OS upgrade, not to an upgrade that is already running or waiting for a retry. To recover from an upgrade that keeps timing out, improve the network connection of the Edge host or wait for the next retry.
-
Changing only
system.upgradeTimeoutMinutesdoes not trigger a repave, reboot, or service restart. Refer to Edge Cluster Upgrade Behavior.
Prerequisites
-
A Palette account with permission to update cluster profiles and clusters.
-
An Edge cluster profile, or an existing Edge cluster and the cluster profile that it uses.
-
Every Edge host of the cluster must run the Palette agent version that ships with Palette 4.10.44 or later. Earlier versions of the agent ignore
system.upgradeTimeoutMinutesand limit each attempt to 15 minutes. -
(Optional) Access to the cluster with kubectl to check the limit that an OS upgrade uses. Refer to Access Cluster with CLI.
Configure the Upgrade Timeout
-
Log in to Palette.
-
From the left main menu, select Profiles.
-
Select the cluster profile of the clusters whose upgrade timeout you want to change, and create a new version of it. Refer to Version a Cluster Profile.
-
In the new cluster profile version, select the OS layer.
-
In the
optionssection, add thesystem.upgradeTimeoutMinutesparameter, and set it to the number of minutes that each upgrade attempt can run.options:
system.uri: "existing-value"
system.upgradeTimeoutMinutes: 90The
existing-valueplaceholder stands for the current value ofsystem.uri. Do not change it.Parameter Description Default system.upgradeTimeoutMinutesThe number of minutes that each OS upgrade attempt can run on an Edge host. Enter a whole number from 1to4320(3 days). If the value is zero, negative, a decimal, text, or greater than4320, the default applies.45 -
Select Confirm Updates, and then select Save Changes.
-
Apply the new cluster profile version to your clusters. The new limit applies to the next OS upgrade of each cluster.
-
To deploy a new cluster, refer to Create Cluster Definition.
-
To update a centrally managed cluster, refer to Update a Cluster.
-
To update a locally managed cluster, upload the new cluster profile version in Local UI. Refer to Update Local Cluster.
-
Validate
Steps 1 through 4 apply to centrally managed clusters. Steps 5 and 6 are optional and apply to both centrally managed and locally managed clusters.
-
Log in to Palette.
-
From the left main menu, select Clusters, and then select your cluster.
-
Select the Profile tab, and then select the OS layer.
-
Confirm that the
optionssection showssystem.upgradeTimeoutMinuteswith the value that you set. -
(Optional) After the next OS upgrade of the cluster starts, issue the following commands to display the limit that the upgrade uses.
export KUBECONFIG=<path-to-kubeconfig>
kubectl get spectrosystemtasks --all-namespaces \
--output custom-columns='NAME:.metadata.name,DEADLINE_SECONDS:.metadata.annotations.spectrocloud\.com/job-active-deadline-seconds'Replace
<path-to-kubeconfig>with the path to the kubeconfig file of the cluster.Example OutputNAME DEADLINE_SECONDS
control-plan 5400 -
Confirm that the
DEADLINE_SECONDScolumn shows the limit in seconds for thecontrol-plantask, and for theworker-plantask if the cluster has worker nodes. For example, a limit of 90 minutes displays as5400.