Configure OS Upgrades

info

This page covers on-demand OS upgrades for agent mode clusters, where you supply the host and own the operating system. For clusters that Palette provisioned from a Palette-built VM image, for example on AWS, Azure, GCP, VMware, or MAAS, refer to OS Patching instead.

Agent mode hosts install and manage their Operating System (OS) outside Palette. This approach brings great flexibility in terms of architecture, but it has the drawback that Palette cannot upgrade, patch or manage the operating systems of the hosts. This can lead to inconsistencies, missed updates, or operational risks.

This page demonstrates how to perform an OS upgrade by leveraging cluster profiles. You will learn how to create your own Kubernetes manifest containing your custom OS upgrade script. Your cluster nodes will then be selected based on configured node labels and upgraded in a single rolling pass. To run the upgrade again, refer to Run the Upgrade Again. Re-applying unchanged manifests does nothing.

Prerequisites

A Palette cluster deployed on one or multiple hosts with the Palette Agent installed. Refer to the Install Agent Mode guide for further details. The cluster should be listed as Healthy and with a Running status.
- The host must have access to the internet and a connection to Palette.
Access to a terminal with network access to your cluster.
Kubectl installed locally. Refer to the Kubernetes Install Tools guide for further details.

Enablement

Log in to Palette.
Navigate to the left main menu and select Clusters.
Select your cluster to access the cluster details page.
Download the kubeconfig file for your cluster. Open a terminal and navigate to the location of the file.
Set the KUBECONFIG environment variable to the file path of the kubeconfig file to enable you to connect to the cluster using kubectl CLI. Refer to the Access Cluster with CLI section for further guidance.
```
export KUBECONFIG=/path/to/your/kubeconfig
```
Execute the following commands to find the spectro-task-XXX namespace of your cluster and save it to the SYSTEM_UPGRADE_NAMESPACE variable. This namespace will be different between clusters.
```
export SYSTEM_UPGRADE_NAMESPACE=$(kubectl get namespaces --no-headers --output custom-columns=":metadata.name" | grep '^spectro-task')
echo $SYSTEM_UPGRADE_NAMESPACE
```
The output will be similar to the following snippet.
```
spectro-task-6851ddd04b1b188784c06291
```
Execute the following command in your terminal, replacing the placeholder with a node label of your choice. This variable allows you to customize which nodes should be updated. The command saves your label to the SYSTEM_UPGRADE_NODE_LABEL variable.
```
export SYSTEM_UPGRADE_NODE_LABEL="REPLACE ME"
```
Apply the node label to all the nodes that you want updated. Execute the command by replacing the placeholder with the name of the node. Repeat this step for each node you want to upgrade.
```
kubectl label node REPLACE-ME $SYSTEM_UPGRADE_NODE_LABEL=
```
Save your upgrade scripts to a file titled upgrades.sh. You can provide any instructions that you want to execute on system upgrade and reboot. The following example provides upgrade instructions for Ubuntu, but you can modify them to work according to your host operating system. The command creates the upgrades.sh file in your local directory.

Palette does not reboot nodes. If an upgrade requires a reboot, your script must reboot the node synchronously, as the example script does. The node stays drained during the reboot. After the node restarts, the task runs the script again to finish the upgrade on that node before it moves to the next node. The first attempt on a rebooted node appears as a failed pod, which is expected.
```
cat << 'EOF' > upgrades.sh
#!/bin/sh
set -e
export DEBIAN_FRONTEND=noninteractive
apt-get --assume-yes update
apt-get --option Dpkg::Options::="--force-confold" dist-upgrade --yes
if [ -f /var/run/reboot-required ]; then
  echo "Reboot required. Rebooting while the node is still drained."
  systemctl reboot
  # The node goes down here. After it boots, the Job starts a new pod that reruns this
  # script. Updates are already installed and /var/run/reboot-required is cleared at
  # boot, so the rerun exits 0.
  sleep 600
  exit 1
fi
EOF
```
The example checks /var/run/reboot-required, which is specific to Ubuntu and Debian hosts. On RHEL-family hosts, use needs-restarting --reboothint from the yum-utils package instead. It exits 1 when a reboot is required and 0 otherwise.

Execute the following commands to create the upgrades.yaml file using your namespace, label, and upgrade script variables.

cat << EOF > upgrades.yaml
---
apiVersion: v1
kind: Secret
metadata:
    name: os-upgrade-script
    namespace: $SYSTEM_UPGRADE_NAMESPACE
type: Opaque
stringData:
    upgrade.sh: |
$(sed 's/^/        /' upgrades.sh)
---
apiVersion: cluster.spectrocloud.com/v1alpha1
kind: SpectroSystemTask
metadata:
    name: os-upgrade-plan
    namespace: $SYSTEM_UPGRADE_NAMESPACE
spec:
    concurrency: 1
    nodeSelector:
        matchExpressions:
            - { key: $SYSTEM_UPGRADE_NODE_LABEL, operator: Exists }
    serviceAccountName: crony
    secrets:
        - name: os-upgrade-script
          path: /host/run/spectro-task/secrets/os-upgrade
    tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/controlplane
          operator: Exists
          effect: NoSchedule
    drain:
        force: true
    version: "1"
    task:
        image: us-docker.pkg.dev/palette-images/third-party/ubuntu:22.04
        command: ["chroot", "/host"]
        args: ["sh", "/run/spectro-task/secrets/os-upgrade/upgrade.sh"]
EOF

The command creates the upgrades.yaml file in your current directory. The YAML file defines the following Kubernetes resources.

Resource	Name	Description
`Secret`	`os-upgrade-script`	Stores the `upgrade.sh` shell script, which defines the upgrade logic to be executed on target nodes. This script is mounted into the upgrade pod via a secret volume.
`SpectroSystemTask`	`os-upgrade-plan`	Describes the upgrade. Palette reconciles the task, drains each labeled node in turn, runs the script against the host filesystem, and returns the node to service when the script completes successfully.

The tolerations list covers the standard control plane taints that current and earlier Kubernetes versions apply. If your nodes carry other taints, such as on GPU nodes or dedicated node pools, add a matching toleration for each taint. Otherwise, the upgrade pod cannot be scheduled on those nodes.

warning

Keep concurrency: 1 when the task targets control plane nodes so that only one etcd member is down at a time. Ensure that your cluster has enough resources to perform rolling upgrades in order to avoid outages.

Navigate back to Palette in your browser. Select Profiles from the left main menu.
Select the cluster profile corresponding to your agent mode cluster.
Click on the version drop-down menu. Select the Create new version option. Fill in the Version input and click Confirm to create a new version of your cluster profile. The new profile version opens.
Click Add manifest. The manifest editor appears. Fill in the Layer name input field. Then, click New Manifest. Input a name for the manifest file. Click on the check or press Enter to open the editor.
Paste the contents of the upgrades.yaml file that you have created in Step 10. Click Confirm Updates to save your manifest. Then, click Save Changes to save your manifest to the cluster profile.
Navigate to the left main menu and select Clusters.
Select your cluster to access the cluster details page.
Click on the Profiles tab.
Select the newly created version of your cluster profile. Click Save.

Palette applies your manifest to the cluster. The Kubernetes resources responsible for the system upgrade are created in the spectro-task-xxx namespace.

Validate

Log in to Palette.
Navigate to the left main menu and select Clusters.
Select your cluster to access the cluster details page.
Download the kubeconfig file for your cluster. Open a terminal and navigate to the location of the file.
Set the KUBECONFIG environment variable to the file path of the kubeconfig file to enable you to connect to it using kubectl CLI. Refer to the Access Cluster with CLI section for further guidance.
```
export KUBECONFIG=/path/to/your/kubeconfig
```
Execute the following commands to find the spectro-task-XXX namespace of your cluster and save it to the SYSTEM_UPGRADE_NAMESPACE variable. This namespace will be different between clusters.
```
export SYSTEM_UPGRADE_NAMESPACE=$(kubectl get namespaces --no-headers --output custom-columns=":metadata.name" | grep '^spectro-task')
echo $SYSTEM_UPGRADE_NAMESPACE
```
The output will be similar to the following snippet.
```
spectro-task-6851ddd04b1b188784c06291
```

Issue the following command to retrieve the secret and the SpectroSystemTask under the spectro-task-xxxxx namespace.

kubectl get secret,spectrosystemtask --namespace $SYSTEM_UPGRADE_NAMESPACE

Confirm the secret secret/os-upgrade-script and the task spectrosystemtask.cluster.spectrocloud.com/os-upgrade-plan were created successfully.

Example Output
NAME                           TYPE     DATA   AGE
secret/cert-renewal-script     Opaque   1      41m
secret/ntp-update-config       Opaque   1      42m
secret/ntp-update-script       Opaque   1      42m
secret/os-upgrade-script       Opaque   1      10m
secret/sshkeys-update-script   Opaque   1      41m
secret/stylus-upgrade          Opaque   1      41m

NAME                                                              AGE
spectrosystemtask.cluster.spectrocloud.com/cert-renewal-task      41m
spectrosystemtask.cluster.spectrocloud.com/os-upgrade-plan        10m
spectrosystemtask.cluster.spectrocloud.com/stylus-upgrade         41m

Palette creates the cert-renewal-task and stylus-upgrade tasks on every cluster. The task that you created is os-upgrade-plan.

Watch the per-node upgrade Jobs as they run.

kubectl get jobs --namespace $SYSTEM_UPGRADE_NAMESPACE --watch

Confirm which nodes finished.
```
kubectl get nodes --label-columns task.cluster.spectrocloud.com/os-upgrade-plan
```
Palette labels each node with the task name and content hash once that node's Job succeeds. A node without the label has not completed yet.
(Rebooted nodes only) Log in to each node that rebooted, and issue the following command to display when the node last booted.
```
uptime --since
```
The timestamp should be after the Job for that node started.

Recover a Failed Node

If the script fails, the node stays cordoned, and the Job retries up to 99 times. To recover the node, use one of the following methods.

Fix the script. Updating the script triggers a rerun.
Delete the task, and then return the node to service. Deleting the task without this step leaves the node cordoned.

Run the Upgrade Again

Palette only re-runs the upgrade when the SpectroSystemTask manifest changes. To trigger another run, update spec.version on the task, publish the change as a new cluster profile version, and apply that version to the cluster.

Log in to Palette.
Select Profiles from the left main menu and open the cluster profile that contains your upgrades.yaml manifest.
Click on the version drop-down menu and select Create new version. Fill in the Version input and click Confirm. The new profile version opens.
Open the manifest you created in the Enablement section. Change spec.version on the SpectroSystemTask to a new value, for example by incrementing the current value. Click Confirm Updates, then click Save Changes.
Navigate to the left main menu and select Clusters. Select your cluster to access the cluster details page.
Click the Profiles tab, select the newly created version of your cluster profile, and click Save.

Palette applies the updated profile, which re-runs the upgrade script on every labeled node.

Prerequisites​

Enablement​

Validate​

Recover a Failed Node​

Run the Upgrade Again​

Prerequisites

Enablement

Validate

Recover a Failed Node

Run the Upgrade Again