Version: latest

Configure Regularly Scheduled OS Upgrades

Agent mode hosts install and manage their Operating System (OS) outside Palette. This approach brings great flexibility in terms of architecture, but it has the drawback that Palette cannot upgrade, patch or manage the operating systems of the hosts. This can lead to inconsistencies, missed updates, or operational risks.

This page demonstrates how to configure regularly scheduled OS upgrades by leveraging cluster profiles and the system upgrade controller already installed by Palette. You will learn how to create your own Kubernetes manifest containing your custom OS upgrade script. Your cluster nodes will then be selected based on configured node labels and upgraded periodically according to a Cron schedule you choose.

Prerequisites

A Palette cluster deployed on one or multiple hosts with the Palette Agent installed. Refer to the Install Agent Mode guide for further details. The cluster should be listed as Healthy and with a Running status.
- The host must have access to the internet and a connection to Palette.
Access to a terminal with network access to your cluster.
Kubectl installed locally. Refer to the Kubernetes Install Tools guide for further details.

Enablement

Log in to Palette.
Navigate to the left Main Menu and select Clusters.
Select your cluster to access the cluster details page.
Download the kubeconfig file for your cluster. Open a terminal and navigate to the location of the file.
Set the KUBECONFIG environment variable to the file path of the kubeconfig file to enable you to connect to it using kubectl CLI. Refer to the Access Cluster with CLI section for further guidance.
```
export KUBECONFIG=/path/to/your/kubeconfig
```
Execute the following commands to find the system-upgrade-XXX namespace of your cluster and save it to the SYSTEM_UPGRADE_NAMESPACE variable. This namespace will be different between clusters.
```
export SYSTEM_UPGRADE_NAMESPACE=$(kubectl get namespaces --no-headers --output custom-columns=":metadata.name" | grep '^system-upgrade')
echo $SYSTEM_UPGRADE_NAMESPACE
```
The output will be similar to the following snippet.
```
system-upgrade-67991934afb6a8ea13ee0e01
```

Provide an upgrade frequency using a Cron format. This is used to configure a Kubernetes CronJob to execute upgrades on a repeating schedule. You can find some common examples of Cron schedules in the following table.

Expression	Description
`0 0 1 1 *`	Once a year at midnight of 1 January
`0 0 1 * *`	Once a month at midnight of the first day of the month
`0 0 * * 0`	Once a week at midnight on Sunday morning
`0 0 * * *`	Once a day at midnight
`0 * * * *`	Once an hour at the beginning of the hour

Execute the following command in your terminal, replacing the placeholder with your preferred Cron schedule. The command saves your chosen schedule to the SYSTEM_UPGRADE_SCHEDULE variable.

export SYSTEM_UPGRADE_SCHEDULE="REPLACE ME"

Execute the following command in your terminal, replacing the placeholder with a node label of your choice. This variable allows you to customize which nodes should be periodically updated. The command saves your label to the SYSTEM_UPGRADE_NODE_LABEL variable.
```
export SYSTEM_UPGRADE_NODE_LABEL="REPLACE ME"
```
Apply the node label to all the nodes that you want updated. Execute the command by replacing the placeholder with the name of the node. Repeat this step for each node you want to upgrade.
```
kubectl label node REPLACE-ME $SYSTEM_UPGRADE_NODE_LABEL=
```
info
Nodes are drained, upgraded, and rebooted one by one. Ensure that your cluster has enough resources to perform rolling upgrades in order to avoid outages.
Save your upgrade scripts to a file titled upgrades.sh. You can provide any instructions that you want to execute on system upgrade and reboot. The following example provides upgrade instructions for Ubuntu, but you can modify them to work according to your host operating system. The command creates the upgrades.sh file in your local directory.
```
cat << 'EOF' > upgrades.sh
#!/bin/sh
set -e
secrets=$(dirname "$0")
export DEBIAN_FRONTEND=noninteractive
apt-get --assume-yes update
apt-get -o Dpkg::Options::="--force-confold" dist-upgrade -q -y --force-yes
EOF
```

Execute the following commands to create the upgrades.yaml file using your namespace, upgrade schedule, labels, and upgrade script variables.

cat << EOF > upgrades.yaml
---
apiVersion: v1
kind: Secret
metadata:
    name: os-upgrade-script
    namespace: $SYSTEM_UPGRADE_NAMESPACE
type: Opaque
stringData:
    upgrade.sh: |
$(sed 's/^/        /' upgrades.sh)
---
apiVersion: v1
kind: Secret
metadata:
    name: os-upgrade-plan
    namespace: $SYSTEM_UPGRADE_NAMESPACE
type: Opaque
stringData:
    plan.yaml: |
        apiVersion: upgrade.cattle.io/v1
        kind: Plan
        metadata:
            name: os-upgrade-plan
            namespace: $SYSTEM_UPGRADE_NAMESPACE
        spec:
            concurrency: 1
            nodeSelector:
                matchExpressions:
                    - { key: $SYSTEM_UPGRADE_NODE_LABEL, operator: Exists }
            serviceAccountName: system-upgrade
            secrets:
                - name: os-upgrade-script
                  path: /host/run/system-upgrade/secrets/bionic
            tolerations:
                - key: node-role.kubernetes.io/master
                  operator: Exists
                  effect: NoSchedule
                - key: node-role.kubernetes.io/controlplane
                  operator: Exists
                  effect: NoSchedule
            drain:
                force: true
            version: bionic
            upgrade:
                image: us-docker.pkg.dev/palette-images/third-party/ubuntu:22.04
                command: ["chroot", "/host"]
                args: ["sh", "/run/system-upgrade/secrets/bionic/upgrade.sh"]
---
apiVersion: batch/v1
kind: CronJob
metadata:
    name: os-upgrade-cronjob
    namespace: $SYSTEM_UPGRADE_NAMESPACE
spec:
    schedule: "$SYSTEM_UPGRADE_SCHEDULE"
    jobTemplate:
        spec:
            template:
                spec:
                    serviceAccountName: system-upgrade
                    containers:
                        - name: os-upgrade-job
                          image: us-docker.pkg.dev/palette-images/third-party/ubuntu:22.04
                          command:
                            - sh
                            - -c
                            - |
                              apt-get update
                              apt-get install -y curl
                              curl -LO "https://dl.k8s.io/release/\$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
                              chmod +x kubectl
                              mv kubectl /usr/local/bin/
                              export KUBECONFIG=/run/kubeconfig
                              kubectl get plan os-upgrade-plan --namespace $SYSTEM_UPGRADE_NAMESPACE
                              if [ \$? -eq 0 ]; then
                                echo "Upgrade plan exists. Retrigger it."
                                VERSION="os-upgrade-plan-\$(date +%Y%m%d%H%M%S)"
                                kubectl patch plan os-upgrade-plan --namespace $SYSTEM_UPGRADE_NAMESPACE --type=json --patch="[{\"op\": \"replace\", \"path\": \"/spec/version\", \"value\": \"\${VERSION}\"}]"
                              else
                                echo "Upgrade plan does not exist. Create it."
                                kubectl get secret os-upgrade-plan --namespace $SYSTEM_UPGRADE_NAMESPACE --output go-template='{{ index .data "plan.yaml" | base64decode }}' | kubectl apply --filename -
                                fi
                    restartPolicy: OnFailure
EOF

The command creates the upgrades.yaml file in your current directory. The YAML file defines the following Kubernetes resources.

Resource	Name	Description
`Secret`	`os-upgrade-script`	Saves the upgrade instructions to an opaque secret.
`Secret`	`os-upgrade-plan`	Saves the upgrade script execution on the cluster nodes using the specified upgrade concurrency and node selector labels. The execution is configured using the `Plan` resource defined by the `system-upgrade-controller` project. Palette has installed all required dependencies for this project.
`CronJob`	`os-upgrade-job`	Schedules the upgrade plan to execute at regular intervals and provides a restart policy should the plan fail.

Navigate back to Palette in your browser. Select Profiles from the left Main Menu.
Select the cluster profile corresponding to your agent mode cluster.
Click on the version drop-down Menu. Select the Create new version option. Fill in the Version input and click Confirm to create a new version of your cluster profile. The new profile version opens.
Click Add manifest. The manifest editor appears. Fill in the Layer name input field. Then, click New Manifest. Input a name for the manifest file. Click on the check or press Enter to open the editor.
Paste the contents of the upgrades.yaml file that you have created in Step 11. Click Confirm Updates to save your manifest. Then, click Save Changes to save your manifest to the cluster profile.
Navigate to the left Main Menu and select Clusters.
Select your cluster to access the cluster details page.
Click on the Profiles tab.
Select the newly created version of your cluster profile. Click Save.

Palette applies your manifest to the cluster. The Kubernetes resources responsible for the system upgrade are created in the system-upgrade-xxx namespace.

Validate

Log in to Palette.
Navigate to the left Main Menu and select Clusters.
Select your cluster to access the cluster details page.
Click Settings and select Download Logs from the dropdown. The Download Logs window appears.
Ensure that Node Logs is selected and click Download. The logs will take a few minutes to generate. A download link will appear once the logs are ready.

The download contains a zip archive of files. Details of all executed upgrades are in a folder with the same name as your system upgrade namespace. You can search for executions of the os-upgrade-plan in this file. The following snippet provides an example of the logging output you will find.

time="2025-02-05T12:41:36Z" level=debug msg="PLAN GENERATING HANDLER: plan=system-upgrade-67a34d38a6c18dc781e61927/os-upgrade-plan@17071, status={Conditions:[{Type:LatestResolved Status:True LastUpdateTime:2025-02-05T12:41:36Z LastTransitionTime: Reason:Version Message:}] LatestVersion:os-upgrade-plan-20250205124022 LatestHash:66a0761c0738aed60e58393b923ea083d13c6783c11dbcc74a64a99a Applying:[edge-4238a4fc050ab635da929b5d0b272380]}" func="github.com/rancher/system-upgrade-controller/pkg/upgrade.(*Controller).handlePlans.func2 " file="/workspace/pkg/upgrade/handle_upgrade.go:68"
time="2025-02-05T12:41:36Z" level=debug msg="PLAN STATUS HANDLER: plan=system-upgrade-67a34d38a6c18dc781e61927/os-upgrade-plan@17071, status={Conditions:[{Type:LatestResolvedStatus:True LastUpdateTime:2025-02-05T12:41:36Z LastTransitionTime: Reason:Version Message:}] LatestVersion:os-upgrade-plan-20250205124022 LatestHash:66a0761c0738aed60e58393b923ea083d13c6783c11dbcc74a64a99a Applying:[edge-4238a4fc050ab635da929b5d0b272380]}" func="github.com/rancher/system-upgrade-controller/pkg/upgrade.(*Controller).handlePlans.func1" file="/workspace/pkg/upgrade/handle_upgrade.go:30"

You can add further logging to your upgrade script if you require granular detail of its execution.

Prerequisites​

Enablement​

Validate​

Prerequisites

Enablement

Validate