Enable Automatic TLS Certificate Rotation on the Zot Primary Registry
Edge clusters that use the in-cluster Zot registry as a primary registry serve a TLS certificate that expires after 365 days. In earlier versions of the Zot pack, this certificate has to be rotated manually before it expires.
Starting with the zot pack version 0.1.89-rev2, the certificate lifecycle for the in-cluster
Zot registry is managed by
cert-manager. After the
upgrade, cert-manager issues the Zot server certificate from a self-signed root Certificate
Authority (CA) and renews the server certificate automatically before it expires. The
reloader pack watches the Kubernetes Secret that stores
the certificate and restarts the Zot pods when the Secret changes, so the rotated certificate is
picked up without manual intervention.
This page explains how to upgrade an existing cluster profile to use this rotation flow.
Required Packs
The following table lists the pack versions used by the upgrade.
| Layer | Pack | Version to select |
|---|---|---|
| Registry | zot | 0.1.89-rev2 |
| Registry | registry-connect | 0.2.1 or later |
| Add-on | reloader | Any 1.4.x version, for example, 1.4.19 |
| Add-on | cert-manager | Any version already supported by your profile |
The zot pack version 0.1.89-rev2 declares reloader as a required pack dependency and
registry-connect version 0.2.1 or later as a recommended companion. You can use the
cert-manager version shipped with Palette or add the cert-manager pack to the profile if you
want to override its settings. The Zot chart provisions the cert-manager ClusterIssuer,
Issuer, and Certificate resources during installation.
Limitations
- The
reloaderpack is required. Without it, cert-manager rotates the Kubernetes Secret that stores the Zot server certificate, but the Zot pods continue to serve the previous certificate from memory until they are restarted manually.
Prerequisites
-
An existing Edge cluster that uses an Edge profile with the in-cluster Zot registry pack (
zotversion0.1.89or earlier) and theregistry-connectpack. -
Permission to edit and version the cluster profile in Palette.
-
cert-manager installed in the target cluster, either through the version shipped with Palette or through the
cert-managerpack in the profile. -
Healthy persistent storage backing the existing Zot Persistent Volume Claim (PVC),
zot-pvcin thezot-systemnamespace. The upgrade does not change the PVC, so image data is preserved.
Enable Automatic TLS Rotation on the Zot Registry
Perform the following steps in the cluster profile, and then apply the new profile version to the target clusters.
-
Log in to Palette.
-
From the left main menu, select Profiles.
-
Select the cluster profile that you want to upgrade.
-
Add or upgrade the reloader add-on pack.
-
If the
reloaderpack is not already in the profile, select Add New Pack and add the pack at a version in the1.4.xline, for example,reloader 1.4.19. -
If the
reloaderpack is already in the profile at a version earlier than1.4.x, upgrade the pack to a1.4.xversion.
The Zot chart writes the annotation
secret.reloader.stakater.com/reload: zot-tlson the Zot StatefulSet. Thereloaderpack watches for this annotation and triggers a pod restart when thezot-tlsKubernetes Secret is updated by cert-manager. -
-
Upgrade the Registry Connect pack to version
0.2.1or later.
-
In the Presets panel of the Registry Connect pack, select Zot Internal Registry with Cert-Manager. The preset name is
zot-registry-preset-cert.Selecting this preset switches the trust bundle reference in
charts.registry-connect.config.certificatesfrom the legacyzot-secretSecret and itscaCrtkey to the cert-manager-managedzot-tlsSecret.charts:
registry-connect:
config:
certificates:
- certificateRef:
kind: Secret
name: zot-tls
namespace: zot-system
key: ca.crtThe remaining values in the preset, such as the sync configuration, in-cluster registry credentials, and storage check, remain equivalent to the standard Zot Internal Registry preset.
-
Select Confirm & Create to finish customizing the Registry Connect pack.
- Upgrade the Zot pack to version
0.1.89-rev2.
-
In the values.yaml file of the Zot pack, confirm the following two parameters are set to
true. Both are enabled by default in the new pack and do not require edits in most cases.charts:
zot:
tls:
certManager:
enabled: true # turns on cert-manager-issued certificates
reloader:
enabled: true # adds the reloader annotation to Zot podsThe remaining defaults under
tls.certManagerdescribe the cert-manager plumbing that the chart provisions. Do not change them unless you have a specific reason to do so.Parameter Default value Description tls.certManager.bootstrapCA.enabledtrueCreates a self-signed ClusterIssuernamededge-registry-ca, a namespace-scopedIssuernamedinternal-ca, and a root CACertificatenamedroot-cain thezot-systemnamespace.tls.certManager.bootstrapCA.rootCA.duration87600h0m0sSets a 10-year duration for the root CA. tls.certManager.bootstrapCA.rootCA.privateKeyRotationPolicyNeverPreserves the root CA private key across renewals so that downstream trust bundles remain valid. tls.certManager.certificate.duration8760h0m0sSets a 1-year duration for the Zot server certificate. tls.certManager.certificate.renewBefore360h0m0sInstructs cert-manager to renew the Zot server certificate 15 days before it expires. -
Select Confirm & Create.
-
Save and publish the new version of the cluster profile.
-
Apply the new profile version to the target cluster or clusters. Refer to Update a Cluster for more information.
infoWhen the profile is applied, Palette rolls out the changes in the standard order. cert-manager reconciles the new
ClusterIssuer,Issuer, root CACertificate, and Zot serverCertificateresources in thezot-systemnamespace, and writes the root CA into theroot-ca-secretSecret and the Zot server certificate into thezot-tlsSecret. The Zot StatefulSet is then upgraded so that its pod template mounts thezot-tlsSecret at/certsand carries the annotationsecret.reloader.stakater.com/reload: zot-tls. Theregistry-connectpack, using the new preset, readszot-system/zot-tlsand distributesca.crtinto the cluster trust bundle so that workloads pulling from the Zot registry trust the new CA. On future rotations, cert-manager rewriteszot-tls, and thereloaderpack restarts the Zot pods so that they pick up the new certificate.
Validate
Use the following commands against the target cluster to confirm that the automatic rotation is in place.
-
Confirm that cert-manager has issued both certificates and marked them as ready.
kubectl --namespace zot-system get certificatesThe output lists the
root-cacertificate and the Zot server certificate. Both must showREADY=True. -
Confirm that the new Kubernetes Secrets exist.
kubectl --namespace zot-system get secret root-ca-secret zot-tls -
Inspect the served certificate. The expiry date must be approximately one year from the current date.
kubectl --namespace zot-system get secret zot-tls --output jsonpath='{.data.tls\.crt}' \
| base64 --decode \
| openssl x509 -noout -subject -issuer -dates -
Confirm that the Zot pods carry the
reloaderannotation and mount thezot-tlsSecret.kubectl --namespace zot-system get statefulset zot --output yaml \
| grep --extended-regexp 'reloader.stakater.com/reload|zot-tls' -
Confirm that the
registry-connectpack references the new CA certificate through thezot-tlsSecret and itsca.crtkey. Replace<cluster-uid>with the unique identifier of your cluster.kubectl --namespace cluster-<cluster-uid> get configmap \
--selector app.kubernetes.io/name=registry-connect --output yaml \
| grep --after-context=2 certificates
The registry endpoint at https://<node-ip>:30003 continues to serve traffic throughout the upgrade. The Zot pods
present the new certificate after the reloader-driven restart completes.
Roll Back the Upgrade
If you need to revert the upgrade, follow these steps.
-
Revert the cluster profile to the previous version. This version uses
zotversion0.1.89-rev1,registry-connectversion0.2.0, and the Zot Internal Registry preset. -
Re-apply the previous profile version to the cluster.
-
Because the earlier
zot-secretSecret and itscaCrtkey are still populated with the pre-upgrade CA, workloads that were trusting it continue to work until the certificate would have naturally expired.
Do not delete the root-ca-secret or zot-tls Kubernetes Secrets during the rollback. Leaving them in place allows a
forward re-upgrade later without cert-manager having to reissue the CA.
The zot-pvc PVC is unchanged by both the upgrade and the rollback, so no image data is lost.
Next Steps
After the upgrade completes, cert-manager renews the Zot server certificate automatically. To change how images are pulled from the registry, or to review the primary registry configuration, refer to the following pages.