Pack Constraints

Pack constraints are a set of rules defined at the pack level to validate the packs for a Profile or a Cluster before it gets created or updated. Packs must be validated before the cluster is submitted to ensure a successful deployment.

info

You can find information about the JSON schema for the pack metadata file in the JSON schema section of the documentation.

Pack Values Constraints

A Spectro Pack currently supports various configurations through a configuration file called values.yaml. The values defined in the config file are applied while deploying the Kubernetes cluster. The values defined in the pack are default values and can be overridden in the Cluster Profile or during the Cluster deployment.

Since the default pack values can be overridden, users may inadvertently set incorrect values leading to cluster deployment failure. These failures can occur at any point during the cluster deployment process. If the system is capable of detecting invalid pack values before the cluster is submitted for deployment, then deployment failures can be overcome to some extent.

Pack value constraints are additional information provided through a template file called schema.yaml in the pack. They define the schema format of the pack values. The pack constraints framework auto-checks for any schema constraints defined in the pack and validates the pack values. This checking occurs while creating or updating Cluster Profiles and Clusters.

Schema Constraints

Every schema constraint consists of a key name and the schema template. The key name must be the complete path of the parameter which is defined in the config file.

Required

Defines whether the pack value is optional or required.

registry.hostname:
 schema: '{{ required }}'

Read only

The pack value is not editable if marked as read only.

registry.hostname:
 schema: '{{ readonly }}'

Format

Defines the pack value format: the value is valid only when the value format matches the format defined in the pack.

Format Syntax

A format template consists of one or more format types along with the optional regex and number range values.

registry.hostname:
  schema: '{{ required | format "${FORMAT_TYPE:/REGEX/ OR [NUMBER RANGE] OR [LIST_OPTIONS]}" }}'

warning

The syntax of the regex accepted is the same general syntax used by Perl, Python, and other languages. More precisely, it is the syntax accepted by RE2 and described here.

Format Types

string

The string format type checks if the input value is a string and supports the regex in the template. If regex is specified in the template then the input value must match the regex.

registry.hostname:
  schema: '{{ format "${string}" }}'
registry.hostname:
  schema: '{{ format "${string:/^([a-z0-9].*)$/}" }}'

number

The number format type checks if the input value is a number, and supports the regex and the number range in the template.

registry.port:
  schema: '{{ format "${number}" }}'
registry.port:
  schema: '{{ format "${number:[5000-5005]}" }}'
registry.port:
  schema: '{{ format "${number:/^(500[0-5])$/}" }}'

boolean

The bool format type checks if the input value is true or false.

registry.private:
  schema: '{{ format "${boolean}" }}'

password

The password format is a string type with masked values in the pack parameters of Cluster profiles and Clusters.

registry.password:
  schema: '{{ format "${password}" }}'
registry.password:
  schema: '{{ format "${password:/^([a-z0-9].*)$/}" }}'

list

The list format checks if the input value matches with any of the options specified in the template.

registry.type:
 schema: '{{ format "${list:[PACK,GIT,CHART]}" }}'

ipv4

The ipv4 format type checks if the input value is a valid ipv4.

registry.hostIp:
  schema: '{{ format "${ipv4}" }}'

version

The version format type checks if the input value is a semantic version.

registry.version:
  schema: '{{ required | format "${version}" }}'

hints

Hints are optional short descriptions of the parameter. If defined in the schema template, these descriptions are visible in the UI while configuring the pack parameters in the Profile or the Cluster. One or more descriptions can be combined by using the pipe(|) separator.

registry.type:
 schema: '{{ hints "description A" "description B" }}'

Examples

Schema constraints can be combined to support multiple validations using a single template.

IP Range

registry.addresses.$[]:
 schema: '{{ required | format "${ipv4} - ${ipv4}" | hints "ip pool range"}}'

registry.addresses.$[] is an array data type in the config file. The schema template defines that the value is required and the format must match - ${ipv4} - ${ipv4}

Examples:

10.10.10.10 - 10.10.10.255 → valid

10.10.10.10 → invalid

10.10.10.10-10.10.10.255 → invalid

String regex

storageType:
 schema: '{{ required | format "${string}, ${string:/size=\d+/}" }}'

Examples:

type-zeroedthick, size=150 → valid

type-zeroedthick, size=150 → invalid

type-zeroedthick, size=s → invalid

Pack Dependency Constraints

Spectro Cloud provides the flexibility to choose any pack of any version in the profile. Clusters are deployed based on the packs selected in the profile. While this works for most of the cases, it is sometimes required to select a minimum or maximum pack version, or to have dependencies between the packs to ensure the Kubernetes cluster is deployed successfully as desired.

Pack dependency constraints are the rules defined in the pack metadata file pack.json. They are used to define the minimum and maximum supported versions, and also to specify which pack is required or not supported. The pack constraints framework auto-checks for any schema constraints defined in the pack and validates the pack values. This checking occurs while creating or updating Cluster Profiles and Clusters.

Pack metadata JSON

Pack dependency constraints must be defined in the pack.json file. The sample pack metadata shown below defines the dependencies under constraints key.

{
  "addonType": "system app",
  "cloudTypes": ["all"],
  "displayName": "Test Pack",
  "kubeManifests": [],
  "layer": "addon",
  "name": "pack-constraints-test",
  "version": "1.0.0",
  "constraints": {
    "dependencies": [
      {
        "packName": "vault",
        "layer": "addon",
        "minVersion": "0.6.0",
        "maxVersion": "",
        "type": "optional"
      },
      {
        "packName": "csi-vsphere-volume",
        "layer": "csi",
        "minVersion": "1.0.0",
        "maxVersion": "",
        "type": "notSupported"
      },
      {
        "packName": "kubernetes",
        "layer": "k8s",
        "minVersion": "1.17.0",
        "maxVersion": "1.18.6",
        "type": "required"
      }
    ]
  }
}

warning

If the minimum and maximum versions are not mentioned, the validation is skipped.

Pack Dependency Attributes

A pack can have one or more dependencies defined in the dependencies array. Each dependency consists of the following attributes.

Attribute	Description
packName	Name of the dependent pack.
layer	The layer type of the dependent pack. Refer to the Layer Types section to learn more.
minVersion	Minimum supported dependent pack version, any version below the minimum version is not valid.
maxVersion	Maximum supported dependent pack version, any version above the maximum version is not valid.
type	The defined type for the dependency. Refer to the Dependency Types section to learn more.

In the example code snippet from earlier, the three dependent packs are identified by unique pack names such as vault, csi-vsphere-volume, and kubernetes. A minVersion, maxVersion, and type are defined for each dependent pack.

"dependencies": [
    {
      "packName": "vault",
      "layer": "addon",
      "minVersion": "0.6.0",
      "maxVersion": "",
      "type": "optional"
    },
    {
      "packName": "csi-vsphere-volume",
      "layer": "csi",
      "minVersion": "1.0.0",
      "maxVersion": "",
      "type": "notSupported"
    },
    {
      "packName": "kubernetes",
      "layer": "k8s",
      "minVersion": "1.17.0",
      "maxVersion": "1.18.6",
      "type": "required"
    }
  ]

Layer Types

The layer attribute defines the layer where the dependent pack can be found in the Cluster Profile. The following table lists the different layer types.

Layer	Description
os	The dependent pack can only be found in the operating system layer of the Cluster Profile. The os layer contains packs such as Ubuntu, CentOS or Bring Your Own OS (BYOOS).
k8s	The dependent pack can only be found in the Kubernetes layer of the Cluster Profile. The k8s layer contains packs such as Palette eXtended Kubernetes (PXK), RKE2 or k3s.
cni	The dependent pack can only be found in the network layer of the Cluster Profile. The cni layer contains packs such as Calico, Cilium, Flannel and Antrea.
csi	The dependent pack can only be found in the storage layer of the Cluster Profile. The csi layer contains packs such as vSphere CSI, Amazon EBS CSI, Amazon EFS, Azure Disk and Portworx.
addon	The dependent pack can only be found in the add-on layers of the Cluster Profile. The types of packs include logging, monitoring, load balancer, authentication, ingress, security, ai, app services, network, storage, registry, servicemesh, and system app. The addon layer contains packs such as ArgoCD, Vault, Nginx, and many more.

Dependency Types

The type attribute defines the type of dependency. The following table lists the different dependency types.

Type	Description
optional	The dependent pack is optional but validates minimum or maximum versions if the pack is selected. In the example, the vault pack is optional.
required	The dependent pack is mandatory and must contain a version within the minimum or maximum supported versions, if defined. In the example, the kubernetes pack is required with a minimum version of 1.17.0 and a max version of 1.18.6. Any Kubernetes version below 1.17.0 and above 1.18.6 is not valid.
notSupported	The pack versions within the range of the mentioned minimum and maximum are not supported. The csi-vsphere-volume pack is not supported if the version selected falls within the min and max versions.

Pack Resource Constraints

A successful Kubernetes Cluster deployment is possible only when the cluster has sufficient hardware requirements. We consider the CPU, Memory, and Disk size as the hardware requirements. The minimum resource requests can be varied depending on the workload to be deployed in the cluster. Spectro Cloud users are allowed to select the desired instance type, and the disk size while configuring the machine pool in the Cluster deployment procedure. If the user selects the instance type which does not satisfy the minimum CPU or Memory or Disk size requirements, then there is a high probability that the cluster deployment may not succeed due to insufficient CPU or Memory or Disk size.

Pack Resource Constraints are a set of rules defined in the pack metadata pack.json to specify the minimum CPU, Memory, and Disk size requirements. The pack constraints framework auto-checks the resource constraints and validates the user-selected instance type specifications before the cluster is submitted for deployment. The total input resource capacity is evaluated against the machine pool size with the actual hardware specifications of a selected instance type.

Pack metadata JSON

Pack resource constraints must be defined in the pack.json file. The sample pack metadata is shown below to define the resources under constraints key.

{
  "addonType": "system app",
  "cloudTypes": ["all"],
  "displayName": "Test Pack",
  "kubeManifests": [],
  "layer": "addon",
  "name": "pack-constraints-test",
  "version": "1.0.0",
  "constraints": {
    "resources": [
      {
        "type": "cpu",
        "minLimit": 2000,
        "components": [
          {
            "resourceRequestParamRef": "requests.cpu",
            "replicaCountParamRef": "replicas",
            "scheduleType": "all"
          }
        ]
      },
      {
        "type": "memory",
        "minLimit": 2048,
        "components": [
          {
            "resourceRequestParamRef": "requests.memory",
            "replicaCountParamRef": "replicas",
            "scheduleType": "worker"
          }
        ]
      },
      {
        "type": "diskSize",
        "minLimit": 10
      }
    ]
  }
}

Pack Resources Attributes

type

The type of resource

• CPU
• Memory
• Disk size

minimum limit

The minimum limit of the resource will be considered during the machine pool validation. The resource limit value is required to have the below unit depending on the resource type. Any change of unit will cause inaccurate computation of the total minimum requirement.

• CPU - Millicore (m)
• Memory - Mibibyte (Mi)
• Disk size - Gigabyte (GB)

components

The minimum limit is the minimum resource requirement for each worker pool in the cluster. This value is sufficient for the basic resource validation, but in some cases where the pack contains one or more associated components, then each component can define its CPU or memory resource requests in the config file values.yaml. In this case, a single minLimit value is not sufficient as the minimum requirements can be different for each component.

info

If the components are defined then minLimit is ignored during resource validation.

The components field is an array of the component which consists of these attributes.

resourceRequestParamRef

Resource requests and limits can be defined in the pack values.yaml. It is required for the pack constraints framework to know the parameter name from where the resource request value can be read during the resource validation. So, the resourceRequestParamRef is the configuration parameter name of the resource request defined in the values.yaml.

replicaCountParamRef

The Kubernetes pod can run in one or more replicas based on the replica count configured in the values.yaml file. The resource request values defined in values.yaml are for one replica, and the requests must be multiplied by the number of replicas which gives the actual minimum requirement. So, the replicaCountParamRef is the configuration parameter name of the replica count defined in the values.yaml

scheduleType

Kubernetes provides a way to schedule the pods on the control plane and worker nodes. Pack Constraints framework must know where the pods are scheduled because the resource validation validates only the control plane machine pool when the pods are scheduled on control plane nodes. Similarly, if the pods are scheduled on worker nodes, then only the worker machine pool will be validated. In the case of daemon sets, the pods are scheduled in both control plane and worker nodes, and the framework validates both control plane and worker machine pool configurations before the cluster is submitted for deployment.

• control-plane - pods are scheduled only on control plane nodes
• worker - pods are scheduled only on worker nodes
• all - pods are scheduled on both control plane and worker nodes

Pack Presets

Pack Presets are the predefined values in a file called presets.yaml in the pack. It contains an array of the presets for the pack, and is visible in the pack parameters of the Cluster profile and the Cluster. Users can select any preset from the available pack presets, and the predefined values of a selected preset are applied automatically by the Spectro Cloud UI. Presets make pack configuration much easier as multiple pack values are updated at a time and the user does not need to understand all the configuration parameters which get changed depending on various factors.

Presets Metadata YAML

This presets.yaml shows two presets

• privatePackRegistry
• publicPackRegistry

with a different set of pre-defined values.

presets:
- name: "privatePackRegistry"
  displayName: "Private Registry"
  group: "registry"
  remove: ["registry.ingress.host"]
  add: |
    registry:
      private: true
      type: "PACK"
      host:
       ip: "127.0.0.1"
       port: 5000
- name: "publicPackRegistry"
  displayName: "Public Registry"
  group: "registry"
  remove: ["registry.ingress.host"]
  add: |
    registry:
      private: false
      type: "PACK"
      host:
       ip: "13.233.2.255"
       port: 80

Preset Attributes

name

Name of the preset. It must be unique.

displayName

Name of the preset. It is visible in the parameters configuration

remove

An array of parameter names. These are removed from the pack values when a preset is selected.

add

A set of values in YAML format. These are added/updated in the pack values when a preset is selected.

group

One or more presets can be categorized into a common group, but only one preset can be selected from the same group of presets.

Pack Readiness Checks

Palette marks a pack as ready when the resources that it deploys are ready. By default, Palette checks Deployments, ReplicaSets, StatefulSets, Jobs, LoadBalancer Services, Argo CD Applications, and Ingresses. The default checks do not cover custom resources. For example, Palette can mark an operator pack as ready while the CephCluster or LinstorCluster that the operator manages has not finished its setup.

Use the readinessCheck section of the pack values to make Palette wait until a custom resource reaches the state that you specify. A pack is ready only when the default checks and every readiness check pass. Packs with a higher install order value wait until the pack is ready.

Pack authors can add readinessCheck as a top-level key in the values.yaml file of a pack, at the same level as pack and charts. You can also add, change, or remove it in the YAML of a cluster profile layer, including Helm chart layers.

warning

In a Helm chart layer, Palette passes the readinessCheck section to Helm together with the chart values. If the chart includes a values.schema.json file that allows only the keys it defines, the chart installation fails because Helm rejects the values. In that case, remove readinessCheck from the layer. This does not affect packs, because Palette passes only the values under charts to Helm.

The following example shows the structure of a readiness check.

readinessCheck:
  v1beta1:
    - name: "<resource-name>"
      namespace: "<resource-namespace>"
      group: "<api-group>"
      version: "<api-version>"
      kind: "<resource-kind>"
      keysToCheck:
        - keyToCheck: "<field-path>"
          expectedValue: "<expected-value>"

Replace <resource-name>, <resource-namespace>, <api-group>, <api-version>, and <resource-kind> with the values that identify the resource to check. Replace <field-path> with the path to the field and <expected-value> with the value that the field must have. For details about each attribute, refer to Readiness Check Attributes.

Readiness Check Attributes

Attribute	Description
v1beta1	The list of resources to check. Each entry identifies one resource and the fields to check on it.
name	The name of the resource. You can use macros.
namespace	(Optional) The namespace of the resource. Set it for every namespaced resource, and omit it for cluster-scoped resources. If you omit it for a namespaced resource, Palette looks in cluster-<cluster-uid>, the namespace of the Pack resource, not in the namespace that the pack installs into.
group	The API group of the resource, such as ceph.rook.io.
version	The API version of the resource, such as v1.
kind	The kind of the resource, such as CephCluster.
keysToCheck	The list of fields to check. Each item has a keyToCheck and an expectedValue. Each entry must have at least one field to check.
keyToCheck	The path to the field in the resource, such as status.phase or status.conditions[0].status. Do not include a leading $..
expectedValue	The value that the field must have for the check to pass. Refer to Expected Values.

The readinessCheck section also accepts a timeout attribute, but Palette ignores it. Palette keeps checking the pack until it is ready.

Older packs might set a single keyToCheck and expectedValue directly on an entry, instead of in keysToCheck. Palette still supports this form and checks the pair together with any items in keysToCheck. Use keysToCheck for new packs.

How Palette Evaluates Readiness Checks

1. Palette fetches the resource by its group, version, kind, name, and namespace. If the resource does not exist yet, Palette reports an error for the pack with the reason ServiceReconcileFailed and tries again.

2. Palette reads each keyToCheck from the resource, converts the value to text, and compares it to expectedValue. The comparison is case-sensitive. If a field does not exist yet, Palette also reports an error and tries again.

3. An entry passes when every key matches. The pack is ready when every entry and the default checks pass.

4. Until the pack is ready, Palette checks it again every 30 seconds.

Palette shows the installation status of each pack in a cluster. For more information, refer to Pack Monitoring.

In the cluster, each pack has a matching Pack resource in the cluster-<cluster-uid> namespace. Palette writes the results of the checks to the status.conditions and status.servicesStatus fields of that resource. Palette records a Warning event on the Pack resource only when a check returns an error, such as a missing resource or field. The event reason is ServiceReconcileFailed or ReconcileError, and the event also appears on the Events tab of the cluster in Palette.

To view the Pack resource, use the following command with the kubeconfig file of the cluster.

kubectl get pack <pack-name> --namespace cluster-<cluster-uid> --output yaml

Replace <pack-name> with the name of the pack and <cluster-uid> with the unique identifier of your cluster.

Expected Values

Palette converts the field value to text before the comparison, so write expectedValue as that text.

Field Value in the Resource	Text That Palette Compares	Matching expectedValue
A boolean, such as ready: true	true	"true"
A condition status, such as status: "True"	True	"True"
A number, such as replicas: 3	3	"3"
A string, such as phase: Available	Available	"Available"

If a pack never becomes ready, check that each expectedValue matches the field value exactly, including capitalization. For example, expectedValue: "True" never matches a boolean field whose value is true.

To check a field in a list, use an index key, such as status.conditions[0].status for the status of the first condition. An index key depends on the order of the items in the list, so confirm the order in the resource before you use it. Avoid filter expressions, such as status.conditions[?(@.type=="Ready")].status. A filter returns a list instead of a single value, so the comparison depends on how the list converts to text.

Readiness Check Examples

The following example waits until a Rook CephCluster reports the Ready phase.

readinessCheck:
  v1beta1:
    - name: "rook-ceph"
      namespace: "rook-ceph"
      group: "ceph.rook.io"
      version: "v1"
      kind: "CephCluster"
      keysToCheck:
        - keyToCheck: "status.phase"
          expectedValue: "Ready"

The following example waits until the first three conditions of a Piraeus LinstorCluster are True. LinstorCluster is a cluster-scoped resource, so the example omits namespace.

readinessCheck:
  v1beta1:
    - name: "linstorcluster"
      group: "piraeus.io"
      version: "v1"
      kind: "LinstorCluster"
      keysToCheck:
        - keyToCheck: "status.conditions[0].status"
          expectedValue: "True"
        - keyToCheck: "status.conditions[1].status"
          expectedValue: "True"
        - keyToCheck: "status.conditions[2].status"
          expectedValue: "True"

Pack Macros

Pack macros are the variables defined in the Cluster profile or in Cluster pack values, and these variables are resolved only at the cluster deployment time.

Types of Macros

System Macros

System macros are variables defined by the system. Users are allowed to use these variables and the system is capable of resolving all the variables to values at the time of cluster deployment.

Syntax

user:
 name: "{{ .spectro.system.[VARIABLE_NAME] }}"

Supported Variables

Macro	Description
{{.spectro.system.user.name}}	The name of the user currently logged in.
{{.spectro.system.user.uid}}	The unique identifier of the user currently logged in.
{{.spectro.system.user.email}}	The email address of the user currently logged in.
{{.spectro.system.tenant.uid}}	The unique identifier of the current tenant.
{{.spectro.system.project.name}}	The name of the project.
{{.spectro.system.project.uid}}	The unique identifier of the project.
{{.spectro.system.clusterprofile.name}}	The name of the cluster profile associated with the current project.
{{.spectro.system.clusterprofile.uid}}	The unique identifier of the cluster profile the pack is part of.
{{.spectro.system.clusterprofile.version}}	The current version of the cluster profile the pack is part of.
{{.spectro.system.cluster.name}}	The name of the cluster.
{{.spectro.system.cluster.uid}}	The unique identifier of the cluster.
{{.spectro.system.cloudaccount.name}}	The name of the cloud account associated with the current project.
{{.spectro.system.cloudaccount.uid}}	The unique identifier of the cloud account associated with the current project.
{{.spectro.system.kubernetes.version}}	The version of Kubernetes currently running on the cluster.
{{.spectro.system.reverseproxy.server}}	The hostname of the reverse proxy server.
{{.spectro.system.reverseproxy.port}}	The port number of the reverse proxy server.
{{.spectro.system.reverseproxy.protocol}}	The protocol used by the reverse proxy server, either HTTP or HTTPS.
{{.spectro.system.reverseproxy.vhostport}}	The port number used by the virtual host on the reverse proxy server.
{{.spectro.system.cloud.type }}	The type of cloud provider being used, such as AWS, GCP, Azure or other providers.
{{.spectro.system.cloud.region }}	The region where the cloud resources are located.
{{.spectro.system.clusterprofile.infra.name}}	The name of the cluster profile.
{{.spectro.system.clusterprofile.infra.uid}}	The unique identifier of the cluster profile.
{{.spectro.system.clusterprofile.infra.version}}	The version of the cluster profile.
{{.spectro.system.cluster.kubevip}}	The IP address of the virtual IP (VIP) assigned to the cluster and load balancer for the control plane. This macro is only available for Edge and vSphere cluster deployments.

Examples

user:
 name: "{{ .spectro.system.user.name }}"
 uid: "{{ .spectro.system.user.uid}}"

Pack Reference Macros

Pack reference macros are custom variables that must be defined in a pack and then can be used as a variable in any pack. If the variable is not defined with a value, then the default value is applied, if specified. If the default value is not specified, then the variable will be resolved to an empty value.

Syntax

k8s:
 version: "{{ .spectro.pack.[PACK_NAME].[VARIABLE_NAME] }}"

PACK_NAME - the name of the pack where the variable is defined

VARIABLE_NAME - the fully qualified name of the variable defined in the pack

Examples

Referencing Kubernetes pack variable version in CentOS pack values:

CentOS values.yaml

k8s:
 version: "{{ .spectro.pack.kubernetes.version }}"

Kubernetes values.yaml

version: 1.18.0

Additional Capabilities

Sprig Template Functions

Users are allowed to use the sprig template functions to modify the resolved variable value.

Examples

user:
 name: "{{ .spectro.system.user.name | upper }}"

upper - sprig template function which converts resolved user name to upper case

How to set the default value?

k8s:
 version: "{{ .spectro.pack.kubernetes.version | default \"1.19.0\"}}"

info

If the variable version is not defined in the pack kubernetes, then the default value 1.19.0 will be applied at deployment. In case the default value is not specified then the empty value will be applied.