Skip to main content
Version: latest

VMware

The following are some architectural highlights of Kubernetes clusters provisioned by Palette on VMware:

  • Kubernetes nodes can be distributed across multiple-compute clusters, which serve as distinct fault domains.

  • Support for static IP as well as DHCP. If your are using DHCP, Dynamic DNS is required.

  • IP pool management for assigning blocks of IPs dedicated to clusters or projects.

  • A Private Cloud Gateway (PCG) that you set up within the environment facilitates communications between the Palette management platform and vCenter installed in the private data center.

    The PCG is Palette's on-prem component to enable support for isolated, private cloud, or data center environments. When the PCG is installed on-prem, it registers itself with Palette and enables secure communications with the private cloud environment.

    vmware_arch_oct_2020.webp

    Refer to the PCG Architecture section to learn more about the PCG architecture.

Zone Tagging

You can use tags to create node zones and regions for your Kubernetes clusters. The node zones and regions can be used to dynamically place Kubernetes workloads and achieve higher availability. Kubernetes nodes inherit the zone and region tags as Labels. Kubernetes workloads can use the node labels to ensure that the workloads are deployed to the correct zone and region.

The following is an example of node labels that are discovered and inherited from vSphere tags. The tag values are applied to Kubernetes nodes in vSphere.

topology.kubernetes.io/region=usdc topology.kubernetes.io/zone=zone3 failure-domain.beta.kubernetes.io/region=usdc
failure-domain.beta.kubernetes.io/zone=zone3
info

To learn more about node zones and regions, refer to the Node Zones/Regions Topology section of the Cloud Provider Interface documentation.

Zone tagging is required to install Palette and is helpful for Kubernetes workloads deployed in vSphere clusters through Palette if they have persistent storage needs. Use vSphere tags on data centers and compute clusters to create distinct zones in your environment. You can use vSphere Tag Categories and Tags to create zones in your vSphere environment and assign them to vSphere objects.

The zone tags you assign to your vSphere objects, such as a datacenter and clusters are applied to the Kubernetes nodes you deploy through Palette into your vSphere environment. Kubernetes clusters deployed to other infrastructure providers, such as public cloud may have other native mechanisms for auto discovery of zones.

For example, assume a vCenter environment contains three compute clusters, cluster-1, cluster-2, and cluster-3. To support this environment you create the tag categories k8s-region and k8s-zone. The k8s-region is assigned to the datacenter, and the k8s-zone tag is assigned to the compute clusters.

The following table lists the tag values for the data center and compute clusters.

vSphere ObjectAssigned NameTag CategoryTag Value
Datacenterdc-1k8s-regionregion1
Clustercluster-1k8s-zoneaz1
Clustercluster-2k8s-zoneaz2
Clustercluster-3k8s-zoneaz3

Create a tag category and tag values for each datacenter and cluster in your environment. Use the tag categories to create zones. Use a name that is meaningful and that complies with the tag requirements listed in the following section.

Tag Requirements

The following requirements apply to tags:

  • A valid tag must consist of alphanumeric characters.

  • The tag must start and end with an alphanumeric characters.

  • The regex used for tag validation is (([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?

vSphere Permissions

The vSphere user account that deploys Palette require access to the following vSphere objects and permissions listed in the following table. Review the vSphere objects and privileges required to ensure each role is assigned the required privileges.

Spectro Root Role Privileges

The Spectro root role privileges are only applied to root objects and data center objects. Select the tab for the vSphere version you are using to view the required privileges for the spectro root role.

vSphere ObjectPrivilege
CNSSearchable
DatastoreBrowse datastore
HostConfiguration
Storage partition configuration
vSphere TaggingCreate and edit vSphere tags
NetworkAssign network
SessionsValidate session
VM Storage PoliciesView VM storage policies
Storage viewsView
warning

If the network is a Distributed Port Group under a vSphere Distributed Switch (VDS), ReadOnly access to the VDS without “Propagate to children” is required.

Spectro Role Privileges

As listed in the table, apply spectro role privileges to vSphere objects you intend to use for Palette installation. A separate table lists Spectro role privileges for VMs by category.

During the installation, images and Open Virtual Appliance (OVA) files are downloaded to the folder you selected. These images are cloned from the folder and applied VMs that deployed during the installation.

Select the tab for the vSphere version you are using to view the required privileges for the spectro role.

vSphere ObjectPrivileges
CNSSearchable
DatastoreAllocate space
Browse datastore
Low-level file operations
Remove file
Update VM files
Update VM metadata
FolderCreate Folder
Delete folder
Move folder
Rename folder
HostLocal operations: Reconfigure VM
NetworkAssign network
ResourceApply recommendation
Assign VM to resource pool
Migrate powered off VM
Migrate powered on VM
Query vMotion
SessionsValidate sessions
Storage policiesView access for VM storage policies is required.
Ensure StorageProfile.View is available.
spectro-templatesRead only. This is the vSphere folder created during the install. For airgap installs, you must manually create this folder.
Storage viewsView
TasksCreate task
Update task
vAppImport
View OVF environment
Configure vAPP application
Configure vApp instance
vSphere taggingAssign or Unassign vSphere Tag
Create vSphere Tag
Delete vSphere Tag
Edit vSphere Tag

The following table lists spectro role privileges for VMs by category. All privileges are for the vSphere object, Virtual Machines.

CategoryPrivileges
Change ConfigurationAcquire disk lease
Add existing disk
Add new disk
Add or remove device
Advanced configuration
Change CPU count
Change memory
Change settings
Change swapfile placement
Change resource
Change host USB device
Configure raw device
Configure managedBy
Display connection settings
Extend virtual disk
Modify device settings
Query fault tolerance compatibity
Query unowned files
Reload from path
Remove disk
Rename
Reset guest information
Set annotation
Toggle disk change tracking
Toggle fork parent
Upgrade VM compatibility
Edit InventoryCreate from existing
Create new
Move
Register
Remove
Unregister
Guest OperationsAlias modification
Alias query
Modify guest operations
Invoke programs
Queries
InteractionConsole Interaction
Power on/off
ProvisioningAllow disk access
Allow file access
Allow read-only disk access
Allow VM download
Allow VM files upload
Clone template
Clone VM
Create template from VM
Customize guest
Deploy template
Mark as template
Mark as VM
Modify customization specification
Promote disks
Read customization specifications
Service ConfigurationAllow notifications
Allow polling of global event notifications
Manage service configurations
Modify service configurations
Query service configurations
Read service configurations
Snapshot ManagementCreate snapshot
Remove snapshot
Rename snapshot
Revert to snapshot
Sphere ReplicationConfigure replication
Manage replication
Monitor replication
vSANCluster: ShallowRekey

Create a VMware Cloud Account

Use the following steps to create a VMware cloud account.

Prerequisites

  • A PCG is installed and available in the VMware environment. Refer to the Deploy to VMware vSphere guide to learn how to deploy a PCG.

In addition to the default cloud account already associated with the private cloud gateway, new user cloud accounts can be created for the different vSphere users.

PropertyDescription
Account NameCustom name for the cloud account
Private cloud gatewayReference to a running cloud gateway
vCenter ServerIP or FQDN of the vCenter server
UsernamevCenter username
PasswordvCenter password
warning

If you change the password for a user account in vCenter, you must also change it in Palette for the same VMware cloud account. We recommend updating the passwords immediately to avoid potentially locking Palette out of vCenter. For guidance, refer to Change VMware Cloud Account Password in Palette.

Change VMware Cloud Account Password

The user account password in vCenter must match the password for the corresponding VMware cloud account in Palette. This section provides steps to change the password in Palette in the event the vCenter password changes.

Prerequisites

  • Access to the vCenter credentials.

Change the Password in Palette

  1. Log in to Palette.

  2. From the Menu Menu navigate to Tenant Settings > Cloud Accounts.

  3. Click the three-dot Menu for the VMware account you want to update, and select Edit.

  4. In the window that opens, update the password in the Password field and click the Validate button.

  5. Confirm your changes.

Validation

Palette validates the password. Incorrect credentials will result in an error. As an extra precaution, try scaling a cluster up or down.

info

In addition to changing the password for a VMware account, Palette provides a way for you to also change the user associated with an account by entering a new username in the Username field. Ensure the new user account has the same permissions as the previous user account in vCenter.

Deploy a VMware Cluster

Prerequisites

  • A PCG is installed and available in the VMware environment. Refer to the Deploy to VMware vSphere guide to learn how to deploy a PCG.

Use the following steps to provision a new VMware cluster.

Deploy Cluster

  1. Provide the basic cluster information like Name, Description, and Tags. Tags are currently not propagated to the Virtual Machines (VMs) deployed on the cloud/data center environments.

  2. Select a Cluster Profile created for the VMware environment. The profile definition will be used as the cluster construction template.

  3. Review and override Pack Parameters as desired. By default, parameters for all Packs are set with values defined in the Cluster Profile.

  4. Provide a vSphere Cloud account and placement information.

    ParameterDescription
    Cloud AccountSelect the desired cloud account.
    VMware cloud accounts with credentials need to be preconfigured
    in the Project Settings section. An account is auto-created as
    part of the cloud gateway setup and is available for
    provisioning of Tenant Clusters if permitted by the administrator.
    DatacenterThe vSphere data center where the cluster nodes will be launched.
    Deployment FolderThe vSphere VM Folder where the cluster nodes will be launched.
    Image Template FolderThe vSphere folder to which the Spectro templates are imported.
    SSH Keys (Optional)Public key to configure remote SSH access to the nodes (User: spectro).
    NTP Server (Optional)Setup time synchronization for all the running nodes.
    IP Allocation strategyDHCP or Static IP
  5. Configure the control plane and worker node pools. Fill out the input fields in the Add node pool page. The following table contains an explanation of the available input parameters.

Control Plane Pool

ParameterDescription
NameA descriptive name for the node pool.
SizeNumber of VMs to be provisioned for the node pool. For the control plane pool, this number can be 1, 3, or 5.
Allow worker capabilitySelect this option for allowing workloads to be provisioned on control plane nodes.
LabelsAdd a label to apply placement constraints on a pod, such as a node eligible for receiving the workload.
TaintsTo set toleration to pods and allow (but do not require) the pods to schedule onto nodes with matching taints.
Instance typeSelect the compute instance type to be used for all nodes in the node pool.
Availability ZonesChoose one or more availability zones. Palette provides fault tolerance to guard against hardware failures, network failures, etc., by provisioning nodes across availability zones if multiple zones are selected.
Disk SizeGive the required storage size

Worker Pool

ParameterDescription
NameA descriptive name for the node pool.
Enable AutoscalerYou can enable the autoscaler by toggling the Enable Autoscaler button. Autoscaler scales resources up and down between the defined minimum and maximum number of nodes to optimize resource utilization.
Set the scaling limit by setting the Minimum Size and Maximum Size, as per the workload the number of nods will scale up from minimum set value to maximum set value and the scale down from maximum set value to minimum set value
SizeNumber of VMs to be provisioned for the node pool.
Rolling UpdateRolling update has two available options. The expand option launches a new node first, then shuts down old one. The contract option shuts down a old one first, then launches new one.details.
LabelsAdd a label to apply placement constraints on a pod, such as a node eligible for receiving the workload.
TaintsTo set toleration to pods and allow (but do not require) the pods to schedule onto nodes with matching taints.
Instance typeSelect the compute instance type to be used for all nodes in the node pool.
Availability ZonesChoose one or more availability zones. Palette provides fault tolerance to guard against hardware failures, network failures, etc., by provisioning nodes across availability zones if multiple zones are selected.
Disk SizeProvide the required storage size
  1. Review settings and deploy the cluster. Provisioning status with details of ongoing provisioning tasks is available to track progress.
info

New worker pools may be added if it is desired to customize certain worker nodes to run specialized workloads. As an example, the default worker pool may be configured with 4 CPUs, 8 GB of memory for general-purpose workloads, and another worker pool with 8 CPUs, 16 GB of memory for advanced workloads that demand larger resources.

Delete a VMware Cluster

The deletion of a VMware cluster results in the removal of all Virtual machines and associated storage disks created for the cluster. The following tasks need to be performed to delete a VMware cluster:

  1. Select the cluster to be deleted from the Cluster View page and navigate to the Cluster Overview page.

  2. Invoke the delete action available on the page: Cluster > Settings > Cluster Settings > Delete Cluster.

  3. Click Confirm to delete.

The Cluster Status is updated to Deleting while the Cluster Resources are being deleted. Provisioning status is updated with the ongoing progress of the delete operation. Once all resources are successfully deleted, the Cluster Status changes to Deleted and is removed from the list of Clusters.

info

The Delete action is only available for Clusters that are fully provisioned. For Clusters that are still in the process of being provisioned, Abort action is available to stop provisioning and delete all resources.

Force Delete a Cluster

A cluster stuck in the Deletion state can be force deleted by the user through the User Interface. The user can go for a force deletion of the cluster, only if it is stuck in a deletion state for a minimum of 15 minutes. Palette enables cluster force delete from the Tenant Admin and Project Admin scope.

  1. Log in to the Palette Management Console.

  2. Navigate to the Cluster Details page of the cluster stuck in deletion mode.

    • If the deletion status is stuck for more than 15 minutes, click the Force Delete Cluster button from the Settings dropdown.

    • If the Force Delete Cluster button is not enabled, wait for 15 minutes. The Settings dropdown will give the estimated time for the auto-enabling of the Force Delete button.

warning

If there are any cloud resources still on the cloud, the user should cleanup those resources before going for the force deletion.