Overview

The following is the detailing of the Microsoft Azure Kubernetes Service (AKS) cluster provisioning by Palette:

  1. The Palette platform enables containerized applications' effortless deployment and management with fully-managed AKS.
  1. It provides the users with server-less Kubernetes, an integrated continuous integration and continuous delivery (CI/CD) experience, and enterprise-grade security and governance.
  1. This unites the development and operations to a single platform achieving faster build, delivery, and scaling of applications with credence.
  1. The infrastructure has event-driven autoscaling and triggers that enable Elastic provisioning for this self-managed infrastructure.
  1. Leverage extensive authentication and authorization capabilities, using Azure Active Directory and dynamic rules enforcement, across multiple clusters with Azure Policy.

aks_cluster_architecture.png

Prerequisites

These prerequisites must be met before deploying an AKS workload cluster:

  1. You need an active Azure cloud account with sufficient resource limits and permissions to provision compute, network, and security resources in the desired regions.
  1. You will need to have permissions to deploy clusters using the AKS service on Azure.
  1. Register your Azure cloud account in Palette as described in the Creating an Azure Cloud Account section below.
  1. You should have a cluster profile created in Palette for AKS.
  1. Associate an SSH key pair to the cluster worker node.

Additional Prerequisites

There are additional prerequisites if you want to set up Azure Active Directory integration for the AKS cluster:

  1. A Tenant Name must be provided as part of the Azure cloud account creation in Palette.
  1. For the Azure client used in the Azure cloud account, these API permissions have to be provided:

    Microsoft GraphGroup.Read.All (Application Type)
    Microsoft GraphDirectory.Read.All (Application Type)
  2. You can configure these permissions from the Azure cloud console under App registrations > API permissions for the specified application.

Palette also enables the provisioning of private AKS clusters via a private cloud gateway (Self Hosted PCGs). The Self-Hosted PCG is an AKS cluster that needs to be launched manually and linked to an Azure cloud account in Palette Management Console. Click here for more..

Creating an Azure Cloud Account

To create an Azure cloud account, we need:

A custom Account Name

  • Client ID
  • Tenant ID
  • Client Secret
  • Tenant Name (optional)
  • Toggle Connect Private Cloud Gateway option and select the Self-Hosted PCG already created from the drop-down menu to link it to the cloud account.

Note:

For existing cloud account go to Edit and toggle the Connect Private Cloud Gateway option to select the created gateway from the drop down menu.

For Azure cloud account creation, we first need to create an Azure Active Directory (AAD) application that can be used with role-based access control. Follow the steps below to create a new AAD application, assign roles, and create the client secret:


  1. Follow the steps described here to create a new Azure Active Directory application. Note down your ClientID and TenantID.
  1. On creating the application, assign a minimum required ContributorRole. To assign any type of role, the user must have a minimum role of UserAccessAdministrator. Follow the Assign Role To Application link learn more about roles.
  1. Follow the steps described in the Create an Application Secret section to create the client application secret. Store the Client Secret safely as it will not be available as plain text later.

Deploying an AKS Cluster


The following steps need to be performed to provision a new cluster:


  1. If you already have a profile to use, go to the Cluster > Add a New Cluster > Deploy New Cluster and select an Azure cloud. If you do not have a profile to use, reference the Creating a Cluster Profile page for steps on how to create one.
  1. Fill the basic cluster profile information such as Name, Description, Tags and Cloud Account.
  1. In the Cloud Account dropdown list, select the Azure Cloud account or create a new one. See the Creating an Azure Cloud Account section above.
  1. Next, in the Cluster profile tab from the Managed Kubernetes list, pick AKS, and select the AKS cluster profile definition.
  1. Review the Parameters for the selected cluster profile definitions. By default, parameters for all packs are set with values defined in the cluster profile.
  1. Complete the Cluster config section with the information for each parameter listed below.

    ParameterDescription
    SubscriptionSelect the subscription which is to be used to access Azure Services.
    RegionSelect a region in Azure in where the cluster should be deployed.
    Resource GroupSelect the resource group in which the cluster should be deployed.
    SSH KeyPublic key to configure remote SSH access to the nodes.
    Static PlacementBy default, Palette uses dynamic placement, wherein a new VPC with a public and private subnet is created to place cluster resources for every cluster. These resources are fully managed by Palette and deleted when the corresponding cluster is deleted.
    Turn on the Static Placement option if it is desired to place resources into preexisting VPCs and subnets. If the user is making the selection of Static Placement of resources, the following placement information needs to be provided:
    Virtual Resource Group: The logical container for grouping related Azure resources.
    Virtual Network: Select the virtual network from dropdown menu.
    Control plane Subnet: Select the control plane network from the dropdown menu.
    Worker Network: Select the worker network from the dropdown.
    Update worker pools in parallelCheck the box to concurrently update the worker pools.

If the Palette cloud account is created with Disable Properties and with Static Placement the network informations from user's Azure account will not be imported to palette account. Hence user can manually input the information for the Control Plane Subnet and the Worker Network (no drop down menu will be available).

  1. Click Next to configure the node pools.

The maximum number of pods per node in an AKS cluster is 250. If you don't specify maxPods when creating new node pools, you receive a default value of 30. This value can be edited from the Kubernetes configuration file at any time by editing the maxPodPerNode value. Refer to the snippet below:

managedMachinePool:
maxPodPerNode: 30
#NOTE: The recommended minimum value set for maxPodPerNode is 30, setting value less than the recommended one can result in palette system pods in pending state.

Node Pools

This section guides you to through configuring Node Pools. As you set up the cluster, the Nodes config section will allow you to customize node pools. AKS Clusters are comprised of System and User node pools, and all pool types can be configured to use the Autoscaler, which scales out pools horizontally based on per node workload counts.

A complete AKS cluster contains the following:


  1. As a mandatory primary System Node Pool, this pool will run the pods necessary to run a Kubernetes cluster, like the control plane and etcd. All system pools must have at least a single node for a development cluster; one (1) node is enough for high availability production clusters, and three (3) or more is recommended.
  1. Worker Node pools consist of one (1) or more per workload requirements. Worker node pools can be sized to zero (0) nodes when not in use.

Creating and Removing Node Pools

During cluster creation, you will default to a single pool.


  1. To add additional pools, click Add Node Pool.
  1. Provide any additional Kubernetes labels to assign to each node in the pool. This section is optional, and you can use a key:value structure, press your space bar to add additional labels, and click the X with your mouse to remove unwanted labels.
  1. To remove a pool, click Remove across from the title for each pool.

Creating a System Node Pool

  1. Each cluster requires at least one (1) system node pool. To define a pool as a system pool, check the box labeled System Node Pool.
Identifying a Node Pool as a System Pool will deactivate taints, and the operating system options within the Cloud Configuration section, as you can not to taint or change their OS from Linux. See the AKS Documentation for more details on pool limitations.

  1. Provide a name in the Node pool name text box. When creating a node, it is good practice to include an identifying name that matches the node in Azure.
  1. Add the Desired size. You can start with three for multiple nodes.
  1. Include Additional Labels. This is optional.
  1. In the Azure Cloud Configuration section, add the Instance type. The cost details are present for review.
  1. Enter the Managed Disk information and its size.
  1. If you are including additional or multiple nodes to make a node pool, click the Add Worker Pool button to create the next node.

Configuring Node Pools

In all types of node pools, configure the following.


  1. Provide a name in the Node pool name text box. When creating a node, it is good practice to include an identifying name.

    Note: Windows clusters have a name limitation of six (6) characters.

  1. Provide how many nodes the pool will contain by adding the count to the box labeled Number of nodes in the pool. Configure each pool to use the autoscaler controller. There are more details on how to configure that below.
  1. Alternative to a static node pool count, you can enable the autoscaler controller, click Enable Autoscaler to change to the Minimum size and Maximum size fields which will allow AKS to increase or decrease the size of the node pool based on workloads. The smallest size of a dynamic pool is zero (0), and the maximum is one thousand (1000); setting both to the same value is identical to using a static pool size.
  1. Provide any additional Kubernetes labels to assign to each node in the pool. This section is optional; you can use a key:value structure. Press your space bar to add additional labels and click the X with your mouse to remove unwanted labels.
  1. In the Azure Cloud Configuration section:
  • Provide instance details for all nodes in the pool with the Instance type dropdown. The cost details are present for review.

New worker pools may be added if you want to customize specific worker nodes to run specialized workloads. As an example, the default worker pool may be configured with the Standard_D2_v2 instance types for general-purpose workloads, and another worker pool with the instance type Standard_NC12s_v3 can be configured to run GPU workloads.

  • Provide the disk type via the Managed Disk dropdown and the size in Gigabytes (GB) in the Disk size field.
A minimum allocation of two (2) CPU cores is required across all worker nodes.

A minimum allocation of 4Gi of memory is required across all worker nodes.


  • When are done setting up all node pools, click Next to go to the Settings page to Validate and finish the cluster deployment wizard.

    Note: Keep an eye on the Cluster Status once you click Finish Configuration as it will start as Provisioning. Deploying an AKS cluster does take a considerable amount of time to complete, and the Cluster Status in Palette will say Ready when it is complete and ready to use.


Deleting an AKS Cluster

The deletion of an AKS cluster results in the removal of all Virtual Machines and associated Storage Disks, created for the cluster. The following tasks need to be performed to delete an AKS cluster:


  1. Select the cluster to be deleted from the Cluster View page and navigate to the Cluster Overview page.
  1. Invoke a delete action available on the page: Cluster > Settings > Cluster Settings > Delete Cluster.
  1. Click Confirm to delete.

The Cluster Status is updated to Deleting while cluster resources are being deleted. Provisioning status is updated with the ongoing progress of the delete operation. Once all resources are successfully deleted, the cluster status changes to Deleted and is removed from the list of clusters.

Force Delete a Cluster

A cluster stuck in the Deletion state can be force deleted by the user through the User Interface. The user can go for a force deletion of the cluster, only if it is stuck in a deletion state for a minimum of 15 minutes. Palette enables cluster force delete from the Tenant Admin and Project Admin scope.

To force delete a cluster:

  1. Log in to the Palette Management Console.
  1. Navigate to the Cluster Details page of the cluster stuck in deletion mode.

    • If the deletion status is stuck for more than 15 minutes, click the Force Delete Cluster button from the Settings dropdown.

    • If the Force Delete Cluster button is not enabled, wait for 15 minutes. The Settings dropdown will give the estimated time for the auto-enabling of the Force Delete button.

If there are any cloud resources still on the cloud, you should clean up those resources before going for the force deletion.

Configuring an Azure Active Directory

The Azure Active Directory (AAD) could be enabled while creating and linking the Azure Cloud account for the Palette Platform, using a simple check box. Once the cloud account is created, you can create the Azure AKS cluster. The AAD-enabled AKS cluster will have its Admin kubeconfig file created and can be downloaded from our Palette UI as the 'Kubernetes config file'. You need to manually create the user's kubeconfig file to enable AAD completely. The following are the steps to create the custom user kubeconfig file:


  1. Go to the Azure console to create the Groups in Azure AD to access the Kubernetes RBAC and Azure AD control access to cluster resources.
  1. After you create the groups, create users in the Azure AD.
  1. Create custom Kubernetes roles and role bindings for the created users and apply the roles and role bindings, using the Admin kubeconfig file.

The above step can also be completed using Spectro RBAC pack available under the Authentication section of Add-on Packs.

  1. Once the roles and role bindings are created, these roles can be linked to the Groups created in Azure AD.
  1. The users can now access the Azure clusters with the complete benefits of AAD. To get the user-specific kubeconfig file, please run the following command:

az aks get-credentials --resource-group <resource-group> --name <cluster-name>


References:

Use Kubernetes RBAC with Azure AD integration

Azure Kubernetes Service (AKS)