Following are some architectural highlights of Amazon Web Services' (AWS) managed Kubernetes clusters (also known as Elastic Kubernetes Service or (EKS)), provisioned by Palette:
- Cluster resources such as Virtual Machines (VMs) can be provisioned into an existing infrastructure (Gateways, VPCs, Subnets etc.) as part of static provisioning as well as new dedicated infrastructure as part of dynamic provisioning.
- Full support for EKS Fargate profiles
Spot instance support
The following prerequisites must be met before deploying an EKS workload cluster:
- You must have an active AWS cloud account with all the permissions listed below in the AWS Cloud Account Permissions section.
- You must register your AWS cloud account in Palette as described in the Creating an AWS Cloud account section below.
- Have an Infrastructure cluster profile already created in Palette for EKS.
- Sufficient capacity in the desired AWS region should exist for the creation of the following resources:
- vCPU
- VPC
- Elastic IP
- Internet Gateway
- Elastic Load Balancers
- NAT Gateway
The following four policies include all the required permissions for provisioning clusters through Palette:
Controller Policy
Last Update: May 25, 2022
{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": ["ec2:AllocateAddress","ec2:AssociateRouteTable","ec2:AttachInternetGateway","ec2:AuthorizeSecurityGroupIngress","ec2:CreateInternetGateway","ec2:CreateNatGateway","ec2:CreateRoute","ec2:CreateRouteTable","ec2:ReplaceRoute","ec2:CreateSecurityGroup","ec2:CreateSubnet","ec2:CreateTags","ec2:CreateVpc","ec2:ModifyVpcAttribute","ec2:DeleteInternetGateway","ec2:DeleteNatGateway","ec2:DeleteNetworkInterface","ec2:DeleteRouteTable","ec2:DeleteSecurityGroup","ec2:DeleteSubnet","ec2:DeleteTags","ec2:DeleteVpc","ec2:DescribeAccountAttributes","ec2:DescribeAddresses","ec2:DescribeAvailabilityZones","ec2:DescribeInstances","ec2:DescribeInternetGateways","ec2:DescribeImages","ec2:DescribeKeyPairs","ec2:DescribeNatGateways","ec2:DescribeNetworkInterfaces","ec2:DescribeNetworkInterfaceAttribute","ec2:DescribeRouteTables","ec2:DescribeSecurityGroups","ec2:DescribeSubnets","ec2:DescribeVpcs","ec2:DescribeVpcAttribute","ec2:DescribeVolumes","ec2:DetachInternetGateway","ec2:DisassociateRouteTable","ec2:DisassociateAddress","ec2:ModifyInstanceAttribute","ec2:ModifyNetworkInterfaceAttribute","ec2:ModifySubnetAttribute","ec2:ReleaseAddress","ec2:RevokeSecurityGroupIngress","ec2:RunInstances","ec2:TerminateInstances","tag:GetResources","elasticloadbalancing:AddTags","elasticloadbalancing:CreateLoadBalancer","elasticloadbalancing:ConfigureHealthCheck","elasticloadbalancing:DeleteLoadBalancer","elasticloadbalancing:DescribeLoadBalancers","elasticloadbalancing:DescribeLoadBalancerAttributes","elasticloadbalancing:ApplySecurityGroupsToLoadBalancer","elasticloadbalancing:DescribeTags","elasticloadbalancing:ModifyLoadBalancerAttributes","elasticloadbalancing:RegisterInstancesWithLoadBalancer","elasticloadbalancing:DeregisterInstancesFromLoadBalancer","elasticloadbalancing:RemoveTags","autoscaling:DescribeAutoScalingGroups","autoscaling:DescribeInstanceRefreshes","ec2:CreateLaunchTemplate","ec2:CreateLaunchTemplateVersion","ec2:DescribeLaunchTemplates","ec2:DescribeLaunchTemplateVersions","ec2:DeleteLaunchTemplate","ec2:DeleteLaunchTemplateVersions"],"Resource": ["*"]},{"Effect": "Allow","Action": ["autoscaling:CreateAutoScalingGroup","autoscaling:UpdateAutoScalingGroup","autoscaling:CreateOrUpdateTags","autoscaling:StartInstanceRefresh","autoscaling:DeleteAutoScalingGroup","autoscaling:DeleteTags"],"Resource": ["arn:*:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*"]},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:*:iam::*:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"],"Condition": {"StringLike": {"iam:AWSServiceName": "autoscaling.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:*:iam::*:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing"],"Condition": {"StringLike": {"iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:*:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot"],"Condition": {"StringLike": {"iam:AWSServiceName": "spot.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:PassRole"],"Resource": ["arn:*:iam::*:role/*.cluster-api-provider-aws.sigs.k8s.io"]},{"Effect": "Allow","Action": ["secretsmanager:CreateSecret","secretsmanager:DeleteSecret","secretsmanager:TagResource"],"Resource": ["arn:*:secretsmanager:*:*:secret:aws.cluster.x-k8s.io/*"]},{"Effect": "Allow","Action": ["ssm:GetParameter"],"Resource": ["arn:*:ssm:*:*:parameter/aws/service/eks/optimized-ami/*"]},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:*:iam::*:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS"],"Condition": {"StringLike": {"iam:AWSServiceName": "eks.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:*:iam::*:role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup"],"Condition": {"StringLike": {"iam:AWSServiceName": "eks-nodegroup.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:aws:iam::*:role/aws-service-role/eks-fargate-pods.amazonaws.com/AWSServiceRoleForAmazonEKSForFargate"],"Condition": {"StringLike": {"iam:AWSServiceName": "eks-fargate.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:ListOpenIDConnectProviders","iam:CreateOpenIDConnectProvider","iam:AddClientIDToOpenIDConnectProvider","iam:UpdateOpenIDConnectProviderThumbprint","iam:DeleteOpenIDConnectProvider"],"Resource": ["*"]},{"Effect": "Allow","Action": ["iam:GetRole","iam:ListAttachedRolePolicies","iam:DetachRolePolicy","iam:DeleteRole","iam:CreateRole","iam:TagRole","iam:AttachRolePolicy"],"Resource": ["arn:*:iam::*:role/*"]},{"Effect": "Allow","Action": ["iam:GetPolicy"],"Resource": ["arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"]},{"Effect": "Allow","Action": ["eks:DescribeCluster","eks:ListClusters","eks:CreateCluster","eks:TagResource","eks:UpdateClusterVersion","eks:DeleteCluster","eks:UpdateClusterConfig","eks:UntagResource","eks:UpdateNodegroupVersion","eks:DescribeNodegroup","eks:DeleteNodegroup","eks:UpdateNodegroupConfig","eks:CreateNodegroup"],"Resource": ["arn:*:eks:*:*:cluster/*","arn:*:eks:*:*:nodegroup/*/*/*"]},{"Effect": "Allow","Action": ["eks:AssociateIdentityProviderConfig","eks:ListIdentityProviderConfigs"],"Resource": ["arn:aws:eks:*:*:cluster/*"]},{"Effect": "Allow","Action": ["eks:DisassociateIdentityProviderConfig","eks:DescribeIdentityProviderConfig"],"Resource": ["*"]},{"Effect": "Allow","Action": ["eks:ListAddons","eks:CreateAddon","eks:DescribeAddonVersions","eks:DescribeAddon","eks:DeleteAddon","eks:UpdateAddon","eks:TagResource","eks:DescribeFargateProfile","eks:CreateFargateProfile","eks:DeleteFargateProfile"],"Resource": ["*"]},{"Effect": "Allow","Action": ["iam:PassRole"],"Resource": ["*"],"Condition": {"StringEquals": {"iam:PassedToService": "eks.amazonaws.com"}}}]}
The following steps need to be performed to provision a new EKS cluster:
- Provide the basic cluster information like Name, Description, and Tags. Tags on a cluster are propagated to the VMs deployed on the cloud/data center environments.
- Select the Cluster Profile created for the EKS cloud. The profile definition will be used as the cluster construction template.
- Review and override pack parameters, as desired. By default, parameters for all packs are set with values defined in the cluster profile.
Provide the AWS Cloud account and configure the cluster.
Parameter Description Cloud Account Select the desired cloud account. AWS cloud accounts with AWS credentials need to be preconfigured in project settings. Static Placement By default, Palette uses dynamic placement, wherein a new VPC with a public and private subnet is created to place cluster resources for every cluster.
These resources are fully managed by Palette and deleted, when the corresponding cluster is deleted. Turn on the Static Placement option if it's desired to place resources into preexisting VPCs and subnets.Region Choose the preferred AWS region where you would like the clusters to be provisioned. SSH Key Pair Name Choose the desired SSH Key pair. SSH key pairs need to be preconfigured on AWS for the desired regions. The selected key is inserted into the VMs provisioned. Cluster Endpoint Access: Select Private or Public or Private & Public, based on how the customer want to establish the communication with the endpoint for the managed Kubernetes API server and your cluster. Public Access CIDR: For Public or Private & Public end point access, give the CIDR values. Enable Encryption The user can enable secret encryption by toggling the Enable Encryption option. Provide the provider KMS key ARN to complete the wizard. Worker Pool Update Optionally enable the option to update the worker pool in parallel.
kubernetes.io/role/elb = 1
sigs.k8s.io/cluster-api-provider-aws/role = public
kubernetes.io/cluster/[ClusterName] = shared
sigs.k8s.io/cluster-api-provider-aws/cluster/[ClusterName] = owned
Configure the Node Pools.
Configure one or more worker node pools. A single worker node will be configured by default.
Parameter Description Name A descriptive name for the node pool. Size Make your choice of minimum, maximum and desired sizes for the worker pool. The size of the worker pool will scale between the minimum and maximum size under varying workload conditions. Taints: Optionally enable node affinity optionally to attracts pods to a set of nodes Labels: Optionally enable Labels to constrain a Pod to only run on a particular set of Node(s) Instance Type Select the AWS instance type to be used for all nodes in the node pool.
Cloud Configuration settings:
Parameter Description Instance Option: Make the selection of instance allocation * On Demand * Spot Instance Type: Make the selection of the instance type Availability Zones: Select at least one availability zone within the VPC Disk Size Make the choice of disk size as per requirement Optionally, create one or more Fargate Profile(s) to aid the provisioning of on-demand, optimized compute capacity for the workload clusters.
Parameter Description Name Provide a name for the Fargate profile. Subnets Pods running on Fargate Profiles are not assigned public IP addresses, so only private subnets (with no direct route to an Internet Gateway) are accepted for this parameter. For dynamic provisioning, this input is not required and subnets are automatically selected. Selectors Define pod selector by providing a target namespace and optionally labels. Pods with matching namespace and app labels are scheduled to run on dynamically provisioned compute nodes.
You can have up to five selectors in a Fargate profile and a pod only needs to match one selector to run using the Fargate profile.
- Configure the Cluster Management options as per user requirements.
- Review the settings and deploy the cluster. Provisioning status with details of ongoing provisioning tasks is available to track progress.
Palette encourages using AWS Key Management Service (KMS) to provide envelope encryption of Kubernetes secrets stored in Amazon Elastic Kubernetes Service (EKS) clusters. This encryption is a defense-in-depth security strategy to protect the sensitive data such as passwords, docker registry credentials, and TLS keys stored as Kubernetes Secrets.
- KMS key created in the AWS console.
- Key Type: Symmetric
- Key Usage: Encrypt and Decrypt
The AWS permissions listed below need to be configured in the AWS account to enable key creation:
kms:CreateGrant
Enable secret encryption at step 4 of EKS cluster creation by toggling the botton and by updating ARN of the encryption key to the wizard.
Choose the instance type and the number of instances to be launched according to the number of pods required for the workload. The number of pods that can be scheduled on the nodes for an instance type needs to be calculated for the same; otherwise, the cluster creation cannot go to completion, as the pods cannot come up on the target cluster, due to resource unavailability.
The following section describes the method of calculating the pod capacity for individual AWS instance types. This will help in making exact choices of desired size of worker pool during cluster creation. We recommend selecting an instance that can support at least 30 pods.
Number of pods = N * (M-1) + 2
Where:
- N is the number of Elastic Network Interfaces (ENI) of the instance type (Maximum network interfaces).
- M is the number of IP addresses of a single ENI (Private IPv4 addresses per interface/IPv6 addresses per interface).
- Values for N and M for each instance type can be referred from this document.
- For instance type = t3.medium
- For values of N = 3, and M = 6 (values derived from AWS document )
- N * (M-1) + 2 = 3(6-1)+2 =17 pods/instances
- In this example, we will need at least two (2) t3.medium instances to reach the minimum of 30 pods threshold.
Hence, while setting the desired size of the worker pool, make the choice as per pod requirement. In the example given above, we need to launch a minimum of two (2) instances of t3.medium to satisfy the resource requirement of an EKS cluster.
The deletion of an EKS cluster results in the removal of all Virtual Machines and associated Storage Disks, created for the cluster. The following tasks need to be performed to delete an EKS cluster:
- Select the cluster to be deleted from the Cluster View page and navigate to the Cluster Overview page.
- Invoke a delete action available on the page: Cluster > Settings > Cluster Settings > Delete Cluster.
- Click Confirm to delete.
Cluster status is updated to Deleting while cluster resources are being deleted. Provisioning status is updated with the ongoing progress of the delete operation. Once all resources are successfully deleted, the cluster status changes to Deleted and is removed from the list of clusters.
A cluster stuck in the Deletion state can be force deleted by the user through the User Interface. The user can go for a force deletion of the cluster only if it is stuck in a deletion state for a minimum of 15 minutes. Palette enables cluster force delete from the tenant admin and project admin scope.
- Log in to the Palette Management Console.
Navigate to the Cluster Details page of the cluster stuck in deletion.
If the deletion is stuck for more than 15 minutes, click the Force Delete Cluster button from the Settings dropdown.
If the Force Delete Cluster button is not enabled, wait for 15 minutes. The Settings dropdown will give the estimated time for the auto-enabling of the force delete button.