Following are some architectural highlights of Amazon Web Services' (AWS) managed Kubernetes clusters (also known as Elastic Kubernetes Service or (EKS)), provisioned by Palette:
- Cluster resources such as Virtual Machines (VMs) can be provisioned into an existing infrastructure (Gateways, VPCs, Subnets etc.) as part of static provisioning as well as new dedicated infrastructure as part of dynamic provisioning.
- Full support for EKS Fargate profiles
Spot instance support
The following prerequisites must be met before deploying an EKS workload cluster:
- You must have an active AWS cloud account with all the permissions listed below in the AWS Cloud Account Permissions section.
- You must register your AWS cloud account in Palette as described in the Creating an AWS Cloud account section below.
- Have an Infrastructure cluster profile already created in Palette for EKS.
- Sufficient capacity in the desired AWS region should exist for the creation of the following resources:
- vCPU
- VPC
- Elastic IP
- Internet Gateway
- Elastic Load Balancers
- NAT Gateway
The following four policies include all the required permissions for provisioning clusters through Palette:
Controller Policy
{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": ["ec2:AllocateAddress","ec2:AssociateRouteTable","ec2:AttachInternetGateway","ec2:AuthorizeSecurityGroupIngress","ec2:CreateInternetGateway","ec2:CreateNatGateway","ec2:CreateRoute","ec2:CreateRouteTable","ec2:CreateSecurityGroup","ec2:CreateSubnet","ec2:CreateTags","ec2:CreateVpc","ec2:ModifyVpcAttribute","ec2:DeleteInternetGateway","ec2:DeleteNatGateway","ec2:DeleteNetworkInterface","ec2:DeleteRouteTable","ec2:DeleteSecurityGroup","ec2:DeleteSubnet","ec2:DeleteTags","ec2:DeleteVpc","ec2:DescribeAccountAttributes","ec2:DescribeAddresses","ec2:DescribeAvailabilityZones","ec2:DescribeInstances","ec2:DescribeInternetGateways","ec2:DescribeImages","ec2:DescribeKeyPairs","ec2:DescribeNatGateways","ec2:DescribeNetworkInterfaces","ec2:DescribeNetworkInterfaceAttribute","ec2:DescribeRouteTables","ec2:DescribeSecurityGroups","ec2:DescribeSubnets","ec2:DescribeVpcs","ec2:DescribeVpcAttribute","ec2:DescribeVolumes","ec2:DetachInternetGateway","ec2:DisassociateRouteTable","ec2:DisassociateAddress","ec2:ModifyInstanceAttribute","ec2:ModifyNetworkInterfaceAttribute","ec2:ModifySubnetAttribute","ec2:ReleaseAddress","ec2:RevokeSecurityGroupIngress","ec2:RunInstances","ec2:TerminateInstances","tag:GetResources","elasticloadbalancing:AddTags","elasticloadbalancing:CreateLoadBalancer","elasticloadbalancing:ConfigureHealthCheck","elasticloadbalancing:DeleteLoadBalancer","elasticloadbalancing:DescribeLoadBalancers","elasticloadbalancing:DescribeLoadBalancerAttributes","elasticloadbalancing:ApplySecurityGroupsToLoadBalancer","elasticloadbalancing:DescribeTags","elasticloadbalancing:ModifyLoadBalancerAttributes","elasticloadbalancing:RegisterInstancesWithLoadBalancer","elasticloadbalancing:DeregisterInstancesFromLoadBalancer","elasticloadbalancing:RemoveTags","autoscaling:DescribeAutoScalingGroups","autoscaling:DescribeInstanceRefreshes","ec2:CreateLaunchTemplate","ec2:CreateLaunchTemplateVersion","ec2:DescribeLaunchTemplates","ec2:DescribeLaunchTemplateVersions","ec2:DeleteLaunchTemplate","ec2:DeleteLaunchTemplateVersions"],"Resource": ["*"]},{"Effect": "Allow","Action": ["autoscaling:CreateAutoScalingGroup","autoscaling:UpdateAutoScalingGroup","autoscaling:CreateOrUpdateTags","autoscaling:StartInstanceRefresh","autoscaling:DeleteAutoScalingGroup","autoscaling:DeleteTags"],"Resource": ["arn:*:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*"]},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:*:iam::*:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"],"Condition": {"StringLike": {"iam:AWSServiceName": "autoscaling.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:*:iam::*:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing"],"Condition": {"StringLike": {"iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:*:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot"],"Condition": {"StringLike": {"iam:AWSServiceName": "spot.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:PassRole"],"Resource": ["arn:*:iam::*:role/*.cluster-api-provider-aws.sigs.k8s.io"]},{"Effect": "Allow","Action": ["secretsmanager:CreateSecret","secretsmanager:DeleteSecret","secretsmanager:TagResource"],"Resource": ["arn:*:secretsmanager:*:*:secret:aws.cluster.x-k8s.io/*"]},{"Effect": "Allow","Action": ["ssm:GetParameter"],"Resource": ["arn:*:ssm:*:*:parameter/aws/service/eks/optimized-ami/*"]},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:*:iam::*:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS"],"Condition": {"StringLike": {"iam:AWSServiceName": "eks.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:*:iam::*:role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup"],"Condition": {"StringLike": {"iam:AWSServiceName": "eks-nodegroup.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:CreateServiceLinkedRole"],"Resource": ["arn:aws:iam::*:role/aws-service-role/eks-fargate-pods.amazonaws.com/AWSServiceRoleForAmazonEKSForFargate"],"Condition": {"StringLike": {"iam:AWSServiceName": "eks-fargate.amazonaws.com"}}},{"Effect": "Allow","Action": ["iam:ListOpenIDConnectProviders","iam:CreateOpenIDConnectProvider","iam:AddClientIDToOpenIDConnectProvider","iam:UpdateOpenIDConnectProviderThumbprint","iam:DeleteOpenIDConnectProvider"],"Resource": ["*"]},{"Effect": "Allow","Action": ["iam:GetRole","iam:ListAttachedRolePolicies","iam:DetachRolePolicy","iam:DeleteRole","iam:CreateRole","iam:TagRole","iam:AttachRolePolicy"],"Resource": ["arn:*:iam::*:role/*"]},{"Effect": "Allow","Action": ["iam:GetPolicy"],"Resource": ["arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"]},{"Effect": "Allow","Action": ["eks:DescribeCluster","eks:ListClusters","eks:CreateCluster","eks:TagResource","eks:UpdateClusterVersion","eks:DeleteCluster","eks:UpdateClusterConfig","eks:UntagResource","eks:UpdateNodegroupVersion","eks:DescribeNodegroup","eks:DeleteNodegroup","eks:UpdateNodegroupConfig","eks:CreateNodegroup"],"Resource": ["arn:*:eks:*:*:cluster/*","arn:*:eks:*:*:nodegroup/*/*/*"]},{"Effect": "Allow","Action": ["eks:AssociateIdentityProviderConfig","eks:ListIdentityProviderConfigs"],"Resource": ["arn:aws:eks:*:*:cluster/*"]},{"Effect": "Allow","Action": ["eks:DisassociateIdentityProviderConfig","eks:DescribeIdentityProviderConfig"],"Resource": ["*"]},{"Effect": "Allow","Action": ["eks:ListAddons","eks:CreateAddon","eks:DescribeAddonVersions","eks:DescribeAddon","eks:DeleteAddon","eks:UpdateAddon","eks:TagResource","eks:DescribeFargateProfile","eks:CreateFargateProfile","eks:DeleteFargateProfile"],"Resource": ["*"]},{"Effect": "Allow","Action": ["iam:PassRole"],"Resource": ["*"],"Condition": {"StringEquals": {"iam:PassedToService": "eks.amazonaws.com"}}}]}
The following steps need to be performed to provision a new EKS cluster:
- Provide basic cluster information like Name, Description, and Tags. Tags on a cluster are propagated to the VMs deployed on the cloud/data center environments.
- Select the Cluster Profile created for the EKS cloud. The profile definition will be used as the cluster construction template.
- Review and override pack parameters, as desired. By default, parameters for all packs are set with values defined in the cluster profile.
Provide the AWS Cloud account and placement information.
Parameter Description Cloud Account Select the desired cloud account. AWS cloud accounts with AWS credentials need to be preconfigured in project settings. Region Choose the preferred AWS region where you would like the clusters to be provisioned. SSH Key Pair Name Choose the desired SSH Key pair. SSH key pairs need to be preconfigured on AWS for the desired regions. The selected key is inserted into the VMs provisioned. Static Placement By default, Palette uses dynamic placement, wherein a new VPC with a public and private subnet is created to place cluster resources for every cluster.
These resources are fully managed by Palette and deleted, when the corresponding cluster is deleted. Turn on the Static Placement option if it's desired to place resources into preexisting VPCs and subnets.
kubernetes.io/role/elb = 1
sigs.k8s.io/cluster-api-provider-aws/role = public
kubernetes.io/cluster/[ClusterName] = shared
sigs.k8s.io/cluster-api-provider-aws/cluster/[ClusterName] = owned
Configure one or more worker node pools. A single worker node will be configured by default.
Parameter Description Name A descriptive name for the node pool. Size Make your choice of minimum, maximum and desired sizes for the worker pool. The size of the worker pool will scale between the minimum and maximum size under varying workload conditions. Instance Type Select the AWS instance type to be used for all nodes in the node pool. Optionally, create one or more Fargate Profile(s) to aid the provisioning of on-demand, optimized compute capacity for the workload clusters.
Parameter Description Name Provide a name for the Fargate profile. Subnets Pods running on Fargate Profiles are not assigned public IP addresses, so only private subnets (with no direct route to an Internet Gateway) are accepted for this parameter. For dynamic provisioning, this input is not required and subnets are automatically selected. Selectors Define pod selector by providing a target namespace and optionally labels. Pods with matching namespace and app labels are scheduled to run on dynamically provisioned compute nodes.
You can have up to five selectors in a Fargate profile and a pod only needs to match one selector to run using the Fargate profile.Review the settings and deploy the cluster. Provisioning status with details of ongoing provisioning tasks is available to track progress.
Choose the instance type and the number of instances to be launched according to the number of pods required for the workload. The number of pods that can be scheduled on the nodes for an instance type needs to be calculated for the same; otherwise, the cluster creation cannot go to completion, as the pods cannot come up on the target cluster, due to resource unavailability.
The following section describes the method of calculating the pod capacity for individual AWS instance types. This will help in making exact choices of desired size of worker pool during cluster creation. We recommend selecting an instance that can support at least 30 pods.
Number of pods = N * (M-1) + 2
Where:
- N is the number of Elastic Network Interfaces (ENI) of the instance type (Maximum network interfaces).
- M is the number of IP addresses of a single ENI (Private IPv4 addresses per interface/IPv6 addresses per interface).
- Values for N and M for each instance type can be referred from this document.
- For instance type = t3.medium
- For values of N = 3, and M = 6 (values derived from AWS document )
- N * (M-1) + 2 = 3(6-1)+2 =17 pods/instances
- In this example, we will need at least two (2) t3.medium instances to reach the minimum of 30 pods threshold.
Hence, while setting the desired size of the worker pool, make the choice as per pod requirement. In the example given above, we need to launch a minimum of two (2) instances of t3.medium to satisfy the resource requirement of an EKS cluster.
The deletion of an EKS cluster results in the removal of all Virtual Machines and associated Storage Disks, created for the cluster. The following tasks need to be performed to delete an EKS cluster:
- Select the cluster to be deleted from the Cluster View page and navigate to the Cluster Overview page.
- Invoke a delete action available on the page: Cluster > Settings > Cluster Settings > Delete Cluster.
- Click Confirm to delete.
Cluster status is updated to Deleting while cluster resources are being deleted. Provisioning status is updated with the ongoing progress of the delete operation. Once all resources are successfully deleted, the cluster status changes to Deleted and is removed from the list of clusters.
A cluster stuck in the Deletion state can be force deleted by the user through the User Interface. The user can go for a force deletion of the cluster only if it is stuck in a deletion state for a minimum of 15 minutes. Palette enables cluster force delete from the tenant admin and project admin scope.
- Log in to the Palette Management Console.
Navigate to the Cluster Details page of the cluster stuck in deletion.
If the deletion is stuck for more than 15 minutes, click the Force Delete Cluster button from the Settings dropdown.
If the Force Delete Cluster button is not enabled, wait for 15 minutes. The Settings dropdown will give the estimated time for the auto-enabling of the force delete button.