Skip to main content
Version: latest

Architecture

Palette enables importing and managing Amazon Elastic Kubernetes Service (Amazon EKS) Hybrid Nodes. For an overview of all necessary prerequisites, refer to Prerequisite setup for hybrid nodes.

These are some of the architectural highlights when using Palette to manage your Amazon EKS Hybrid Nodes.

  • Add a Cilium Container Network Interface (CNI) layer to your Amazon EKS cluster to handle networking for hybrid nodes using affinity rules.

  • Create hybrid node pools comprising edge hosts that have been registered with Palette.

  • Define cluster profiles to collectively manage your hybrid nodes. Each cluster profile for a hybrid node pool includes the following configurable layers:

    • If you are using Appliance Mode, configure the Operating System (OS) layer to reference the provider image built during the EdgeForge workflow and optional customizations for your hybrid nodes.

    • Configure the Kubernetes layer to specify the correct Amazon EKS Distro kubelet version to be installed on hybrid nodes.

Hybrid Network Connectivity

Network connectivity between your on-prem environments, edge locations, and Amazon EKS cluster must be established before Palette can manage your Amazon EKS Hybrid Nodes.

In the following example, an Amazon EKS cluster is connected to an on-prem data center and edge location through an AWS Transit Gateway and AWS Site-to-Site Virtual Private Network (VPN).

Example Amazon EKS Hybrid Nodes network architecture

Hybrid network connectivity can be configured using a variety of methods, such as:

Refer to Network-to-Amazon VPC connectivity options for guidance on all available options.

Configuration Requirements

If using a VPN or AWS Direct Connect between AWS and your on-prem and edge environments, review the following configuration requirements.

AWS

Traffic routing in the Amazon EKS VPC requires the following mapping for hybrid nodes:

  • Route table entries mapping hybrid node CIDR ranges to VPN endpoint.
    For example, Hybrid Node CIDR 10.200.0.0/16 → VPN endpoint 172.16.0.1.

  • Route table entries mapping hybrid pod CIDR ranges to VPN endpoint.
    For example, Hybrid Pod CIDR 192.168.0.0/16 → VPN endpoint 172.16.0.1.

  • For AWS Direct Connect, map traffic to appropriate private subnet CIDR.
    For example, both CIDRs 10.200.0.0/16 & 192.168.0.0/16 → Private subnet 172.16.1.0/24.

For AWS VPNs, configure two static routes for each of the following CIDRs:

  • Hybrid Node CIDR block.
    For example, Hybrid Node CIDR 10.200.0.0/16 → VPN endpoint 172.16.0.1.

  • Hybrid Pod CIDR block.
    For example, Hybrid Pod CIDR 192.168.0.0/16 → VPN endpoint 172.16.0.1.

If you're using a Virtual Private Gateway or Transit Gateway, route propagation can be enabled to automatically populate your VPC route tables. Ensure you verify your route tables after propagation.

On-Prem and Edge Locations

For on-prem and edge VPNs, set up IPsec Phase 1 tunnels with Phase 2 security associations for the following:

  • Hybrid Node subnet to EKS VPC CIDR.
    For example, Hybrid Node subnet 10.201.0.0/16 → EKS VPC CIDR 10.100.0.0/16.

  • Hybrid Node pod CIDR to EKS VPC CIDR.
    For example, Hybrid Node Pod CIDR 192.168.0.0/16 → EKS VPC CIDR 10.100.0.0/16.

You should also configure Border Gateway Protocol (BGP) or static routes on your on-prem or edge location router to ensure network traffic reaches the correct hybrid nodes. For static routing, this is explained in more detail during the Configure Hybrid Node Networking for VPN Solutions steps.

A route must exist to send all traffic destined for the Amazon EKS VPC through a centralized VPN gateway, or alternatively, a unique VPN server IP can be defined for each hybrid node during the Create Hybrid Node Pool steps.

Operating System Compatibility

Palette supports the same operating systems as AWS. Refer to Prepare operating system for hybrid nodes for details.

Supported Edge Hosts

Palette allows you to use your edge hosts as your Amazon EKS Hybrid Nodes. Your edge hosts need to be registered with Palette before you can add them to your node pools.

If you want to use your edge hosts as Amazon EKS Hybrid Nodes, they must have been registered using one of the following methods:

warning

Your edge host package managers must have up-to-date package indexes. This is to ensure that dependency packages for nodeadm can be successfully downloaded and installed.

For example, on Ubuntu, you would issue the following command.

sudo apt-get update

Adjust to your operating system and package manager on your edge hosts.

Bind Mount Requirements

If you are using Appliance Mode to deploy your edge hosts, ensure the following bind mounts are specified in the user data configuration. Add the following install block to your Edge installer user data file.

install:
extra-dirs-rootfs:
- /eks-hybrid
bind_mounts:
- /eks-hybrid
- /etc/aws
- /etc/containerd
- /etc/eks
- /etc/iam
- /etc/modules-load.d
- /var/lib/amazon

This snippet ensures that the required directories are mounted and available on your edge hosts, which are required for EKS Hybrid Nodes to function correctly.

Build Provider Images with Specific Arguments

If using the Appliance Mode, you must include the following in your EdgeForge .arg file during the build steps for provider images.

K8S_DISTRIBUTION=nodeadm
K8S_VERSION=<kubernetesVersion> # supported versions: [ 1.28.0 | 1.29.0 | 1.30.0 | 1.31.0 ]

Replace <kubernetesVersion> with your version of Kubernetes. For example, 1.29.0.

Authentication and Access Management

Palette supports the following authentication methods for your hybrid nodes:

Refer to Prepare credentials for hybrid nodes for guidance on how to set up credentials for your hybrid nodes.

Amazon EKS Cluster Requirements

  • Hybrid Node Enablement: The cluster must be enabled for hybrid nodes, as outlined in Create an Amazon EKS cluster with hybrid nodes.

  • Container Network Interface (CNI): The AWS VPC CNI must be used as the cluster’s CNI as it is needed for cloud worker nodes to function. Although the AWS Console may display a warning indicating that the AWS VPC CNI is not supported for hybrid nodes, this can be safely disregarded.

  • Worker Node Requirements for the Palette Agent: At least one worker node is required to host the Palette agent, which is essential for Palette to manage the cluster. Due to the EKS architecture, the Palette agent cannot be installed on the EKS control plane.

    • Minimum Instance Type: Worker nodes must use an instance type of at least t3.xlarge to ensure adequate resources. AWS sets a default storage of 20 GB for Linux-based EKS worker nodes (the diskSize parameter), and we recommend this as the minimum size.

    • Ongoing Node Availability: To maintain continuous management capabilities, at least one worker node should remain available at all times for the Palette agent to operate effectively.