Design Proposal #002: Existing VPC
Design Proposal #002: Existing VPC
STATUS: This proposal is a working draft, it will get refined and augment as needed. If any non-trivial changes are need to functionality defined here, in particular the user experience, those changes should be suggested via a PR to this proposal document. Any other changes to the text of the proposal or technical corrections are also very welcome.
Initial implementation of eksctl intentionally avoided having too many options for user to customise the VPC configuration. That was in oder to simplify how the tools works. Since then, many users asked for various features in relation to custom VPC.
There a few specific use-cases:
- co-location with kops #50
- set custom CIDR #279
- private/public subnets #120
- use any existing VPC #42
- use same subnet/AZ for a nodegroup #232
Out of the above, we already have support for 1 & 2 (as of 0.1.8) as well as 3 (as of 0.1.9).
The main challenge with these customisations is user experience and how different flags interact. Ultimately, once Cluster API support is implemented (planned for 0.3.0), some of these flags will not be needed and there will be more fine-grain parameters available via Cluster API. This proposal sets out flags that should serve well for most common use-cases until 0.3.0.
Flags defined in 0.1.9
These are fairly simple, and rather more user-friendly. These are assumed satisfactory for the purpose of this proposal.
--vpc-from-kops-cluster=cluster-1.k8s.local: import subnets from given kops cluster
--node-private-networking: use private subnets for initial nodegroup
--vpc-cidr=10.9.0.0/16: use 10.9.0.0/16 as the global CIDR for the VPC
Complete set of flags required for using existing VPC/subnets
In order to use eksctl with any custom VPC, one has to provide subnet ID. VPC ID can be derived by making an API call.
So the following flags will be needed:
--vpc-private-subnets="subnet-1,subnet-2" --vpc-public-subnets="subnet-3,subnet-4": use given subnet IDs, detect VPC ID
Also, the following combinations must be valid:
--vpc-private-subnets="subnet-1,subnet-2": use only given private subnet IDs, detect VPC ID
--vpc-public-subnets="subnet-3,subnet-4": use given public subnet IDs, detect VPC ID
User will have to make sure IDs are correct as well as their private/public designations.
For some use-cases, specifying VPC ID maybe preferred, and in that case the VPC maybe the default one (in which case it is ought to be possible to use default subnets or specific subnets). Hence the following combinations ought to be valid:
--vpc-id=default: use default VPC and all of its existing subnets
--vpc-id=default --vpc-private-subnets="subnet-1,subnet-2" --vpc-public-subnets="subnet-3,subnet-4": use default VPC, and given subnet IDs
--vpc-id="vpc-xxx": use given VPC, create dedicated subnets
--vpc-id="vpc-xxx" --vpc-private-subnets="subnet-1,subnet-2" --vpc-public-subnets="subnet-3,subnet-4": use given VPC ID, and given subnets (same as if
--vpc-idwas unset, but less explicit)
Needless to say that when none of
--vpc-* flags given, eksctl will create a dedicated VPC and subnets (public as well as
private). As more flags are being added, grouping them in
--help output would be very helpful; especially because most of
these flags are for advanced usage.
## Security Groups
It must be noted that security groups are managed by eksctl only, as certain configuration is required to ensure cluster is is fully functional. Unlike with subnets and VPC, security groups can be update after they were created and any ingress and/or egress rules can be added or removed. If more advanced usage is commonly required, a separate design proposal will be required. It is also expected that advanced functionality will be available via Cluster API (which is planned for 0.3.0).
It would be plausible to also provide utility for managing VPC, i.e.:
eksctl <create|get|delete> vpc --name=<vpcName>: manage VPC
eksctl create cluster --vpc-name=<vpcName>: create cluster in the given VPC
This would allow users to create VPCs with recommended configurations without having to worry about the requirements, yet they will be able to do VPC resource management at a different level of access. It should be noted that similar functionality is likely to be required for IAM resources (subject to separate proposal).
Notes on AWS CNI driver
It can be safely assumed (based on simple tests) that the driver is meant to coexist very happily with anything you have in your VPC, from plain EC2 instances, through load balancers and gateways to Lambda functions. It is thereby ought to be possible to share a VPC between two or more cluster running separate instances of the driver. It is expected that driver will try and use what IP addresses are available, which is only a subject to IP address space usage, of course. Allocations and other properties of the driver can be monitored via CloudWatch and/or Prometheus.
However, when very small subnets are used in a VPC, or there are too many resources using up the IP address space, one will ultimately run into issue sooner than later. If that’s the case, VPC peering or some other alternative should be consider. Peering should already be possible, and custom CIDR flag is available as of 0.1.8.
It must be noted, that there exists a plan to provide Weave Net as an overlay option. It maybe possible to use IPv6 also, but more experiments and research will need to be done (it not clear if EKS supports IPv6 properly yet).
Requirements of custom VPC
When user decides to use any of
--vpc-* flags, it is up to them to ensure that:
- suffiecent IP addresses are available
- suffiencent number of subnets (minimum 2)
- public/private subnets are configured correctly (i.e. routing and internet/NAT gateways are configured as intended)
- tagging of subnets
kubernetes.io/cluster/<name>tag set to either
kubernetes.io/role/internal-elbtag set to
1for private subnets
- any other tags
There maybe other requirements imposed by EKS or Kubernetes, and it is entirely up to the user to stay up-to-date on any requirements
and/or recommendations, and find ways in which those work well with the practices at their own organisations and any requirements imposed
by it. It is not our goal to keep this section of the proposal up to date. However,
eksctl create cluster aims to always provide an
up-to-date configuration with a dedicated VPC.
eksctl - a CLI for Amazon EKS
eksctl is a simple CLI tool for creating clusters on EKS - Amazon’s new managed Kubernetes service for EC2. It is written in Go, and uses CloudFormation.
You can create a cluster in minutes with just one command –
eksctl create cluster!
To download the latest release, run:
curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp sudo mv /tmp/eksctl /usr/local/bin
Alternatively, macOS users can use Homebrew:
brew install weaveworks/tap/eksctl
You will need to have AWS API credentials configured. What works for AWS CLI or any other tools (kops, Terraform etc), should be sufficient. You can use
or environment variables. For more information read AWS documentation.
To create a basic cluster, run:
eksctl create cluster
A cluster will be created with default parameters
- exciting auto-generated name, e.g. “fabulous-mushroom-1527688624”
m5.largenodes (this instance type suits most common use-cases, and is good value for money)
- use official AWS EKS AMI
- dedicated VPC (check your quotas)
- using static AMI resolver
Once you have created a cluster, you will find that cluster credentials were added in
~/.kube/config. If you have
kubectl v1.10.x as well as
aws-iam-authenticator commands in your PATH, you should be
able to use
kubectl. You will need to make sure to use the same AWS API credentials for this also. Check EKS docs for instructions. If you installed
eksctl via Homebrew, you should have all of these dependencies installed already.
$ eksctl create cluster 2018-10-26T16:22:17+01:00 [ℹ] using region us-west-2 2018-10-26T16:22:19+01:00 [ℹ] setting availability zones to [us-west-2a us-west-2b us-west-2c] 2018-10-26T16:22:19+01:00 [ℹ] subnets for us-west-2a - public:192.168.0.0/19 private:192.168.96.0/19 2018-10-26T16:22:19+01:00 [ℹ] subnets for us-west-2b - public:192.168.32.0/19 private:192.168.128.0/19 2018-10-26T16:22:19+01:00 [ℹ] subnets for us-west-2c - public:192.168.64.0/19 private:192.168.160.0/19 2018-10-26T16:22:19+01:00 [ℹ] using "ami-0a54c984b9f908c81" for nodes 2018-10-26T16:22:19+01:00 [ℹ] creating EKS cluster "floral-unicorn-1540567338" in "us-west-2" region 2018-10-26T16:22:19+01:00 [ℹ] will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup 2018-10-26T16:22:19+01:00 [ℹ] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --name=floral-unicorn-1540567338' 2018-10-26T16:22:19+01:00 [ℹ] creating cluster stack "eksctl-floral-unicorn-1540567338-cluster" 2018-10-26T16:33:03+01:00 [ℹ] creating nodegroup stack "eksctl-floral-unicorn-1540567338-nodegroup-0" 2018-10-26T16:36:44+01:00 [✔] all EKS cluster resource for "floral-unicorn-1540567338" had been created 2018-10-26T16:36:44+01:00 [✔] saved kubeconfig as "/Users/ilya/.kube/config" 2018-10-26T16:36:46+01:00 [ℹ] the cluster has 0 nodes 2018-10-26T16:36:46+01:00 [ℹ] waiting for at least 2 nodes to become ready 2018-10-26T16:37:22+01:00 [ℹ] the cluster has 2 nodes 2018-10-26T16:37:22+01:00 [ℹ] node "ip-192-168-25-215.us-west-2.compute.internal" is ready 2018-10-26T16:37:22+01:00 [ℹ] node "ip-192-168-83-60.us-west-2.compute.internal" is ready 2018-10-26T16:37:23+01:00 [ℹ] kubectl command should work with "~/.kube/config", try 'kubectl get nodes' 2018-10-26T16:37:23+01:00 [✔] EKS cluster "floral-unicorn-1540567338" in "us-west-2" region is ready $
To list the details about a cluster or all of the clusters, use:
eksctl get cluster [--name=<name>] [--region=<region>]
To create the same kind of basic cluster, but with a different name, run:
eksctl create cluster --name=cluster-1 --nodes=4
A default StorageClass (gp2 volume type provisioned by EBS) will be added automatically when creating a cluster. If you want to prevent this, use the
--storage-class flag. For example:
eksctl create cluster --storage-class=false
To write cluster credentials to a file other than default, run:
eksctl create cluster --name=cluster-2 --nodes=4 --kubeconfig=./kubeconfig.cluster-2.yaml
To prevent storing cluster credentials locally, run:
eksctl create cluster --name=cluster-3 --nodes=4 --write-kubeconfig=false
eksctl manage cluster credentials under
~/.kube/eksctl/clusters directory, run:
eksctl create cluster --name=cluster-3 --nodes=4 --auto-kubeconfig
To obtain cluster credentials at any point in time, run:
eksctl utils write-kubeconfig --name=<name> [--kubeconfig=<path>] [--set-kubeconfig-context=<bool>]
To use a 3-5 node Auto Scaling Group, run:
eksctl create cluster --name=cluster-5 --nodes-min=3 --nodes-max=5
To use 30
c4.xlarge nodes and prevent updating current context in
eksctl create cluster --name=cluster-6 --nodes=30 --node-type=c4.xlarge --set-kubeconfig-context=false
In order to allow SSH access to nodes,
~/.ssh/id_rsa.pub by default, to use a different SSH public key, e.g.
eksctl create cluster --ssh-access --ssh-public-key=my_eks_node_id.pub
To use a pre-existing EC2 key pair in
us-east-1 region, you can specify key pair name (which must not resolve to a local file path), e.g. to use
eksctl create cluster --ssh-access --ssh-public-key=my_kubernetes_key --region=us-east-1
To add custom tags for all resources, use
NOTE: Until [https://github.com/weaveworks/eksctl/issues/25] is resolved, tags cannot be applied to EKS cluster itself, but most of other resources (e.g. EC2 nodes).
eksctl create cluster --tags environment=staging --region=us-east-1
To configure node volume size, use the
eksctl create cluster --node-volume-size=50
us-east-1you are likely to get
UnsupportedAvailabilityZoneException. If you do, copy the suggested zones and pass
eksctl create cluster --region=us-east-1 --zones=us-east-1a,us-east-1b,us-east-1d. This may occur in other regions, but less likely. You shouldn’t need to use
To delete a cluster, run:
eksctl delete cluster --name=<name> [--region=<region>]
The initial nodegroup can be scaled by using the
eksctl scale nodegroup command. For example, to scale to 5 nodes:
eksctl scale nodegroup --name=<name> --nodes=5
If the desired number of nodes is greater than the current maximum set on the ASG then the maximum value will be increased to match the number of requested nodes. And likewise for the minimum.
Scaling a nodegroup works by modifying the nodegroup CloudFormation stack via a ChangeSet.
NOTE: Scaling a nodegroup down/in (i.e. reducing the number of nodes) may result in errors as we rely purely on changes to the ASG. This means that the node(s) being removed/terminated aren’t explicitly drained. This may be an area for improvement in the future.
eksctl create cluster instatiates a dedicated VPC, in order to avoid interference with any existing resources for a
variety of reasons, including security, but also because it’s challenging to detect all the settings in an existing VPC.
Default VPC CIDR used by
192.168.0.0/16, it is divided into 8 (
/19) subnets (3 private, 3 public & 2 reserved).
Initial nodegroup is create in public subnets, with SSH access disabled unless
--allow-ssh is specified. However, this implies
that each of the EC2 instances in the initial nodegroup gets a public IP and can be accessed on ports 1025 - 65535, which is
not insecure in principle, but some compromised workload could risk an access violation.
If that functionality doesn’t suit you, the following options are currently available.
change VPC CIDR
If you need to setup peering with another VPC, or simply need larger or smaller range of IPs, you can use
--vpc-cidr flag to
change it. You cannot use just any sort of CIDR, there only certain ranges that can be used in AWS VPC.
use private subnets for initial nodegroup
If you prefer to isolate initial nodegroup from the public internet, you can use
When used in conjunction with
--ssh-access flag, SSH port can only be accessed inside the VPC.
use existing VPC: shared with kops
You can use a VPC of an existing Kubernetes cluster managed by kops. This feature is provided to facilitate migration and/or cluster peering.
If you have previously created a cluster with kops, e.g. using commands similar to this:
export KOPS_STATE_STORE=s3://kops kops create cluster cluster-1.k8s.local --zones=us-west-2c,us-west-2b,us-west-2a --networking=weave --yes
You can create an EKS cluster in the same AZs using the same VPC subnets (NOTE: at least 2 AZs/subnets are required):
eksctl create cluster --name=cluster-2 --region=us-west-2 --vpc-from-kops-cluster=cluster-1.k8s.local
use existing VPC: any custom configuration
Use this feature if you must configure a VPC in a way that’s different to how dedicated VPC is configured by
eksctl, or have to
use a VPC that already exists so your EKS cluster gets shared access to some resources inside that existing VPC, or you have any
other use-case that requires you to manage VPCs separately.
You can use an existing VPC by supplying private and/or public subnets using
It is up to you to ensure which subnets you use, as there is no simple way to determine automatically whether a subnets is private or
public, because configurations vary.
Given these flags,
eksctl create cluster will determine the VPC ID automatically, but it will not create any routing tables or other
resources, such as internet/NAT gateways. It will, however, create dedicated security groups for the initial nodegroup and the control
You must ensure to provide at least 2 subnets in different AZs. There are other requirements that you will need to follow, but it’s entirely up to you to address those. For example, tagging is not strictly necessary, tests have shown that its possible to create a functional cluster without any tags set on the subnets, however there is no guarantee that this will always hold and tagging is recommended.
- all subnets in the same VPC, within the same block of IPs
- sufficient IP addresses are available
- sufficient number of subnets (minimum 2)
- internet and/or NAT gateways are configured correctly
- routing tables have correct entries and the network is functional
- tagging of subnets
kubernetes.io/cluster/<name>tag set to either
kubernetes.io/role/internal-elbtag set to
1for private subnets
There maybe other requirements imposed by EKS or Kubernetes, and it is entirely up to you to stay up-to-date on any requirements and/or recommendations, and implement those as needed/possible.
Default security group settings applied by
eksctl may or may not be sufficient for sharing access with resources in other security
groups. If you wish to modify the ingress/egress rules of the either of security groups, you might need to use another tool to automate
changes, or do it via EC2 console.
If you are in doubt, don’t use a custom VPC. Using
eksctl create cluster without any
--vpc-* flags will always configure the cluster
with a fully-functional dedicated VPC.
To create a cluster using 2x private and 2x public subnets, run:
eksctl create cluster \ --vpc-private-subnets=subnet-0ff156e0c4a6d300c,subnet-0426fb4a607393184 \ --vpc-public-subnets=subnet-0153e560b3129a696,subnet-009fa0199ec203c37
To create a cluster using 3x private subnets and make initial nodegroup use those subnets, run:
eksctl create cluster \ --vpc-private-subnets=subnet-0ff156e0c4a6d300c,subnet-0549cdab573695c03,subnet-0426fb4a607393184 \ --node-private-networking
To create a cluster using 4x public subnets, run:
eksctl create cluster \ --vpc-public-subnets=subnet-0153e560b3129a696,subnet-0cc9c5aebe75083fd,subnet-009fa0199ec203c37,subnet-018fa0176ba320e45
If you’d like to use GPU instance types (i.e. p2 or p3 ) then the first thing you need to do is subscribe to the EKS-optimized AMI with GPU Support. If you don’t do this then node creation will fail.
After subscribing to the AMI you can create a cluster specifying the GPU instance type you’d like to use for the nodes. For example:
eksctl create cluster --node-type=p2.xlarge
The AMI resolvers (both static and auto) will see that you want to use a GPU instance type (p2 or p3 only) and they will select the correct AMI.
Once the cluster is created you will need to install the NVIDIA Kubernetes device plugin. Check the repo for the most up to date instructions but you should be able to run this:
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.11/nvidia-device-plugin.yml
addonsupport has been added as part of 0.2.0 it is envisioned that there will be a addon to install the NVIDIA Kubernetes Device Plugin. This addon could potentially be installed automatically as we know an GPU instance type is being used.
Latest & Custom AMI Support
With the 0.1.2 release we have introduced the
--node-ami flag for use when creating a cluster. This enables a number of advanced use cases such as using a custom AMI or querying AWS in realtime to determine which AMI to use (non-GPU and GPU instances).
--node-ami can take the AMI image id for an image to explicitly use. It also can take the following ‘special’ keywords:
|static||Indicates that the AMI images ids embedded into eksctl should be used. This relates to the static resolvers.|
|auto||Indicates that the AMI to use for the nodes should be found by querying AWS. This relates to the auto resolver.|
If, for example, AWS release a new version of the EKS node AMIs and a new version of eksctl hasn’t been released you can use the latest AMI by doing the following:
eksctl create cluster --node-ami=auto
With the 0.1.9 release we have introduced the
--node-ami-family flag for use when creating the cluster. This makes it possible to choose between different offically supported EKS AMI families.
--node-ami-family can take following keywords:
|AmazonLinux2||Indicates that the EKS AMI image based on Amazon Linux 2 should be used. (default)|
|Ubuntu1804||Indicates that the EKS AMI image based on Ubuntu 18.04 should be used.|
Developer use-case (0.2.0)
It should suffice to install a cluster for development with just a single command. Here are some examples:
To create a cluster with default configuration (2
m5.large nodes), run:
eksctl create cluster
The developer may choose to pre-configure popular addons, e.g.:
- Weave Net:
eksctl create cluster --networking weave
eksctl create cluster --addons helm
- AWS CI tools (CodeCommit, CodeBuild, ECR):
eksctl create cluster --addons aws-ci
- Jenkins X:
eksctl create cluster --addons jenkins-x
- AWS CodeStar:
eksctl create cluster --addons aws-codestar
- Weave Scope and Flux:
eksctl create cluster --addons weave-scope,weave-flux
It should be possible to combine any or all of these addons.
It would also be possible to add any of the addons after cluster was created with
eksctl create addons.
Manage EKS the GitOps way (0.3.0)
eksctl aims to be compliant with GitOps model, and can be used as part of a GitOps toolkit!
eksctl apply --cluster-config prod-cluster.yaml will manage cluster state declaratively.
eksctld will be a controller inside of one cluster that can manage multiple other clusters based on Kubernetes Cluster API definitions (CRDs).
Code contributions are very welcome. If you are interested in helping make eksctl great then see our contributing guide.
Get in touch
Create an issue, or login to Weave Community Slack (#eksctl) (signup).
Original Gophers drawn by Ashley McNamara, unique E, K, S, C, T & L Gopher identities had been produced with Gopherize.me.