Setup your cluster with GitOps

Setup your cluster with GitOps

Welcome to eksctl GitOps Quick Starts. In this guide we will show you how to launch fully-configured Kubernetes clusters that are ready to run production workloads in minutes: easy for you to get started running Kubernetes on EKS and to launch standard clusters in your organisation.

At the end of this, you will have a Kubernetes cluster including control plane, worker nodes, and all of the software needed for code deployment, monitoring, and logging.

Quick Start to GitOps

GitOps is a way to do Kubernetes application delivery. It works by using Git as a single source of truth for Kubernetes resources and everything else. With Git at the center of your delivery pipelines, you and your team can make pull requests to accelerate and simplify application deployments and operations tasks to Kubernetes.

Using GitOps Quick Starts will get you set up in next to no time. You will benefit from a setup that is based on the experience of companies who run workloads at scale.

Prerequisites

To use EKS, you need to have your AWS account set up.

Next you will have to have the following tools installed:

Getting ready for GitOps

The main point of GitOps is to keep everything (config, alerts, dashboards, apps, literally everything) in Git and use it as a single source of truth. To keep your cluster configuration in Git, please go ahead and create an empty repository. On Github for example, follow these steps.

Standing up your cluster

First we follow the usual steps to stand up a cluster on EKS. Please review the list of flags to see if you need to tweak it for your purposes. In essence it is going to be a variation of:

eksctl create cluster

Once it is finished, you should be able to check the cluster contents and see some system workloads:

kubectl get nodes
NAME                                              STATUS   ROLES    AGE   VERSION
ip-192-168-15-6.eu-central-1.compute.internal     Ready    <none>   39s   v1.13.8-eks-cd3eb0
ip-192-168-64-189.eu-central-1.compute.internal   Ready    <none>   38s   v1.13.8-eks-cd3eb0

Applying GitOps

The following command will set up your cluster with the app-dev profile, the first GitOps Quick Start. All of the config files you need for a production-ready cluster will be in the git repo you have provided and those components will be deployed to your cluster. When you make changes in the configuration they will be reflected on your cluster.

This is an experimental feature. To enable it, set the environment variable EKSCTL_EXPERIMENTAL=true.

Experimental features are not stable and their command name and flags may change.

The most important ingredient using gitops apply is your config repository (which includes your workload manifests, etc). You can start with an empty repository and push that to Git, or use the one you intend to deploy to the cluster.

What will happen during the following command is:

  • eksctl will add the definition of your EKS cluster itself to Git, so you can operate it through pull requests.
  • It will also add the app-dev GitOps Quick Start profile to it, which comes with a lot of very useful services and config. It is how we feel EKS app-dev clusters are best run.

Run this command from your local check-out of your config repository. Or tweak output-path accordingly.

EKSCTL_EXPERIMENTAL=true eksctl \
        gitops apply \
        --quickstart-profile app-dev \
        --git-url git@github.com:example/my-eks-config \
        --git-email your@email.com \
        --cluster your-cluster-name

Let us go through the specified arguments one by one:

  • --quickstart-profile: this is the name of one of the profiles we put together, so you can easily pick and choose and will not have to start from scratch every time. We use app-dev here.
  • --git-url: this points to a Git URL where the configuration for your cluster will be stored. This will contain config for the workloads and infrastructure later on.
  • --git-email: the email used to commit changes to your config repository.
  • cluster: the name of your cluster. Use eksctl get cluster to see all clusters in your default region.

There are more arguments and options, please refer to the GitOps reference of eksctl which details all the flags and resulting directory structure.

The command will take a while to run and it’s a good idea to scan the output. You will note a similar bit of information in the log like this one:

[ℹ]  Flux will only operate properly once it has write-access to the Git repository
[ℹ]  please configure git@github.com:YOURUSER/eks-quickstart-app-dev.git so that the following Flux SSH public key has write access to it
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8msUDG9tEEWHKKJw1o8BpwfMkCvCepeUSMa9iTVK6Bmxeu2pA/ivBS8Qgx/Lg8Jnu4Gk2RbXYMt3KL3/lcIezLwqipGmVvLgBLvUccbBpeUpWt+SlW2LMwcMOnhF3n86VOYjaRPggoPtWfLhFIfnkvKOFLHPRYS3nqyYspFeCGUmOzQim+JAWokf4oGOOX4SNzRKjusboh93oy8fvWk8SrtSwLBWXOKu+kKXC0ecZJK7G0jW91qb40QvB+VeSAbfk8LJZcXGWWvWa3W0/woKzGNWBPZz+pGuflUjVwQG5GoOq5VVWu71gmXoXBS3bUNqlu6nDobd2LlqiXNViaszX

Copy the lines starting with ssh-rsa and add it as a deploy key, to e.g. Github. There you can easily do this in the Settings > Deploy keys > Add deploy key. Just make sure you check Allow write access as well.

The next time Flux syncs from Git, it will start updating the cluster and actively deploying.

This will set up Flux on your cluster and load GitOps Quick Start config files into your repo. It will use templating to add your cluster name and region to the configuration so that key cluster components that need those values can work (e.g. alb-ingress).

If you run git pull next, you will see that Flux has commited them to your config repository already.

In our case we are going to see these new arrivals in the cluster:

kubectl get pods -n kubernetes-dashboard
kubernetes-dashboard   dashboard-metrics-scraper-f7b5dbf7d-kwz7n   1/1     Running   0          4m
kubernetes-dashboard   kubernetes-dashboard-7447f48f55-2pl66       1/1     Running   0          4m

All of the cluster configuration can be easily edited in Git now. Welcome to a fully GitOpsed world!

Your GitOps cluster

Welcome to your fully GitOpsed cluster. By choosing the app-dev Quick Start profile, you will now also have the following components running in your cluster:

It’s easy to confirm if all of this is up and running for you. Let’s check podinfo and see if it’s up.

kubectl get service --namespace demo
NAME      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
podinfo   ClusterIP   10.100.255.220   <none>        9898/TCP   2m

Now let’s port-forward the service, so we can easily access it:

kubectl port-forward -n demo svc/podinfo 9898:9898

If you open localhost:9898 in your browser, you will see

podinfo up and running

Congratulations to your GitOpsed cluster on EKS!

Advanced setups

eksctl gitops apply can largely be decomposed into

  1. eksctl install flux
  2. eksctl generate profile

So for more complex use cases, you will want to run these steps separately and on your own and adjust flags and options as necessary. The first command installs Flux and links it to a Git repo that you provide. The second generates the manifest files from the GitOps Quick Start profile locally, so that you can edit them before pushing to your Git repo.

Configuring Flux

Here we will install Flux, the Kubernetes GitOps operator in your cluster. It will take care of deployments for you.

EKSCTL_EXPERIMENTAL=true eksctl install flux \
    --git-url <git-url> \
    --git-email <email-of-committer> \
    --name <cluster-name>

Additional options to the command are explained in our docs on install flux.

Now we run e.g.:

EKSCTL_EXPERIMENTAL=true eksctl install flux \
    --git-url git@github.com:YOURUSER/flux-get-started \
    --git-email your@email.org \
    --name wonderful-wardrobe-1565767990

After about a minute your cluster will have flux running, which will monitor your git repository once you added an SSH key to e.g. Github deploy keys. The Flux Helm Operator and Tiller will be deployed in your cluster too.

This key can be found at the end of the output of the command, this might for example be:

[...]
[ℹ]  Flux will only operate properly once it has write-access to the Git repository
[ℹ]  please configure git@github.com:YOURUSER/flux-get-started so that the following Flux SSH public key has write access to it
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCt8no0/F3+kD1YukH6sVIv1ONcy9+01G2/AQe1CQA+uRHaioep41U3ghROU7CoM1yTjG+eLYgu26UMvkXVbOmEm+1697adh4qz/yCF0E7JtCIIXGn/1XrLb6OxgtlGKdJ4fTUdxtQSyTvWqjxQhC4ute9hnHWU8oiSrNaq5D20P5x8sgPf4V0A5YWD5S4YliJcIupTzrD7zjhh6TyP5fqhPLHPBZFHStHq0DSD+Gi6vXZz1s9UmuAnxP8pkIlrW22xJyFbsmcjJuks5FvmLo8uJMeWTx5t+3WKWp8ZKrbDJFUWQ8aVMByHYq1c3doevM28CHwz/

Copy the lines starting with ssh-rsa and add it as a deploy key, to e.g. Github. There you can easily do this in the Settings > Deploy keys > Add deploy key. Just make sure you check Allow write access as well.

The next time Flux syncs from Git, it will start updating the cluster and actively deploying. If you use the flux-get-started repo from above, here’s what you will see in your cluster:

kubectl get pods -n demo
NAME                       READY   STATUS    RESTARTS   AGE
podinfo-5f4bd464b4-4vf7k   1/1     Running   0          58m
podinfo-5f4bd464b4-hkgzt   1/1     Running   0          58m

Remember that you can further tweak the installation of Flux as discussed in our install flux docs.

Handcrafting your configuration

In this second step we will use eksctl generate profile, so you can easily handcraft your cluster configuration and maintain it in Git, just like you would for the workloads you are running.

During the previous call (eksctl install flux), we instructed Flux to watch a config repository and deploy changes to the cluster. This config repository is where the workloads are defined. Now we will add the config of the infrastructure tooling as well.

eksctl has the ability to use a base config for for the infrastructure tooling. So if your organisation has already experience in setting up clusters and use the same defaults, it makes sense to use those as a base config. You could be entirely starting from scratch here too. What we will do in this part of the tutorial is using weaveworks/eks-quickstart-app-dev, which is the app-dev GitOps Quick Start profile. To create your own base config repository check out the documentation.

Now please run:

EKSCTL_EXPERIMENTAL=true eksctl generate profile \
        --name wonderful-wardrobe-1565767990 \
        --git-url git@github.com:weaveworks/eks-quickstart-app-dev.git \
        --profile-path ~/dev/flux-get-started/cluster-config

Let’s break this down here. eksctl generate profile at the very least wants:

  • --name: the name of the cluster - check eksctl get cluster to see what the name of yours is
  • --git-url: the Git URL of the base config of your cluster
  • --profile-path: a local path: this is an empty new directory (here cluster-config) you create in your local checkout of the config repository, which we used in the previous command

The base config can be something you and your organisation tailored to your needs, but can be something like our app-dev Quick Start profile as well. It is meant to be a starting point for clusters you can iterate over.

So after all this preface, what happens when we run the command? eksctl will check out the base config (here we use a copy of the GitOps example config) in an empty sub-directory (cluster-config) of our local checkout of flux-get-started.

All that is left now to get our cluster components managed by Flux is to commit them to our config repository:

cd ~/dev/flux-get-started/cluster-config
git add .
git commit -m "add cluster config"
git push

Flux will pick this up in its next sync and make the changes to your cluster.

In our case we are going to see these new arrivals in the cluster:

kubectl get pods -n kubernetes-dashboard
kubernetes-dashboard   dashboard-metrics-scraper-f7b5dbf7d-kwz7n   1/1     Running   0          4m
kubernetes-dashboard   kubernetes-dashboard-7447f48f55-2pl66       1/1     Running   0          4m

All of the cluster configuration can be easily edited in Git now. Welcome to a fully GitOpsed world!

Conclusion

We look forward to hearing your thoughts and feedback. Please get in touch and let us know how things worked out for you.