Introduction

In this post, I will show you how to manage the deployment and update of Helm charts into you Kubernetes cluster using the CI/CD tool provided by Github.

Helm charts can be daunting to manage at first, but in the long run they remove the hassle of managing your micro-services using self-written Kubernetes manifests by letting you use a set of curated templates to deploy an application into your cluster. We can compare it to a package manager like apt or yum if you are familiar with some Linux distributions but for Kubernetes.

To show you an example, to install an apache web server into an Ubuntu instance, you would use those commands:

$ apt update
$ apt install apache2

To achieve the same result in Kubernetes using Helm you can use a similar set of commands:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm repo update
$ helm install my-release bitnami/apache

As you can see, Helm does not have any repository configured from the start, that is why we are using the helm repo add command to be able to search for, download and install a specific chart.

For the purpose of this post, i will use the chart cert-manager.

Prerequisite

In order to follow this post, you will need to have:

an already deployed Kubernetes cluster, in my case i will use GKE.
a set of credentials with proper permissions.
a GitHub Repository to manage your workflows.

Create your workflow

Prepare your values file

For Cert-manager, we will need a file to specify some configuration changes:

global:
  podSecurityPolicy:
    enabled: true

prometheus:
  enabled: false

I do have podSecurityPolicies enabled in my cluster, so i need to ensure the deployment of this chart to comply with my security standards, thus setting to true the global.podSecurityPolicy.enabled key. The prometheus.enabled key value is set to false for the simple reason that i don't have Prometheus running on this cluster at the moment (i should but it costs money hehe).

Be sure to save this value file in your repository.

Design the different steps

Any workflow hosted by Github Actions will need defined number of steps to have an impact on the underlying infrastructure. They are often similar to what a human operator would need to do manually with some little differences that will need to be adapted to a workflow:

Connecting to a bastion or similar instance will turn into launching a GitHub Action worker on a specified environment (Ubuntu for exemple).
Install the gcloud CLI tool into the worker and set it up with your set of credentials.
Install Helm 3 into the worker.
Use the gcloud CLI to authenticate into the cluster.
Install the helm repository needed for the chart installation.
Launch an installation of the chart using the --dry-run flag.
Launch the installation of the chart.

Those are the base of what we want our workflow to do.

Turning our steps into code

Setup the workflow

First, let's dump our code right here:

name: dry-runs

on: 
  pull_request:
    branches: 
    - '*'

jobs:
  kubernetes:
    runs-on: ubuntu-latest
    steps:
      - name: setup gcloud
        uses: google-github-actions/setup-gcloud@master
        with:
          service_account_email: ${{ secrets.GHUB_ACTION_CI_GCP_EMAIL }}
          service_account_key: ${{ secrets.GHUB_ACTION_CI_GCP_KEY }}
      - name: setup Helm3
        run: |
          curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
          chmod 700 get_helm.sh
          ./get_helm.sh
          helm plugin install https://github.com/databus23/helm-diff
      - name: authenticating into cluster
        run: |
          gcloud config set project jlops-me
          gcloud container clusters get-credentials primaris-cluster --region=europe-west1-c
      - name: Checkout
        uses: actions/checkout@v2
      - name: install CRDs for cert-manager
        run: kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/v0.13.1/deploy/manifests/00-crds.yaml --dry-run=client
      - name: helm cert-manager install dryrun
        run: |
          helm repo add $CHART_REPO $REPO_URL
          helm repo update
          if helm list -n $RELEASE_NAME | grep $RELEASE_NAME > /dev/null ; then if ! helm diff upgrade $RELEASE_NAME $CHART_REPO/$CHART_NAME --namespace $CHART_NAMESPACE --version $CHART_VERSION -f $VALUES_PATH --detailed-exitcode; then helm upgrade $RELEASE_NAME $CHART_REPO/$CHART_NAME --namespace $CHART_NAMESPACE --version $CHART_VERSION -f $VALUES_PATH --dry-run; else echo "There is no changes for this release."; fi; else helm install $RELEASE_NAME $CHART_REPO/$CHART_NAME --namespace $CHART_NAMESPACE --version $CHART_VERSION -f $VALUES_PATH --dry-run; fi
        env: 
          CHART_VERSION: v0.13.1
          CHART_NAMESPACE: cert-manager
          CHART_REPO: jetstack
          REPO_URL: https://charts.jetstack.io
          CHART_NAME: cert-manager
          RELEASE_NAME: cert-manager
          VALUES_PATH: infrastructure/kubernetes/helm3/cert-manager/values.yaml

Let's take a step back and explain each important details from this workflow:

The name key will let Github know how to name this workflow and the on keys will allow this workflow to be triggered whenever a Pull Request is created and is updated. The deployment workflow will be using the merge event and will be described later.
The jobs key will let you define a set of jobs into the workflow. You can call it whatever you feel like (ex: kubernetes).
runs-on will let you run the worker in an Ubuntu environment.
The steps key will translate into what we previously explained and will need an entire block of explaining.

Workflow explanation

Basic explanation

Now for the steps:

setup gcloud: this step will use an action created by the community that will download, install and configure glcoud. For this to work you will also need a set of credentials. You will be able to find how to setup those using this previous article.
checkout: This will let us use the values files needed in order to apply the configuration we want in the chart.
setup Helm3: this step will directly translate the Helm installation procedure into a set of commands run by the worker. For instance: download, change the permission of the script and running it.
authenticating into cluster: on this one we will let the worker use the appropriate GCP project where your cluster is located then authenticate into it.
install CRDs for cert-manager: this chart needs some Custom Resources Definitions. The step will apply a distant kubernetes manifest to ensure the CRD's needed by cert-manager to work are present in the cluster.
helm cert-manager install dryrun: this step is present to dry-run the chart installation in the cluster. This part will need more explanations and there are specific reasons on why things are done this way.

How to handle the deployment of releases

Repositories

The fifth step of this job is quite verbal. Let's first talk about the repositories. As we explained in the introduction, charts work the same way a package provided by a package manager does. At first a repository should be added to the instance, following that, the package manager needs to update the list of package, then and only then it will be able to download a package and install it locally.

Helm is like that with a key difference, they both have repositories on your local instance, but it does not install packages locally, it does so in your remote cluster (depending on your used Kubernetes context!). We could then create a single step about installing the repository into the Github Action worker and another one for the installation dry-run. That would be fine for a test environment as i tried on my own cluster, but as time goes by and other needs appear, you might need to install more than one Helm chart. It would be better to design the workflow for easy repeatability rather than having to add code snippets here and there.

That is why we will use a single step to handle everything from the repository install to the chart installation dry-run. This means there is only one place for an operator to copy/paste/modify the code to deploy a new chart.

Adding logic to the installation

Now we need to deconstruct a bit more this huge one liner we added in the dry-run step by formatting it into something more readable:

If the release exists in the cluster
then
  if there is a difference between the release and the dry-run 
  then
    run an dry-run upgrade of the existing release
  else
    exit the one-liner echoing the release has no changes
  end if
else 
  run a dry-run installation of the release
end if

Now why do we want this specific logic ? Just running plainly the helm upgrade or helm install commands on the CI even tho there is no changes will create a new release of the chart into the cluster. Helm 3 saves the configuration of the releases in secrets object into the same namespace as the chart. Now i'm sure that you can imagine that as times goes and the number of Pull Requests being merged into the main branch of the repository, the number of secrets will also increase and you will end up with something like this, which is not cool:

$ kubectl get secrets -n cert-manager
sh.helm.release.v1.cert-manager.v10   helm.sh/release.v1                    1      30d
sh.helm.release.v1.cert-manager.v11   helm.sh/release.v1                    1      29d
sh.helm.release.v1.cert-manager.v12   helm.sh/release.v1                    1      21d
sh.helm.release.v1.cert-manager.v13   helm.sh/release.v1                    1      21d
sh.helm.release.v1.cert-manager.v14   helm.sh/release.v1                    1      21d
sh.helm.release.v1.cert-manager.v5    helm.sh/release.v1                    1      30d
sh.helm.release.v1.cert-manager.v6    helm.sh/release.v1                    1      30d
sh.helm.release.v1.cert-manager.v7    helm.sh/release.v1                    1      30d
sh.helm.release.v1.cert-manager.v8    helm.sh/release.v1                    1      30d
sh.helm.release.v1.cert-manager.v9    helm.sh/release.v1                    1      30d

The dry-run does not have any impact on this, but it is better to have a dry-run and a deployment workflow that are similar as close as possible for maintaining purposes.

Using environment variables

In order to be able to repeat a the installation process with another chart we need to be able to easily change the values we need for:

the version of the chart we want, so we won't be stuck using the latest available version of the chart. CHART_VERSION: v0.13.1.
the chart namespace, because we do not want everything to be deployed int the default namespace. CHART_NAMESPACE: cert-manager.
the name of the repository, in order to tell the step what the repository should be named. CHART_REPO: jetstack.
the URL of the repository so Helm knows where to find the chart on the internet. REPO_URL: https://charts.jetstack.io.
the name of the chart that will be installed into the cluster. CHART_NAME: cert-manager.
the name of the release, you can call it however you want as long as it make sense i guess ? RELEASE_NAME: cert-manager.
the path towards the values of the chart. VALUES_PATH: infrastructure/kubernetes/helm3/cert-manager/values.yaml

Environment variables are tied to the steps they are used with, there is therefore no need to worry of changing their values in a later step.

From Dry Run to Deploying

The only difference between those 2 workflows will be the triggering event, and the deployment one liner. For the triggering event, you can change it to:

on: 
  push:
    branches: 
    - master

For the one liner, you will only need to remove the --dry-run flag from the command as such:

if helm list -n $RELEASE_NAME | grep $RELEASE_NAME > /dev/null ; then if ! helm diff upgrade $RELEASE_NAME $CHART_REPO/$CHART_NAME --namespace $CHART_NAMESPACE --version $CHART_VERSION -f $VALUES_PATH --detailed-exitcode; then helm upgrade $RELEASE_NAME $CHART_REPO/$CHART_NAME --namespace $CHART_NAMESPACE --version $CHART_VERSION -f $VALUES_PATH; else echo "There is no changes for this release."; fi; else helm install $RELEASE_NAME $CHART_REPO/$CHART_NAME --namespace $CHART_NAMESPACE --version $CHART_VERSION -f $VALUES_PATH; fi

You now have a nice and proper Helm dry-run and deployment CI/CD pipeline!