How to add TLS termination in Kubernetes (GKE) using a sidecar container approach with Traefik v2.

GKE is a great product, Google clearly have put the work in to make their managed Kubernetes offering as streamlined as possible, but nothing is perfect and if you want to expose your HTTPS service on a port other than 80 or 443 then you’re going to have to find an alternative to a GKE Ingress.

There are many solutions to this problem eg. use another ingress provider or perhaps a service but they tend to be complicated, are overkill for the problem or add load to your cluster that could be put to better use.

The simplest solution is to use a Kubernetes service of the LoadBalancer variety, this deploys an L4 load balancer which allows you to expose any port you desire. Unfortunately an L4 load balancer has no concept of TLS and thus can’t offload the burden of encrypting and decrypting your secure communication. One solution is to deploy a sidecar container alongside your container to do the TLS termination within the deployment. In this tutorial we will use Traefik for this purpose.

Initial deployment

To start off we need some service that we want to expose to a non standard port. A simple whoami service will do for us in this case. The following yaml creates a namespace to put our deployment in, creates the deployment and adds a service to expose the deployment. Note that it’s not possible to use a standard Kubernetes Ingress manifest for this purpose as it’s only compatible with ports 80 and 443 when using GKE load balancers. Instead we use a LoadBalancer service.

manifests/whoami-v1.yml

apiVersion: v1
kind: Namespace
metadata:
  name: traefik-sidecar
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: traefik-sidecar
  name: whoami
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami
        image: containous/whoami
---
apiVersion: v1
kind: Service
metadata:
  namespace: traefik-sidecar
  name: whoami
spec:
  type: LoadBalancer
  ports:
  - name: http
    targetPort: 80
    port: 80
  selector:
    app: whoami

Great, now we have a service, we can get the external IP by running the command kubectl get svc and looking for the external ip --in this case 35.195.132.57

λ  traefik-sidecar-proxy kubectl get service
NAME              TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
whoami            LoadBalancer   10.63.253.70    35.195.132.57    80:31671/TCP     1m

We can then see the page by navigating to the given IP address

Insecure WhoAmI via IP

Adding DNS and configuring a TLS certificate

Great, but now the NSA can see the content of the page as it flies through their routers somewhere deep in the internet, we have to add some TLS encryption to keep our data safe from prying eyes. To do this we will first need to create an A record in our DNS pointing to the load balancer's IP address. How to do this will depend on your DNS provider. Once done you should be able to navigate to the url and see the same page as before.

Insecure WhoAmI via domain

Now that DNS is sorted we need a certificate, the easiest way to achieve this is with letsencrypt and cert-manager. I will assume you have this setup already but if you don’t the installation instructions are here. For the challenge we will be using HTTP-01, this is a simpler approach that requires fewer permissions to achieve. Apply the following manifest to create an Issuer to verify your certificate and a Certificate..

manifests/certs.yml

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  namespace: traefik-sidecar
  name: traefik-sidecar-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: production-issuer-secret
    solvers:
    - http01:
       ingress:
         class: "gce"
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  namespace: traefik-sidecar
  name: traefik-sidecar
spec:
  secretName: traefik-sidecar-secret-tls
  dnsNames:
  - traefik-sidecar.bigdataboutique.com
  issuerRef:
    name: traefik-sidecar-issuer
    kind: Issuer
    group: cert-manager.io

This will create an ingress object in order for Let’s Encrypt to verify that you are allowed to create this certificate. Find its IP like so;

λ  traefik-sidecar-proxy kubectl get ingress
NAME            CLASS    HOSTS                                 ADDRESS        PORTS     AGE
pulse-ingress   <none>   traefik-sidecar.bigdataboutique.com   62.542.82.11   80        29d

Point your A record at that IP and wait for the certificate to resolve. You can run kubectl get certificate -w to watch the certificate object, wait for the Ready column to switch to True.

Finally change your A record back to the IP of your L4 load balancer, use kubectl get service if you forgot it.

Configuring Traefik V2 as a TLS terminating proxy

And with all that in place it’s time to add the sidecar! First we need to create a few Traefik configuration files, these are split into two parts; static and dynamic. The static file configures global properties like entrypoints, providers and dashboard configuration, while dynamic configuration contains how requests are handled. Dynamic configurations are so named because they can be dynamically changed during runtime without disrupting services to users. This terminology could be a little confusing given that our dynamic configuration will be static for this use case.

Entrypoints define the ports and addresses that our proxy should be listening on, while providers describe how the proxy will handle requests to the entry points. In this tutorial we will open port 9200 and be using a file provider. The configuration looks like this;

traefik/static.yml

entryPoints:
  app:
    address: :9200

providers:
  file:
    filename: /etc/traefik/dynamic.yml

Simple really, the dynamic configuration is a bit more complicated.

First since we are trying to host a http service we need to use a http block. Then within that we need to define the http services that our proxy is terminating TLS for; since we are using a sidecar approach this will be localhost and the port is whatever port your service is running on (note this is required to be different to the port you want to expose to the world).

Next we define the routers. Routers connect up the entrypoint of the proxy to the service you are trying to expose, rules to direct the incoming traffic and TLS options.

Finally we need to define a TLS section, this lets us customise how the proxy handles TLS and lets you specify where to find certificates and to restrict allowed cypher suites if you want to have a really secure deployment. Altogether the dynamic config looks something like this.

traefik/dynamic.yml

http:
  services:
    app:
      loadBalancer:
        servers:
        - url: "http://localhost:80/"
  routers:
    app:
      service: app
      entrypoints:
      - app
      rule: Host(`{{ env "HOSTNAME" }}`)
      tls:
        certresolver: resolver
        domains:
          - main: "{{ env "HOSTNAME" }}"


tls:
  certificates:
    - certFile: /etc/tls/tls.crt
      keyFile: /etc/tls/tls.key

Using the proxy as a sidecar

Now we modify the app deployment to include the Traefik proxy and add the appropriate environment variables and volumes. First we need to generate the configmap for our Traefik configuration, this can be done using the following command kubectl create configmap traefik-sidecar-config --from-file=./traefik --namespace traefik-sidecar

manifests/whoami-v1.yml

apiVersion: v1
kind: Namespace
metadata:
  name: traefik-sidecar
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: traefik-sidecar
  name: whoami
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
      - name: whoami
        image: containous/whoami
      - name: "reverse-proxy"
        image: "traefik:v2.4.8"
        env:
        - name: HOSTNAME
          value: traefik-sidecar.bigdataboutique.com
        volumeMounts:
        - name: "tls"
          mountPath: "/etc/tls/"
        - name: "traefik-config"
          mountPath: "/etc/traefik/"
      volumes:
        - name: "tls"
          secret:
            secretName: "traefik-sidecar-secret-tls"
        - name: "traefik-config"
          configMap:
            name: "traefik-sidecar-config"
---
apiVersion: v1
kind: Service
metadata:
  namespace: traefik-sidecar
  name: whoami
spec:
  type: LoadBalancer
  ports:
  - name: http
    targetPort: 9200
    port: 9200
  selector:
    app: whoami

And we’re done. Navigate to the domain and you will see SSL protection provided by Let’s Encrypt.

Secure WhoAmI via domain

So today we have learned to deploy a service in GKE with TLS termination using a sidecar proxy approach, I hope this was helpful!