Managing secrets in Kubernetes is not a straightforward thing.

Creating them manually via command line is a bad practice and makes problem managing, updating, and adding passwords. There are quite a few tools that address this problem, the most popular of them is Hashicorp Vault. While I very much like what Hashicorp is doing and what Vault provides, it generates a big overhead in managing it and most of the systems/companies don’t need it so I wanted do use something simple, manageable and secure for all of our workloads at The Remote Company.

A little of the background - we use Helm charts for each of our apps. They are deployed to GKE with Github Actions, utilizing Skaffold as a build/deploy tool, so we wanted something compatible with it. Queue the Helm Secrets plugin from Zendesk - wonderful implementation of an even more wonderful Mozilla SOPS. SOPS is an editor from Mozilla able to encrypt the data with AWS KMS, GCP KMS, Azure Key Vault, and PGP. Originally, it was written in Python, but Mozilla rewrote it in Golang making it easier to build and redistribute as Golang compiles everything into a single binary. Helm secrets is a helm plugin that uses SOPS in the background to encrypt/decrypt the data, but also automatically applies the secrets to Kubernetes cluster and attaches it to the deployment on demand (we will cover it a bit later).

The instructions that follow are how you would manage it from macOS.

The tools that we will be using are:

  • Helm
  • Helm secrets
  • Skaffold
  • GCloud SDK
  • Mozilla SOPS
  • Kubectl

Dependency setup

Install dependencies via brew.

brew cask install google-cloud-sdk
brew install kubectl helm skaffold sops
helm plugin install https://github.com/zendesk/helm-secrets

Login to our Google Cloud project. Follow the instructions on setting up your project and compute region and zone.

gcloud auth login
gcloud auth application-default login

Creating KMS key

Create Google KMS key used for encryption and decryption of the content

gcloud kms keyrings create sops --location global
gcloud kms keys create sops-key --location global --keyring sops --purpose encryption
gcloud kms keys list --location global --keyring sops

# You should see similar output to

NAME                                                                  PURPOSE          PRIMARY_STATE
projects/my-project/locations/global/keyRings/sops/cryptoKeys/sops-key ENCRYPT_DECRYPT  ENABLED

Setup secrets

You should have a helmchart in your project. In your helmchart folder, create .sops.yaml file.

creation_rules:
  - gcp_kms: "projects/my-project/locations/global/keyRings/sops/cryptoKeys/sops-key"

In order to easily diff in git, you would need .gitattributes in the same folder with sopsdiffer.

secrets.yaml diff=sopsdiffer
secrets.*.yaml diff=sopsdiffer

In our helmchart, we will need secrets file. When you start, you can add secrets as plain text in that file, later on, we will encrypt them with SOPS. In this example, we will create secrets.dev.yaml in our helmchart folder.

secrets:
  first-password: very-secure
  second-password: much-locked
  third-password: so-encrypted

Now we can encrypt the secrets file. The file will be encrypted with KMS key.

helm secrets enc secrets.dev.yaml

It should look something like this:

secrets:
  first-password: ENC[AES256_GCM,data:0DKw4JAoAv8kg139TfEatrrhdVEhBKFtFYpuNBVlxDqd4qc6u6SaK1MbsGUqHLuuMhyCuUNl2MepmMwm2QsgVqx7euBIeqKSe6ZUYdiAKuwFrGpLzQJ==,type:str]
  second-password: ENC[AES256_GCM,data:Mv1Frwz6CT2Jnq5EmBgZJl6sn7cYzXFceLSlOiu5djguJi6cRJAW90cpIkwpcuvYc3ulyZagoB7vfO2nL17IOa3hZ4GZRhzXZXqo0wDAgXFjucR1L4B==,type:str]
  third-password: ENC[AES256_GCM,data:B9FxMPPSOisPrlnsDfwilwtagxVfWbqCwWJw7XsfqElRqitvCpkB3anvj1uW0hpruJ96LwAJDwplqu8iaZmz14OyeKMLREuCpwRx5b4OhEUbRGTtESA==,type:str]
sops:
  kms: []
  gcp_kms:
    - resource_id: projects/my-project/locations/global/keyRings/sops/cryptoKeys/sops-key
      created_at: "2020-09-04T11:21:03Z"
      enc: somevalue=
  azure_kv: []
  hc_vault: []
  lastmodified: "2020-09-30T15:12:35Z"
  mac: ENC[AES256_GCM,data:somevalue==,type:str]
  pgp: []
  unencrypted_suffix: _unencrypted
  version: 3.6.1

You can see that our secrets are now encrypted and they can be referenced in helm template.

Referencing secrets in helm values

First, we need to create a template for helm so it can iterate over your secrets and add them automatically. Here’s the example of the template:

apiVersion: v1
kind: Secret
metadata:
  name: {{ include "app.fullname" . }}
  labels: {{- include "app.labels" . | nindent 4 }}
type: Opaque
data:
  {{- range $key, $val := .Values.secrets }}
  {{ $key }}: {{ $val | b64enc | quote }}
  {{- end }}

This template goes over secrets file, looks for the secrets key and then adds all the keys referenced with their values to a Secret object in k8s.

Usually, your secrets are in env and you would have a template in helm that can iterate over your env variables and attach them to the deployment. Here is one example of a template for k8s deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "app.fullname" . }}
  labels:
    {{- include "app.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "app.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "app.selectorLabels" . | nindent 8 }}
    spec:
    {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
    {{- end }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image }}"
          imagePullPolicy: {{ .Values.imagepullPolicy }}
          {{- if .Values.volumeMounts }}
          volumeMounts:
            {{- range .Values.volumeMounts }}
            - name: {{ .name }}
              mountPath: {{ .mountPath }}
              {{- if .subPath }}
              subPath: {{ .subPath }}
              {{- end }}
              readOnly: {{ .readOnly }}
            {{- end }}
          {{- end }}
          env:
            {{- range .Values.env }}
            - name: "{{ .name }}"
              value: "{{ .value }}"
            {{- end }}
            {{- range .Values.envExtra }}
            - name: "{{ .name }}"
              value: "{{ .value }}"
            {{- end }}
            {{- range .Values.secretEnv }}
            - name: "{{ .name }}"
              valueFrom:
                secretKeyRef:
                  name: "{{ .secretKeyRefName }}"
                  key: "{{ .secretKeyRefKey }}"
            {{- end }}
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /probe
              port: http
          readinessProbe:
            httpGet:
              path: /probe
              port: http
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- if .Values.volumeMounts }}
      volumes:
        {{- range .Values.volumeMounts }}
        - name: {{ .name }}
          secret:
            secretName: {{ .name }}
        {{- end }}
      {{- end }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
    {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
    {{- end }}

You can see env and secretEnv values, which we will use to define env and secret env variables in our values file.

env:
  - name: SOME_GREAT_ENV
    value: "notasecretvariable"
secretEnv:
  - name: COOL_ENV
    secretKeyRefName: app
    secretKeyRefKey: first-password
  - name: SLIGHTLY_LESS_COOL_ENV
    secretKeyRefName: app
    secretKeyRefKey: second-password
  - name: CHILL_ENV
    secretKeyRefName: app
    secretKeyRefKey: third-password

As you can see, we added a simple env which is not a secret, and then used secretEnv to add a couple of variables that we have in our secret. secretKeyRefName in this case will be the name of the helm release, as our secrets template will create an object under that name. Next, we will just use the key that we entered into our secrets.dev.yaml to let Kubernetes know which env should be referenced to which secret. After that, we are ready to go for deployment.

Deploying with Skaffold

I won’t get into the whole flow with Github Actions, Helm charts, and Skaffold, but will show you how you can simply tell Skaffold to utilize Helm secrets automatically.

apiVersion: skaffold/v1
kind: Config
profiles:
  - name: dev
    build:
      artifacts:
        - image: our-repo/our-image
          docker:
            dockerfile: docker/Dockerfile
      tagPolicy:
        gitCommit: {}
    deploy:
      helm:
        releases:
          - name: app
            chartPath: helmchart
            namespace: app
            values:
              image: our-repo/our-image
            valuesFiles:
              - helmchart/values.dev.yaml
              - helmchart/secrets.dev.yaml
            useHelmSecrets: true
            wait: true
        flags:
          global: []
          upgrade: ["--debug", "--install", "--timeout", "6000s"]

A couple of important things here:

  • You need to add useHelmSecrets: true so Skaffold knows that you want to utilize Helm secrets plugin and that it can apply the secrets automatically
  • valuesFiles needs to contain both a path to values and secrets file in order for a build to succeed.
  • Your Google IAM user must have KMS Encrypter role (if you want to encrypt the data), and KMS Decrypter role (used for CIs as they don’t need to encrypt the data, just decrypt).
  • You would need to login to your GKE cluster before running Skaffold.

Afterward you would just call skaffold run -p dev and watch the tools do its work. If you want to edit, or add some value, you would simply run helm secrets edit helmchart/secrets.dev.yaml and it would open the decrypted file in your $EDITOR. You can add data, change passwords, etc, and save the file and it would automatically get encrypted so you can commit it to git and push it to your CI.

This sounds like a lot of steps to manage “such a simple thing” as secrets, but you only need to do this when you start a project and forget about it. If you have any questions - let me know in the comments.