Cristian Greco

I write code sometimes.

6 minute read

We’re ready to deploy some apps to our cluster, but we haven’t yet figured out how to store sensitive data.

When we deploy our observability stack (Prometheus, Grafaka and Loki), we’ll need to create an admin user to be able to login. We’ll also need to define an email address where we’ll receive our alerts. This is not information we want to make public, yet it would be ideal if we could store it in source control to track it, change it, and share it with other contributors. For example maybe you want your colleague to rotate the password, or maybe they also want to be emailed when alerts fire.

The concept we’re looking for here is called "encryption at rest".

Encryption at rest is a process that protects data that is stored on a disk or backup media (in our case in files in our Git repository) by scrambling it so that it can only be decrypted with a key.

To achieve this we’ll be using Bitnami’s sealed secrets, the use of which is also well documented by Flux.

Create the Flux Manifests

The source code is available on GitHub.

In a previous post, we set up a Kubernetes cluster on our home server and ran Flux bootstrap. Flux created a Git repository in your name, which we now need to clone down and modify. Its content will look like this:

.
└── clusters
    └── production
        └── flux-system
            ├── gotk-components.yaml
            ├── gotk-sync.yaml
            └── kustomization.yaml

4 directories, 3 files
1 You can define states for several clusters. In our case we’ll just have the one. We’ll call our home server "production".

Create /clusters/production/infrastructure.yaml:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-controllers
  namespace: flux-system
spec:
  interval: 10m0s (1)
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers (2)
  prune: true (3)
  wait: true (4)
  timeout: 5m0s
1 Flux will monitor the Git repository for changes at the given path every 10 minutes. This means that when you push a change, it can take up to 10 minutes for Flux to detect the change and start reconciling. This interval can be made more or less aggressive depending on your needs.
2 The path in the Git repository to monitor for changes. Note that we are not pointing to a specific file, rather to a whole directory. That’s because we can have many infrastructure components. Flux will look for a kustomization.yaml file in this directory, similar to an index.html for a website. We’ll create this file shortly.
3 Flux will remove the resources if they are removed from Git. This ensures the cluster state matches what’s in Git.
4 Flux will wait up to 5 minutes before marking the reconciliation as failed if it hasn’t completed within that time.

Create /infrastructure/controllers/sealed-secrets.yaml:

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: sealed-secrets
  namespace: flux-system
spec:
  interval: 24h (1)
  url: https://bitnami-labs.github.io/sealed-secrets

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: sealed-secrets
  namespace: flux-system
spec:
  interval: 30m (2)
  chart:
    spec:
      chart: sealed-secrets
      sourceRef:
        kind: HelmRepository
        name: sealed-secrets
      version: "2.x" (3)
  releaseName: sealed-secrets-controller
  targetNamespace: flux-system
  install:
    crds: Create
  upgrade:
    crds: CreateReplace
1 Poll the Bitnami Helm chart repository every 24 hours for new releases.
2 Every 30 minutes, check if a new release matching our criteria is available.
3 The versions we will allow Flux to deploy. 2.x means if our current version is 2.0.0, we’ll automatically accept an upgrade to 2.0.1 or 2.1.0. By keeping the major version the same, we shouldn’t run into any breaking changes. This assumption is based on semantic versioning.

Create /infrastructure/controllers/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - sealed-secrets.yaml

The resultant structure should look like this:

.
├── clusters
│   └── production
│       ├── flux-system
│       │   ├── gotk-components.yaml
│       │   ├── gotk-sync.yaml
│       │   └── kustomization.yaml
│       └── infrastructure.yaml (1)
└── infrastructure (2)
    └── controllers
        ├── kustomization.yaml
        └── sealed-secrets.yaml

6 directories, 6 files

Apply the Flux Manifests

Before we apply the manifests, let’s SSH onto the server and start watching for Flux events:

flux events --watch (1)
1 flux events -w for short.

Now for the beauty of Flux. Simply commit and push the changes, and we’ll soon see Flux kick in:

REASON                  OBJECT                                  MESSAGE
NewArtifact             GitRepository/flux-system               stored artifact for commit 'Move infra-controllers to root'
ReconciliationSucceeded Kustomization/flux-system               Reconciliation finished in 493.127254ms, next run in 10m0s
Progressing             Kustomization/infra-controllers         HelmRelease/flux-system/sealed-secrets created
                                                                HelmRepository/flux-system/sealed-secrets created
ChartPullSucceeded      HelmChart/flux-system-sealed-secrets    pulled 'sealed-secrets' chart with version '2.16.2'
ArtifactUpToDate        HelmChart/flux-system-sealed-secrets    artifact up-to-date with remote revision: '2.16.2'
NewArtifact             HelmRepository/sealed-secrets           stored fetched index of size 70.95kB from 'https://bitnami-labs.github.io/sealed-secrets'
HelmChartCreated        HelmRelease/sealed-secrets              Created HelmChart/flux-system/flux-system-sealed-secrets with SourceRef 'HelmRepository/flux-system/sealed-secrets'
InstallSucceeded        HelmRelease/sealed-secrets              Helm install succeeded for release flux-system/sealed-secrets-controller.v1 with chart sealed-secrets@2.16.2
ReconciliationSucceeded Kustomization/infra-controllers         Reconciliation finished in 20.341635841s, next run in 10m0s
Progressing             Kustomization/infra-controllers         Health check passed in 20.047899653s

Flux detected the change in Git. Observed that the cluster state differs from what’s in Git, and reconciled the difference. We see ReconciliationSucceeded, meaning everything went through. Indeed, there’s a new pod running in the cluster:

kubectl get pods -n flux-system -l app.kubernetes.io/name=sealed-secrets
NAME                                         READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-769d7d7bb4-x9n5h   1/1     Running   0          20h

Note also that Flux pulled version 2.16.2, which matched our version constraint in the HelmRelease manifest.

Create Our First Secret

When the sealed-secrets pod starts for the first time, it generates a public and private key pair. We can use the public key to encrypt our secrets. The encrypted secrets can be stored in Git without risk of being decrypted (as long as the private key is kept safe). The encrypted secrets can be applied into the cluster, where sealed-secrets is able to decrypt them with the private key and create regular secrets within the cluster. Here’s how that looks:

Diagram

Extract the Sealed Secrets Public Key

Kubeseal (which we installed in the first post with Ansible) can pull the public key directly from the sealed secrets pod. Let’s do that and save it to ~/pub-sealed-secrets.pem on the server:

kubeseal --fetch-cert \
    --controller-name=sealed-secrets-controller \
    --controller-namespace=flux-system \
    > ~/pub-sealed-secrets.pem

Create a Sealed Secret

Let’s create a test secret for demo purposes. First, we’ll create a regular secret containing our super secret password:

kubectl -n default create secret generic test-secret \
  --from-literal=password='super-secret-password' \
  --dry-run=client -o yaml \
  > test-secret.yaml

This will create a file called test-secret.yaml but not apply it into the cluster. It will have the following contents:

apiVersion: v1
data:
  password: c3VwZXItc2VjcmV0LXBhc3N3b3Jk
kind: Secret
metadata:
  creationTimestamp: null
  name: test-secret
  namespace: default

This manifest is not safe to commit. The password is just base64 encoded and can be easily decoded by anyone:

$ echo "c3VwZXItc2VjcmV0LXBhc3N3b3Jk" | base64 -d
super-secret-password

Let’s now turn test-secret.yaml into a sealed secret:

kubeseal --format=yaml --cert=pub-sealed-secrets.pem \
  < test-secret.yaml > test-secret-sealed.yaml

This will create a file called test-secret-sealed.yaml with the following contents:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: test-secret
  namespace: default
spec:
  encryptedData:
    password: AgBhMbZahfqsha0lmOR1HpJaueathee6/7snUl5WEz3PUqRTbTvwMu6ie48B870OhWMQYBVIYYnBmsFZDZHFWBBCHMMxxlVzc5y8Spn701PdruA5tBIYcQbiqDzdUOHIeAk3iASHaMuWoqaJgtx+Ovz+/kKeWbQqBlnbwN3AOzzWK8/0Ye9ZNZIVi1/U7yOdBIpltUB0AXJnb+7RnGGfkXL0n7CcBkxWbE19iwfjQt34QfCLktsQaxCDg+hwZVMnVkLnojs5XuB3eHeEIeMu8Xv8YkXtPZZIIlRUNYLbX1YqeK29auroZHjDhDfK0DbCDl+vlXNJdIoXQ3ac5qSczDyMfvLinp8emc602ksFjte95+lsMW8DvHY5/7BYihArMyjM3jmr/70QxjPgCcKTavhdhjQvJIG/e2YOyq7XxCR7ghU1QDdRPuVS9WWOxkVWSRD2Uk39IFO4+IvV0xViX8WfrCDTkJevAPGxnqnpc8SLlHyBa2mRGWEXIgrg+uX5xXfuU/Se7FFDBKDj4QZ39zNCgrmVvoQuPYXh1DMXBgETaF659rBq1Ur1dr4++C6v/MNYLBHGUPk1gJI17EH9L4RYkSqi62rI9H39knyiBJISwdOFo28fFpoZc2TMniJp5liHaOTDqMOT/jOZUPltA3tzBlu24U9OGSqFVu0tetUlk/3nLxvbDyW6s+P9vq3yAvUVX2G1QH9+0j0DN3MVOrC9PR3cW9c=
  template:
    metadata:
      creationTimestamp: null
      name: test-secret
      namespace: default

Hell yeah! Now this is a file that we can safely commit to Git, as only our sealed secrets pod can decrypt it. You can go ahead and delete the test-secret.yaml file, we don’t need it anymore.

Apply the Sealed Secret

kubectl apply -f test-secret-sealed.yaml

Let’s have a look at the sealed-secrets pod logs:

kubectl logs -n flux-system -l app.kubernetes.io/name=sealed-secrets
time=2024-11-23T23:47:53.552Z level=INFO msg=Updating key=default/test-secret
time=2024-11-23T23:47:53.572Z level=INFO msg="Event(v1.ObjectReference{Kind:\"SealedSecret\", Namespace:\"default\", Name:\"test-secret\", UID:\"fe5f0f76-3fd0-4d89-bb20-3843d7e205f1\", APIVersion:\"bitnami.com/v1alpha1\", ResourceVersion:\"43686\", FieldPath:\"\"}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully"
time=2024-11-23T23:47:53.583Z level=INFO msg="update suppressed, no changes in spec" sealed-secret=default/test-secret

The logs are telling us that a secret has been created. Let’s have a look:

kubectl get secrets -n default
NAME          TYPE     DATA   AGE
test-secret   Opaque   1      5s

Let’s have a look at test-secret:

kubectl get secret/test-secret -n default -o yaml
apiVersion: v1
data:
  password: c3VwZXItc2VjcmV0LXBhc3N3b3Jk
kind: Secret
metadata:
  creationTimestamp: "2024-11-23T23:47:53Z"
  name: test-secret
  namespace: default
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: test-secret
    uid: fe5f0f76-3fd0-4d89-bb20-3843d7e205f1
  resourceVersion: "43688"
  uid: 10824c16-390d-42e6-81ed-19906436fe9d
type: Opaque

That password looks similar to the base64 encoded password from earlier:

$ echo "c3VwZXItc2VjcmV0LXBhc3N3b3Jk" | base64 -d
super-secret-password

Success!

Summary

We’re able to encrypt secrets, which can be stored in source control. We can apply these encrypted secrets into the cluster, and they will be succesfully decrypted and applied as regular secrets within the cluster.

Next Steps

In the next post we’ll be deploying Prometheus, Grafana and Loki, with custom alerts that will email us when our server CPU/memory/disk utilisation, and even temperature, get too high.

You may also enjoy