Securing Kubernetes Secrets with Kubeseal: A Hands-On Guide#

Kubernetes Secrets management presents a significant challenge, especially in GitOps workflows where tracking infrastructure as code is essential. While Kubernetes provides the Secret resource for managing sensitive data, these objects are only base64-encoded, not encrypted. This limitation creates problems when you need to store configuration in version control.

Enter Sealed Secrets and its CLI tool, Kubeseal. This practical guide will walk you through setting up and using this solution to securely manage your Kubernetes Secrets.

Problem Statement#

In a GitOps workflow:

• You need to store Kubernetes manifests in Git
• Standard Kubernetes Secrets are only base64-encoded
• Storing Secrets in Git would expose sensitive information

Sealed Secrets solves this by:

• Encrypting your Secret data using asymmetric cryptography
• Allowing you to safely commit the encrypted “SealedSecret” to Git
• Providing a controller that can decrypt the Secret in the cluster

Prerequisites#

• A Kubernetes cluster with admin access
• kubectl configured to communicate with your cluster
• Helm (optional, for controller installation)

Installation#

1. Install the Sealed Secrets Controller#

The controller needs to be installed in your cluster. There are two primary methods:

Using Helm:

# Add the Sealed Secrets Helm repository
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets

# Update repositories
helm repo update

# Install the Sealed Secrets controller
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

Using kubectl:

# Apply the latest release
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.21.0/controller.yaml

Verify the installation:

kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets

2. Install the Kubeseal CLI#

On macOS:

brew install kubeseal

On Linux:

# Download the appropriate version for your platform
KUBESEAL_VERSION=0.21.0
wget "https://github.com/bitnami-labs/sealed-secrets/releases/download/v${KUBESEAL_VERSION}/kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz"
tar -xvzf kubeseal-${KUBESEAL_VERSION}-linux-amd64.tar.gz
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

On Windows:

Download the binary from the GitHub releases page ↗ and add it to your PATH.

Using Kubeseal#

Basic Workflow#

The workflow consists of:

1. Creating a regular Kubernetes Secret
2. Sealing it with Kubeseal
3. Applying the SealedSecret to your cluster

Let’s walk through a complete example:

1. Create a Regular Kubernetes Secret#

Create a file named my-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
  namespace: my-application
type: Opaque
stringData:
  username: admin
  password: supersecret-password

2. Seal the Secret#

Use the kubeseal command to encrypt the Secret:

kubeseal -f my-secret.yaml -w sealed-secret.yaml

This creates a file named sealed-secret.yaml containing the encrypted version. It will look something like:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: database-credentials
  namespace: my-application
spec:
  encryptedData:
    password: AgBy8hCF8...truncated...
    username: AgAk4Zx1g...truncated...
  template:
    metadata:
      name: database-credentials
      namespace: my-application
    type: Opaque

3. Apply the SealedSecret to Your Cluster#

Now you can safely commit this encrypted SealedSecret to your Git repository and apply it to your cluster:

kubectl apply -f sealed-secret.yaml

The controller in your cluster will:

1. Detect the new SealedSecret
2. Decrypt it using its private key
3. Create a regular Kubernetes Secret with the decrypted data

4. Verify the Secret was Created#

kubectl get secret database-credentials -n my-application -o yaml

Your applications can now use this Secret normally, just like any other Kubernetes Secret.

Advanced Usage#

Using Different Scopes#

By default, SealedSecrets are scoped to a specific name and namespace. You can change this behavior using the --scope flag:

# Namespace-wide: can be unsealed with any name within the same namespace
kubeseal -f my-secret.yaml -w sealed-secret.yaml --scope namespace-wide

# Cluster-wide: can be unsealed in any namespace
kubeseal -f my-secret.yaml -w sealed-secret.yaml --scope cluster-wide

Working with Multiple Clusters#

Each Sealed Secrets controller has its own encryption key. To seal a secret for a specific cluster:

# Fetch the public key from the target cluster
kubeseal --fetch-cert --controller-name=sealed-secrets --controller-namespace=kube-system > pub-cert.pem

# Use the certificate to seal the secret
kubeseal -f my-secret.yaml -w sealed-secret.yaml --cert pub-cert.pem

Updating Existing Secrets#

When you need to update a Secret:

1. Create a new regular Secret with the updated values
2. Seal it to create a new SealedSecret
3. Apply the new SealedSecret to the cluster

The controller will update the existing Secret with the new values.

Key Rotation#

For security reasons, you might want to rotate the encryption keys periodically:

kubectl exec -n kube-system -l app.kubernetes.io/name=sealed-secrets -it -- sealed-secrets-controller --key-renew=true

After key rotation, existing SealedSecrets will still work, but new ones will be encrypted with the new key.

Best Practices#

1. Namespace in metadata: Always include the namespace in the SealedSecret metadata to ensure it’s deployed to the correct namespace.

2. Secret naming convention: Use a consistent naming convention for your Secrets to make them easily identifiable.

3. Backup the controller’s encryption key: If you lose this, you won’t be able to decrypt your SealedSecrets.

   kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.yaml

   bash

4. Version control: Maintain a history of your SealedSecrets in Git to track changes over time.

5. Avoid frequent updates: Each update creates a new encrypted value, cluttering your Git history.

Troubleshooting#

Secret Not Decrypting#

If your SealedSecret doesn’t decrypt into a Secret:

1. Check controller logs:

   kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets

   bash

2. Verify the SealedSecret is in the same namespace as defined in its metadata.

3. Ensure the controller has the correct decryption key.

Recovering From Key Loss#

If you’ve lost the controller’s key, but have a backup:

1. Delete the current controller:

   kubectl delete -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.21.0/controller.yaml

   bash

2. Apply the backed-up secret key:

   kubectl apply -f sealed-secrets-key.yaml

   bash

3. Reinstall the controller:

   kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.21.0/controller.yaml

   bash

Conclusion#

Sealed Secrets provides a robust solution for managing Kubernetes Secrets in GitOps workflows. With Kubeseal, you can safely encrypt your sensitive data and store it in version control, making your infrastructure truly code-driven while maintaining security.

Remember, while Sealed Secrets significantly improves your security posture, it’s just one component of a comprehensive security strategy. Always follow the principle of least privilege and regularly audit your secrets management practices.