Managing Secrets Securely in Kubernetes

The Problem with Default Kubernetes Secrets

By default, Kubernetes Secrets store data as base64-encoded values, which is not true encryption.

# Default secrets can be decoded easily
kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d

Using External Secrets Operator

Installation

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets

Connecting to AWS Secrets Manager

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1

Creating an ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: prod/database
        property: username
    - secretKey: password
      remoteRef:
        key: prod/database
        property: password

Best Practices

1. Apply Strict RBAC

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["specific-secret"]
  verbs: ["get"]

2. Enable Encryption at Rest

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: <BASE64_ENCODED_SECRET>
    - identity: {}

Conclusion

Managing secrets securely is essential to maintaining a Kubernetes cluster. Using External Secrets Operator together with a cloud provider’s secrets manager greatly enhances security.