Integrating Kubernetes Secrets with AWS Secrets Manager using External Secrets Operator (ESO)

In this story, we will learn how to use External Secrets Operator (ESO) to configure and create secrets backed by AWS Secrets Manager.

ESO is a Kubernetes operator used to integrate the Kubernetes cluster with the external management tools like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault etc to read the information and automatically injects in to a Kubernetes secret.

Prerequisites:

1. AWS Active Subscription
2. AWS EKS Cluster
3. Tools installed: kubectl, helm, eksctl, aws-cli

Step 1: Create a secret in AWS Secrets Manager

With the AWS Secrets Manager as external tool, we are ready to create a secret in AWS. As the Secrets Manager supports the secret in JSON format with key/value pairs, I have chosen below JSON object to store as a secret with name as dev/backend_credentials.

{
  "backend": {"username": "dbadmin", "password": "@WftEst&esuDKU"}
}

Step 2: Create an AWS IAM Identity Provider for the EKS Cluster

Here I am using EKS Service Account credentials for AWS Authentication. To use AWS Identity and Access Management (IAM) roles for service accounts, an IAM OIDC provider must exist for your cluster. After following the instructions mention here, below OIDC provider is created.

Step 3: Create an AWS IAM Policy and IAM Role

Now that we have the secret and OIDC provider, lets create the IAM Role with below custom IAM policy and Trust Relationship and name it as secretsManagerRole.

Trust Relationships:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<<account-id>>:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/D2E46F43348E6E48690F38DB202562A3"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/D2E46F43348E6E48690F38DB202562A3:sub": "system:serviceaccount:eso:eso-service-account"
                }
            }
        }
    ]
}

Custom Inline Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [ "*" ]
        }
    ]
}

Step 4: Install ESO Helm chart

In order to customize the ESO chart installation with IAM role we have created above, I have created the below values.yaml file and passed as an input to the helm install command.

values.yaml:

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<<account-id>>:role/secretsManagerRole"
  name: "eso-service-account"

Create a namespace eso in EKS cluster and install the ESO helm chart under that namespace

$ kubectl create ns eso
namespace/eso created
Step 5: Create SecretStore Resource in the EKS cluster

Finally add the helm repo for external secrets operator and then install the helm chart.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets-operator external-secrets/external-secrets -n eso --values ./values.yaml

Verify the deployment and ensure there are no errors.

$ kubectl get all -n eso
NAME                                                           READY   STATUS    RESTARTS   AGE
pod/external-secrets-operator-759f5486fb-c7gqf                 1/1     Running   0          78s
pod/external-secrets-operator-cert-controller-5f94555d-7hxfz   1/1     Running   0          78s
pod/external-secrets-operator-webhook-5c6ffc6c57-gzp4d         1/1     Running   0          78s

NAME                                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/external-secrets-operator-webhook   ClusterIP   10.100.191.216   <none>        443/TCP   78s

NAME                                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/external-secrets-operator                   1/1     1            1           78s
deployment.apps/external-secrets-operator-cert-controller   1/1     1            1           78s
deployment.apps/external-secrets-operator-webhook           1/1     1            1           78s

NAME                                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/external-secrets-operator-759f5486fb                 1         1         1       78s
replicaset.apps/external-secrets-operator-cert-controller-5f94555d   1         1         1       78s
replicaset.apps/external-secrets-operator-webhook-5c6ffc6c57         1         1         1       78s

and also verify the service account is created successfully.

$ kubectl get sa -n eso
NAME                               SECRETS   AGE
default                            1         10m
eso-service-account                1         25s
external-secrets-cert-controller   1         25s
external-secrets-webhook           1         25s

Step 5: Create SecretStore Resource in the EKS cluster

We are ready with all the cluster add-ons provisioned and installed successfully. Lets create ESO API Custom Resources (SecretStore and ExternalSecret) . SecretStore is one such resource which specifies how to access external API. One instance of SecretStore resource maps to one instance of external API.

secretStore.yaml:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-backend-credentials
  namespace: eso
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-service-account

Run below command to create the resource

$ kubectl create -f secretStore.yaml
secretstore.external-secrets.io/secretstore-backend-credentials created

Ensure the resource is created without any errors

$ kubectl get ss -n eso
NAME                              AGE   STATUS   READY
secretstore-backend-credentials   13m   Valid    True

Step 6: Create ExternalSecret Resource in the EKS cluster

The other API resource is ExternalSecret, which describes what data to be fetched, how frequently and how the data needs to be transformed and saved as a Kubernetes Secret. Repeat the steps mentioned in step#5 with below resources and commands.

externalSecret.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret-backend-credentials
  namespace: eso
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: secretstore-backend-credentials
    kind: SecretStore
  target:
    name: secret-backend-credentials
    creationPolicy: Owner
  data:
    - secretKey: dbUsername
      remoteRef:
        key: dev/backend_credentials
        property: backend.username 
    - secretKey: dbPassword
      remoteRef:
        key: dev/backend_credentials
        property: backend.password

$ kubectl create -f externalSecret.yaml
externalsecret.external-secrets.io/external-secret-backend-credentials created
$
$ kubectl get es -n eso
NAME                                  STORE                             REFRESH INTERVAL   STATUS         READY
external-secret-backend-credentials   secretstore-backend-credentials   1m                 SecretSynced   True

Note here that additionally there is Kubernetes Secret created with the name given in spec.targe.name of the externalSecret manifest file.

$ kubectl get secrets -n eso
NAME                                              TYPE                                  DATA   AGE
default-token-nbqcs                               kubernetes.io/service-account-token   3      29m
eso-service-account-token-jvrjc                   kubernetes.io/service-account-token   3      19m
external-secrets-cert-controller-token-gzbv6      kubernetes.io/service-account-token   3      19m
external-secrets-operator-webhook                 Opaque                                4      19m
external-secrets-webhook-token-thmqb              kubernetes.io/service-account-token   3      19m
secret-backend-credentials                        Opaque                                2      82s
sh.helm.release.v1.external-secrets-operator.v1   helm.sh/release.v1                    1      19m

Step 7: Validate

Let’s retrieve the values of the secret by decoding the base64 values. Verify that the credentials are matching with what we have created in step#1.

$ kubectl get secret secret-backend-credentials -n eso -o yaml
apiVersion: v1
data:
  dbPassword: QFdmdEVzdCZlc3VES1U=
  dbUsername: ZGJhZG1pbg==
immutable: false
kind: Secret
metadata:
  annotations:
    reconcile.external-secrets.io/data-hash: 851f34a0f606bd77b41455945deb84ee
  creationTimestamp: "2022-11-21T23:47:41Z"
  name: secret-backend-credentials
  namespace: eso
  ownerReferences:
  - apiVersion: external-secrets.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ExternalSecret
    name: external-secret-backend-credentials
    uid: a1da20c2-8161-4f64-9c8b-817a3afcb7a2
  resourceVersion: "7024"
  uid: 11c8f51a-0bf3-41aa-9e0c-8ef46bb53039
type: Opaque
$
$
$ echo "QFdmdEVzdCZlc3VES1U=" | base64 -d
@WftEst&esuDKU
$
$ echo "ZGJhZG1pbg==" | base64 -d
dbadmin

Step 8: Refresh the Secrets and Validate

In this step, lets modify the secret values and validate if they are automatically refreshed in EKS. I have changed the password to newvalue.

Retrieve the password value from the EKS cluster and verify it is matching.

$ kubectl get secret secret-backend-credentials -n eso -o yaml
apiVersion: v1
data:
  dbPassword: bmV3dmFsdWU=
  dbUsername: ZGJhZG1pbg==
immutable: false
kind: Secret
metadata:
  annotations:
    reconcile.external-secrets.io/data-hash: deedba3d134320963742df456552ade9
  creationTimestamp: "2022-11-21T23:47:41Z"
  name: secret-backend-credentials
  namespace: eso
  ownerReferences:
  - apiVersion: external-secrets.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ExternalSecret
    name: external-secret-backend-credentials
    uid: a1da20c2-8161-4f64-9c8b-817a3afcb7a2
  resourceVersion: "8239"
  uid: 11c8f51a-0bf3-41aa-9e0c-8ef46bb53039
type: Opaque
$
$ echo "bmV3dmFsdWU=" | base64 -d
newvalue

Step 9: Clean up

Delete the Kubernetes resources first and following that delete the IAM Role and IAM OIDC provider .

$ kubectl delete -f secretStore.yaml
secretstore.external-secrets.io "secretstore-backend-credentials" deleted

yevke@DESKTOP-MDGA8T8 MINGW64 ~/story1
$ kubectl delete -f externalSecret.yaml
externalsecret.external-secrets.io "external-secret-backend-credentials" deleted

$ helm delete external-secrets-operator -n eso
release "external-secrets-operator" uninstalled

$ kubectl get all -n eso
No resources found in eso namespace.

$ kubectl delete ns eso
namespace "eso" deleted