Integrating Kubernetes Secrets with AWS Secrets Manager using External Secrets Operator (ESO)

In this story, we will learn how to use External Secrets Operator (ESO) to configure and create secrets backed by AWS Secrets Manager.

ESO is a Kubernetes operator used to integrate the Kubernetes cluster with the external management tools like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault etc to read the information and automatically injects in to a Kubernetes secret.

Prerequisites:

1. AWS Active Subscription
2. AWS EKS Cluster
3. Tools installed: kubectl, helm, eksctl, aws-cli

Step 1: Create a secret in AWS Secrets Manager

With the AWS Secrets Manager as external tool, we are ready to create a secret in AWS. As the Secrets Manager supports the secret in JSON format with key/value pairs, I have chosen below JSON object to store as a secret with name as dev/backend_credentials.

{
"backend": {"username": "dbadmin", "password": "@WftEst&esuDKU"}
}

Step 2: Create an AWS IAM Identity Provider for the EKS Cluster

Here I am using EKS Service Account credentials for AWS Authentication. To use AWS Identity and Access Management (IAM) roles for service accounts, an IAM OIDC provider must exist for your cluster. After following the instructions mention here, below OIDC provider is created.

Step 3: Create an AWS IAM Policy and IAM Role

Now that we have the secret and OIDC provider, lets create the IAM Role with below custom IAM policy and Trust Relationship and name it as secretsManagerRole.

Trust Relationships:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<<account-id>>:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/D2E46F43348E6E48690F38DB202562A3"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/D2E46F43348E6E48690F38DB202562A3:sub": "system:serviceaccount:eso:eso-service-account"
}
}
}
]
}

Custom Inline Policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
"Resource": [ "*" ]
}
]
}

Step 4: Install ESO Helm chart

In order to customize the ESO chart installation with IAM role we have created above, I have created the below values.yaml file and passed as an input to the helm install command.

values.yaml:


serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::<<account-id>>:role/secretsManagerRole"
name: "eso-service-account"

Create a namespace eso in EKS cluster and install the ESO helm chart under that namespace

$ kubectl create ns eso
namespace/eso created
Step 5: Create SecretStore Resource in the EKS cluster

Finally add the helm repo for external secrets operator and then install the helm chart.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets-operator external-secrets/external-secrets -n eso --values ./values.yaml

Verify the deployment and ensure there are no errors.

$ kubectl get all -n eso
NAME READY STATUS RESTARTS AGE
pod/external-secrets-operator-759f5486fb-c7gqf 1/1 Running 0 78s
pod/external-secrets-operator-cert-controller-5f94555d-7hxfz 1/1 Running 0 78s
pod/external-secrets-operator-webhook-5c6ffc6c57-gzp4d 1/1 Running 0 78s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/external-secrets-operator-webhook ClusterIP 10.100.191.216 <none> 443/TCP 78s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/external-secrets-operator 1/1 1 1 78s
deployment.apps/external-secrets-operator-cert-controller 1/1 1 1 78s
deployment.apps/external-secrets-operator-webhook 1/1 1 1 78s

NAME DESIRED CURRENT READY AGE
replicaset.apps/external-secrets-operator-759f5486fb 1 1 1 78s
replicaset.apps/external-secrets-operator-cert-controller-5f94555d 1 1 1 78s
replicaset.apps/external-secrets-operator-webhook-5c6ffc6c57 1 1 1 78s

and also verify the service account is created successfully.

$ kubectl get sa -n eso
NAME SECRETS AGE
default 1 10m
eso-service-account 1 25s
external-secrets-cert-controller 1 25s
external-secrets-webhook 1 25s

Step 5: Create SecretStore Resource in the EKS cluster

We are ready with all the cluster add-ons provisioned and installed successfully. Lets create ESO API Custom Resources (SecretStore and ExternalSecret) . SecretStore is one such resource which specifies how to access external API. One instance of SecretStore resource maps to one instance of external API.

secretStore.yaml:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: secretstore-backend-credentials
namespace: eso
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: eso-service-account

Run below command to create the resource

$ kubectl create -f secretStore.yaml
secretstore.external-secrets.io/secretstore-backend-credentials created

Ensure the resource is created without any errors

$ kubectl get ss -n eso
NAME AGE STATUS READY
secretstore-backend-credentials 13m Valid True

Step 6: Create ExternalSecret Resource in the EKS cluster

The other API resource is ExternalSecret, which describes what data to be fetched, how frequently and how the data needs to be transformed and saved as a Kubernetes Secret. Repeat the steps mentioned in step#5 with below resources and commands.

externalSecret.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: external-secret-backend-credentials
namespace: eso
spec:
refreshInterval: 1m
secretStoreRef:
name: secretstore-backend-credentials
kind: SecretStore
target:
name: secret-backend-credentials
creationPolicy: Owner
data:
- secretKey: dbUsername
remoteRef:
key: dev/backend_credentials
property: backend.username
- secretKey: dbPassword
remoteRef:
key: dev/backend_credentials
property: backend.password
$ kubectl create -f externalSecret.yaml
externalsecret.external-secrets.io/external-secret-backend-credentials created
$
$ kubectl get es -n eso
NAME STORE REFRESH INTERVAL STATUS READY
external-secret-backend-credentials secretstore-backend-credentials 1m SecretSynced True

Note here that additionally there is Kubernetes Secret created with the name given in spec.targe.name of the externalSecret manifest file.

$ kubectl get secrets -n eso
NAME TYPE DATA AGE
default-token-nbqcs kubernetes.io/service-account-token 3 29m
eso-service-account-token-jvrjc kubernetes.io/service-account-token 3 19m
external-secrets-cert-controller-token-gzbv6 kubernetes.io/service-account-token 3 19m
external-secrets-operator-webhook Opaque 4 19m
external-secrets-webhook-token-thmqb kubernetes.io/service-account-token 3 19m
secret-backend-credentials Opaque 2 82s
sh.helm.release.v1.external-secrets-operator.v1 helm.sh/release.v1 1 19m

Step 7: Validate

Let’s retrieve the values of the secret by decoding the base64 values. Verify that the credentials are matching with what we have created in step#1.

$ kubectl get secret secret-backend-credentials -n eso -o yaml
apiVersion: v1
data:
dbPassword: QFdmdEVzdCZlc3VES1U=
dbUsername: ZGJhZG1pbg==
immutable: false
kind: Secret
metadata:
annotations:
reconcile.external-secrets.io/data-hash: 851f34a0f606bd77b41455945deb84ee
creationTimestamp: "2022-11-21T23:47:41Z"
name: secret-backend-credentials
namespace: eso
ownerReferences:
- apiVersion: external-secrets.io/v1beta1
blockOwnerDeletion: true
controller: true
kind: ExternalSecret
name: external-secret-backend-credentials
uid: a1da20c2-8161-4f64-9c8b-817a3afcb7a2
resourceVersion: "7024"
uid: 11c8f51a-0bf3-41aa-9e0c-8ef46bb53039
type: Opaque
$
$
$ echo "QFdmdEVzdCZlc3VES1U=" | base64 -d
@WftEst&esuDKU
$
$ echo "ZGJhZG1pbg==" | base64 -d
dbadmin

Step 8: Refresh the Secrets and Validate

In this step, lets modify the secret values and validate if they are automatically refreshed in EKS. I have changed the password to newvalue.

Retrieve the password value from the EKS cluster and verify it is matching.

$ kubectl get secret secret-backend-credentials -n eso -o yaml
apiVersion: v1
data:
dbPassword: bmV3dmFsdWU=
dbUsername: ZGJhZG1pbg==
immutable: false
kind: Secret
metadata:
annotations:
reconcile.external-secrets.io/data-hash: deedba3d134320963742df456552ade9
creationTimestamp: "2022-11-21T23:47:41Z"
name: secret-backend-credentials
namespace: eso
ownerReferences:
- apiVersion: external-secrets.io/v1beta1
blockOwnerDeletion: true
controller: true
kind: ExternalSecret
name: external-secret-backend-credentials
uid: a1da20c2-8161-4f64-9c8b-817a3afcb7a2
resourceVersion: "8239"
uid: 11c8f51a-0bf3-41aa-9e0c-8ef46bb53039
type: Opaque
$
$ echo "bmV3dmFsdWU=" | base64 -d
newvalue

Step 9: Clean up

Delete the Kubernetes resources first and following that delete the IAM Role and IAM OIDC provider .

$ kubectl delete -f secretStore.yaml
secretstore.external-secrets.io "secretstore-backend-credentials" deleted

yevke@DESKTOP-MDGA8T8 MINGW64 ~/story1
$ kubectl delete -f externalSecret.yaml
externalsecret.external-secrets.io "external-secret-backend-credentials" deleted

$ helm delete external-secrets-operator -n eso
release "external-secrets-operator" uninstalled

$ kubectl get all -n eso
No resources found in eso namespace.

$ kubectl delete ns eso
namespace "eso" deleted

--

--

DevOps | Certified Kubernetes Admin | AWS | Terraform

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store