Accessing GCP Secret Manager from GKE Cluster using Murmur

Made using Canva

Secret manager implementation is one of the industry's best practices that currently are ubiquitous standards. The awareness of secret manager implementation can be seen from a survey that AKeyLess did that showed some interesting highlights:

Quoted from AKeyLess article

As an overview, most security experts place the secrets manager implementation on the top priority list since it gains significant impact in terms of risk mitigation.

Suppose you want more information about the definition of Secret Manager and its standard, it can be easily surfed from the internet since many articles can explain it more comprehensively.

I will focus on talking about one of the Secret Manager’s products offered by GCP since it’s easy to try and implement for organizations/companies that use GCP as their Cloud Foundation.

There are other alternatives to Secret Manager products like Vault by Hashicorp, Azure Key Vault, AWS Secrets Manager, and many more so, you can use it based on your conditions/use cases/team size/other considerations.

You can consider GCP Secret Manager for several considerations:

1. Secret Manager platform that has capabilities to log all of the changes and audit for user activities related to secrets key
2. Secret Manager platform that is ready to use and natively integrated into GCP
3. Secret Manager platform that can expose secret value end-to-end without exposing the value to humans (unless from the Secret Manager platform itself)

For implementing GCP Secret Manager, a common way to consume the secret is by using Client Libraries (integrate GCP Secret Manager to code itself), and setting up Application Default Credentials so our applications or services can access the secret values.

Going more specific for applications that run on top of GKE, the approach for integrating GKE to this Secret Manager might be the same, or you can do some experimentation for new integration made by GCP (when this article was created, the Secret Manager add-on was currently on Preview phase).

The secret Manager add-on is currently in the Preview phase

But by default, If you’re deploying your application/service on top of GKE, we can use workload identity to access the GCP Secret Manager.

It can be more challenging if you don’t want or aren’t able to modify the source code. For example, You’re deploying 3rd third-party service inside your cluster and aren’t able to modify the code as part of a business agreement. But don’t worry, we can still utilize the GCP Secret Manager. Let me introduce your Murmur!

Say hello to Murmur

Murmur has developed using a plug-and-play paradigm to make the application able to get the secret value from the Secret Manager Provider without modifying our application code. We can get secret value from GCP Secret Manager by including murmur Binary with our application. Let’s use a simple diagram to understand what Murmur does:

an overview of murmur workflow. source: self-drawn

The workflow starts when we run murmur to run our application as a subprocess using the following command:

./murmur run -- [command to run our apps]

At first, murmur will start checking all of the ENV values. if ENV contains the pattern to get value from the secret manager, it will start the auth process to the GCP Secret manager using the Secret Manager library.

Here’s one example of getting a secret from the GCP Secret manager:

gcpsm:my-project-name/my_secret_sauce

this pattern will ask murmur to get my_secret_sauce from project my-project-name, by default it will fetch the latest version. You can specify which secret version you want to fetch as well:

gcpsm:my-project-name/my_secret_sauce#2

Murmur will use a service account attached to the instance (Instance Attached Service Account) or deployment (Workload Identity). If the service account has permission to access the secret, replace the ENV value with the fetched result.

The replacement process only happens at the subprocess level, so if someone can check the ENV value inside the instance or container, they can’t see the fetch results.

Integrating Murmur with Kubernetes Deployment

There are two approaches to integrating our application with Murmur:

1. Add murmur during the Dockerbuild process
2. Copy murmur binary to Kubernetes Pod using initContainer

Before we dive into both approaches, we need to make sure we already have the following resources:

1. Create a GCP Service Account for Accessing the GCP Secret Manager and have a workload identity user role
2. Create a Kubernetes Service Account that bonded with the GCP Service Account

for example, you have a GCP SA called secret-accessor:

gcloud iam service-accounts create secret-accessor \
    --project=my-awesome-project

then let’s give it access to our secret sauce

gcloud secrets add-iam-policy-binding YOUR_SECRET_SAUCE \
    --member="serviceAccount:serviceAccount:secret-accessor@my-awesome-project.iam.gserviceaccount.com" \
    --role="roles/secretmanager.secretAccessor"

after that, add let’s allow GCP SA as workload identity user

gcloud iam service-accounts add-iam-policy-binding secret-accessor@my-awesome-project.iam.gserviceaccount.com \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:my-awesome-project.svc.id.goog[your-kube-namespace/kube-sa-name]"

After we finish the setup process for the Service Account, let’s go into both approaches:

The 1st approach can be done by adding the following line in your Dockerbuild file:

COPY --from=ghcr.io/busser/murmur:latest /murmur /bin/murmur

Then, change the point to start the application using a murmur command.

# from this:
ENTRYPOINT ["/bin/run-my-app"]

# to this:
ENTRYPOINT ["/bin/murmur", "run", "--", "/bin/run-my-app"]

The 2nd approach is done by utilizing the initContainer workflow, so we can copy the Murmur binary using a sharing folder. Let’s take an example:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  namespace: my-app 
spec:
  serviceAccountName: secret-accessor
  initContainers:
    - name: copy-murmur
      image: ghcr.io/busser/murmur:latest
      command: ["cp", "/murmur", "/shared/murmur"]
      volumeMounts:
        - name: shared
          mountPath: /shared
  containers:
    - name: my-app
      image: my-app:latest
      command: ["/shared/murmur", "run", "--", "/bin/run-my-app"]
      volumeMounts:
        - name: shared
          mountPath: /shared
  volumes:
    - name: shared
      emptyDir: {}

Let’s use the 2nd approach which has more deployment flexibility. Here’s a visualization about the murmur deployment strategy on Kubernetes:

Murmur workflow on Kubernetes Platform. source: self-drawn

When the pod is created using the deployment manifest or pod manifest, it will run the container with a shared volume called “shared”. if you’re looking at the init container command it will copy the binary of murmur inside the container to the shared volume.

command: ["cp", "/murmur", "/shared/murmur"]

Next, this shared volume will be attached to our application container, and then we will override the container init command with the murmur run command:

command: ["/shared/murmur", "run", "--", "/bin/run-my-app"]

Example of log output if murmur can discover the GCP Secret Manager value:

[murmur] overloading YOUR_SECRET_SAUCE

One of the common problems during this setup is you’re not setting up the service account’s permission correctly. You will see this kind of log in your application container:

* YOUR_SECRET_SAUCE: could not resolve reference: failed to access secret "YOUR_SECRET_SAUCE" version "latest": rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).
This error could be caused by a missing IAM policy binding on the target IAM service account.
For more information, refer to the Workload Identity documentation:
https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to

This can be because you do not attach the GCP SA access to the secret manager or the GCP SA can’t authenticate as a workload identity user. Another potential issue can be because you’re not set the GKE Cluster run using Workload Identity Federation for GKE

Conclusion

Murmur is helping to integrate our application with GCP Secret Manager without changing our code, you can consider this approach because Murmur offers more than 1 Secret provider compatibility:

Current murmur compatibility at version v6.0.1

So when it comes to multi-cloud deployment or cloud migration, the secret manager transition can be easier since the compatibility layer is handled by murmur. But, if you’re considering using GCP for the rest of your life, it seems like Murmur is too overkill for that.

The main reason isn’t just that, another reason it doesn’t expose the secret value inside the container if you’re trying to execute the murmur command there. So no one knows the secret value unless they have access to the GCP Secret Manager dashboard.

You can consider GCP Secret Manager since it has a monthly free tier for doing some trials/POC.

But again, use a secret manager provider based on the goal you want to achieve.

Choose Wisely 🤫