Anthos Config Management and Config Connector

In the past several posts we looked at Config Connector for Kubernetes. With Config Connector you can create GCP resources, like Spanner or PubSub, using declarative K8s model. In this post we will show how Config Connector works together with Anthos Config Management (ACM).

Why ACM?

In all the examples before, we actuated GCP resources by simply running kubectl apply on a file or directory, containing declarative configs. This, by itself, however, does not provide full configuration-as-code workflow. First of all, you need to code review the configurations before they are applied. This is to make sure, that the same code, that was reviewed, was eventually submitted. Secondly, you want to apply same or similar configuration on many clusters reliably, without having to run the same script over and over. Finally, to prevent “shadow ops”, i.e. unapproved changes on your clusters.

In this post we will extend the configuration we built earlier, WordPress powered by GCP MySQL database with Workload Identity for secure keyless IAM. We will demonstrate how we can use Anthos Config Management and Config Connector to automatically sync and apply this configuration from Git repo to your K8s clusters and GCP projects.

To start, fork this repo: gcp-kcc-samples. Then, clone your fork:

git clone https://github.com/[my-git-user-name]/gcp-kcc-samples.git repo-name/

GCP project and cluster

Let’s provision a project that will be hosting our main Kubernetes cluster:

export PROJECT_ID=[PROJECT_ID]
export BILLING_ACCOUNT=[BILLING_ACCOUNT]
export SA_EMAIL="cnrm-system@${PROJECT_ID}.iam.gserviceaccount.com"

gcloud projects create ${PROJECT_ID} --name=${PROJECT_ID} 
gcloud alpha billing projects link ${PROJECT_ID} --billing-account $BILLING_ACCOUNT
gcloud config set project $PROJECT_ID

# enable container API
gcloud services enable container.googleapis.com --project ${PROJECT_ID}

# enable SQL admin API
gcloud services enable sqladmin.googleapis.com --project ${PROJECT_ID}

# create service account that will be used for Config Connector
gcloud iam service-accounts create cnrm-system --project ${PROJECT_ID}
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member "serviceAccount:${SA_EMAIL}" --role roles/owner
gcloud iam service-accounts keys create --iam-account "${SA_EMAIL}" ./key.json

Let’s now create our Kubernetes cluster.

export CLUSTER_ID=[CLUSTER_NAME]
export ZONE=[ZONE]
gcloud beta container clusters create ${CLUSTER_ID} --identity-namespace=${PROJECT_ID}.svc.id.goog --zone $ZONE
gcloud container clusters get-credentials $CLUSTER_ID --zone=$ZONE

Before we proceed with ACM configuration, let’s create two additional GCP projects: they will be holding GCP resources for DEV and PROD environments. Run the following configuration, after replacing [PROJECT_ID] and [BILLING_ACCOUNT] with your custom values:

    export PROJECT_ID=[PROJECT_ID]
    export SA_EMAIL="cnrm-system@${PROJECT_ID}.iam.gserviceaccount.com"
    export BILLING_ACCOUNT=[BILLING_ACCOUNT]
    
    gcloud projects create ${PROJECT_ID}-dev --name=${PROJECT_ID}-dev
    gcloud alpha billing projects link ${PROJECT_ID}-dev --billing-account $BILLING_ACCOUNT
    gcloud projects add-iam-policy-binding ${PROJECT_ID}-dev --member "serviceAccount:${SA_EMAIL}" --role roles/owner
    gcloud services enable sqladmin.googleapis.com --project ${PROJECT_ID}-dev

    gcloud projects create ${PROJECT_ID}-prod --name=${PROJECT_ID}-prod
    gcloud alpha billing projects link ${PROJECT_ID}-prod --billing-account $BILLING_ACCOUNT
    gcloud projects add-iam-policy-binding ${PROJECT_ID}-prod --member "serviceAccount:${SA_EMAIL}" --role roles/owner
    gcloud services enable sqladmin.googleapis.com --project ${PROJECT_ID}-prod

As you see, this step currently requires using gcloud command. Config Connector will soon have a capability to create GCP projects. At this point, this step, including project creation and IAM will then be replaced with declarative configuration directly through Anthos Config Management.

We are now ready to initialize our infrastructure. This diagram shows what we are about to build:

As you can see, we are going two environments running WordPress application powered by GCP MySQL database. The configuration between dev and prod environments will vary partially, for example, the size of DB, SQL credentials are examples of different configurations. When initialized, wp-dev and wp-prod will become Kubernetes namespaces. On GCP side, they will map to two different projects, that we created earlier.

Anthos Config Management

All the subsequent steps will be done assuming you are working in the local clone of your repo in src/wp-acm/ directory. Let’s install ACM operator. Here I’m explicitly referencing version 1.1.0.

# download
gsutil cp gs://config-management-release/released/1.1.0/config-management-operator.yaml config-management-operator.yaml
# apply CRD
kubectl apply -f config-management-operator.yaml

Before we proceed to the next step, if you haven’t already, make sure that you check in all the changes in your local repo fork.

At this point, we are ready to initialize Anthos Config Management and start sync process. Let’s modify and apply the following configuration with kubectl apply. Note that configConnector and policyController settings are enabled.

apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
spec:
  clusterName: cluster-1
  git:
    syncRepo: https://github.com/[your-git-user-name]/gcp-kcc-samples.git
    syncBranch: master
    secretType: none
    policyDir: "src/wp-acm/acm-root"
  configConnector:
    enabled: true
  policyController:
    enabled: true

Please see ACM documentation for explanations of all fields.

Wait some time for resources to sync up. Once cnrm-system namespace is created, propagate permissions from Config Connector service account to cnrm-system namespace, where we have Config Connector pod.

kubectl create secret generic gcp-key --from-file ./key.json --namespace cnrm-system

If you run kubectl get namespaces you will see new namespaces created and after a while if you run kubectl describe iamserviceaccount --all-namespaces, you will see new SQL service accounts.

What We Built

Once this is applied, Anthos Config Management Syncer should start creating objects. Let us review in more detail all the configs and what they represent:

• acm-root
  • cluster: folder containing the configs that applies for the whole cluster
    • ns-must-have-cost-center.yaml
  • namespaces
    • online
      • wp: abstract namespace, contains common objects across all namespaces
        • wp-dev
          • namespace.yaml
          • <<more resource files, specific to wp-dev>>
        • wp-prod
          • namespace.yaml
          • <<more resource files, specific to wp-prod>>
        • <<common resources, same in wp-prod and wp-dev>>
  • system
    • repo.yaml: mandatory file container repo config

See ACM documentation for more details on the repro structure and what each folder represents.

Once everything is synced, this is what you should have running in each of the two namespaces.

To verify the WordPress sites that were created, you can obtain the external site IP address by running kubectl get svc --namespace=wp-dev (or wp-prod). Let us review all the key components of our configuration.

GCP Resources Created using Config Connector

In each of the two projects, we created:

• SQL Instance (yaml), SQL User (yaml) and SQL Database (yaml)
• service account (yaml)
• IAM policy to give the service account cloudsql/client permission
• IAM policy to give our K8s account workloadIdentityUser permission on the service account (yaml)

This post gives more details on each of these resources and using Config Connector configuration.

Secure IAM with Workload Identity

Workload identity enables keyless and secure IAM by propagating permissions from your Google service account to your K8s service account. This requires:

• Google service account (yaml)
• K8s service account (yaml – note the annotation)
• Workload Identity Policy (yaml – note, that the member is K8s service account)

See this post for more details on declarative Workload Identity provisioning.

Constraints with Gatekeeper

Gatekeeper constraint that ensures that each namespace has required labels – this is ns-must-have-cost-center.yaml under cluster folder.

Importantly, you can apply Gatekeeper constraints on GCP objects created with Config Connector. More example on this in this post.

This is it! If you need to make any changes to your configuration, apply your changes with checking in the config to Git.