My deployment to Azure using Terraform and Flux

To deploy my needed infrastructure and applications to Azure, I use a combination of Terraform and Flux, all running from GitHub Actions workflows. This is only a high level overview and some details have been excluded or skimmed over for brevity.

Terraform

For me, one of the biggest limitations of Terraform is how bad it is at DRY (i.e. Don’t Repeat Yourself). I wanted to maintain a pure Terraform solution whilst trying to minimise the repetition but also allow the easy spinning up of new clusters as needed. I also knew I needed a “global” environment as well as various deployment environments but, for now, development and production will suffice.

Folder Structure

The top level of my Terraform definition is one of two folders; scope-global and scope-environment. These folders serve as the entry point for either a global environment (of which there is only one normally of course) or deployment environment respectively.

There are three additional top level folders; collections, environments and modules.

Modules contains definitions for components such as a key vault or AKS cluster along with supporting elements e.g. additional node pools or bootstrapping a cluster with Flux.

The environments folder contains tfvars files for each environment (e.g. development, production and global) along with a common base.tfvars which is applied to all environments first, before environment specific variants.

The final folder, collections, contains two folders – environment and global. The latter is only one layer deep and defines any global resources such as a global DNS zone and global key vault.

The environment folder is a series of nested folders which defines the components that will be deployed at each of these levels. The levels are environment, region, farm and cluster. This allows, for example, an environment level key vault be defined or a farm level Azure App Config.

Variables

To aid with data passing between layers, a few variables have been set up which get passed through the layers and can have data added as they pass down the chain. These variables are defaults, created_infrastructure and credentials. As an example, when an environment level key vault is created, it’s ID and name are added to created_infrastructure so that access can be set up for the AKS cluster using a managed identity.

Flux

Bootstrapping

When a cluster is provisioned, it is automatically given a cluster folder in the Flux repo and has a cluster_variables config map created to store values that either Flux or Helm charts being applied by Flux may need including things like the region or global key vault’s name.

Folder Structure

The top level folder is the clusters folder and it contains a _template folder along with a folder per environment. The individual environment folders then contain a folder for the region, the farm, and finally the cluster (e.g. development/uksouth/f01/c01). The c01 folder is based on the _template folder.

The remaining folders are applications, clients, infrastructure, redirects and services with each of these being referenced from the c01 folder.

The infrastructure folder contains setup manifests and Helm releases for things like Istio, External DNS or External Secrets Operator.

The redirects folder is split up by environment and defines any redirects which should be added for that environment. These are managed using a redirects Helm chart and a series of values passed to its HelmRelease object.

The services folder allows you to define services which should be deployed at the various levels of global, environmental, regional, farm or cluster level. There is a definitions folder containing the base definition for each service.

The applications folder defines applications which should be deployed to specific clusters and, as with services, there is a definitions folder which contains the default configuration. These are generally non-targeted applications such as a commercial landing page.

The final folder is clients which contains a definition for any client applications. It’s quite likely this folder may only contain a single definition if only a single SaaS application (note this is single application, not single microservice) exists. There are then the usual nested environment, region, farm and cluster folders with each cluster defining the clients that are deployed to that specific instance.