Hi all,
As someone who’s spent countless hours setting up and managing infrastructure, I know how tedious and error-prone it can be to create a kubernetes clusters on different environments or service providers. But what if I told you there’s a way to make this process a whole lot easier? Enter Terraform - a powerful tool for automating infrastructure deployment - which many of you readers may already be aware of. So let’s skip the broad introduction and get right into the details.

First the module is available on Github under the following domain: https://github.com/deB4SH/terraform-proxmox-cloud-init-kubernetes

What does this module do?

This Terraform module takes care of creating a Kubernetes cluster on proxmox using cloud-init and kubeadm.

With it, you’ll be able to:

• Automate the creation of a Kubernetes cluster with just a few variables
• Customize the scale of your cluster

Important Sidenote: currently the module is not able to scale up multiple control planes - I’m planning to implement it down the road.

How does everything work?

The module is pretty much ready to run on your infrastructure you simply need to configure some values.
Within the variables.tf you can easily review all available configuration: https://github.com/deB4SH/terraform-proxmox-cloud-init-kubernetes/blob/main/variables.tf
For example a value you need to overlay is the user_password and user_pub_key to automatically add your credentials to each vm.

A typical configuration based on this module may look like the following listing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

module "kubernetes" {
  providers = {
    proxmox = proxmox.YOUR_NODE_NAME
  }
  source = "github.com/deB4SH/terraform-proxmox-cloud-init-kubernetes?ref=0.1"

  user = var.user
  user_password = var.user_password
  user_pub_key = var.user_pub_key
  node_name = var.YOUR_NODE_NAME.node_name

  vm_dns = {
    domain = "."
    servers = ["10.10.10.2"]
  }
  
  vm_ip_config = {
    address = "10.10.10.10/24"
    gateway = "10.10.10.1"
  }

  vm_datastore_id = "local-lvm"

  workers = [
    {
      node = "YOUR_NODE_NAME"
      name = "worker01"
      vm_cpu_cores = 6
      vm_memory = 12288
      ip = "10.10.10.11/24"
      id_offset = 1
      image_type = "amd64"
    }
  ]

  vm_images = [ 
  {
    name = "amd64"
    filename = "kubernetes-debian-12-generic-amd64-20240507-1740.img"
    url = "https://cloud.debian.org/images/cloud/bookworm/20240507-1740/debian-12-generic-amd64-20240507-1740.qcow2"
    checksum = "f7ac3fb9d45cdee99b25ce41c3a0322c0555d4f82d967b57b3167fce878bde09590515052c5193a1c6d69978c9fe1683338b4d93e070b5b3d04e99be00018f25"
    checksum_algorithm = "sha512"
    datastore_id = "nas"
  },
  {
    name = "arm64"
    filename = "kubernetes-debian-12-generic-arm64-20240507-1740.img"
    url = "https://cloud.debian.org/images/cloud/bookworm/20240507-1740/debian-12-generic-arm64-20240507-1740.qcow2"
    checksum = "626a4793a747b334cf3bc1acc10a5b682ad5db41fabb491c9c7062001e5691c215b2696e02ba6dd7570652d99c71c16b5f13b694531fb1211101d64925a453b8"
    checksum_algorithm = "sha512"
    datastore_id = "nas"
  }
  ]
  
}

To break things down. The provider block configures the proxmox endpoint you using to do the api calls to. For a overview how to use this provider please review the official documentation here: https://registry.terraform.io/providers/bpg/proxmox/latest/docs
The source block configures the module to use. In this case we are referencing version 0.1 of the module. Don’t be confused by this version - it’s just a development number for now.

The following blocks are delegated blocks towards each vm and configure their respective parts. For example the dns block configures the available dns the vm should use.
vm_ip_config describes currently the ip address of the control plane. This will change in future when this module allows multiple control planes.

The list around the configuration value for workers describe the amount of vms you want to create as worker nodes for your cluster.

Last but not least: You are able to configure multiple vm images for amd64 and or arm. It would also be possible to inject different images for workers this way.

What are you getting after a successful deployment

Well a barebones kubernetes cluster. No networking. No fancy load balancing pods. Pretty much nothing. A blank slate to work on.

1
2
3
4

KUBECONFIG=config k get nodes  
NAME                      STATUS     ROLES           AGE     VERSION  
kubernetes-controlplane   NotReady   control-plane   5m21s   v1.30.2  
worker01                  NotReady   <none>          106s    v1.30.2

A good next step would be adding a network component to your cluster.
I like cilium and prepared an umbrella helm-chart for this setup in my helm-charts repository. Available here: https://github.com/deB4SH/helm-charts/tree/main/charts/cilium
After a successful apply your nodes should become ready for workloads.

What’s next for this module?

Good question! There are several things this module is currently missing out on.

First thing - it would be awesome if this module also allows you to create highly available kubernetes clusters with multiple control planes
It would also be awesome if there is some configuration available to automatically join a new cluster towards existing gitops solutions like argocd or flux to automatically apply services like cilium, external-dns, external-secrets or sops any many more.
Lastly: this module may be a bit cluttered configuration-wise and doesn’t follow any particular standard currently. Standardizing this would surely improve configurability and ease up the usage.

As always - I’m hoping this blog post helps others to get your toes into the big blue ocean of platform engineering on proxmox with kubeadm.

Sources

• Terraform Module Proxmox Kubernetes
• Helm Charts Repository
• Terraform Provider Proxmox
• Cilium

Hi all,
I wanted to move all my artifacts back into my homelab to be able to run it airgapped.
To reduce the overhead of running multiple services to achieve this goal I’ve comitted myself on running the sonatype nexus repository manager. Through their broad community support multiple package types are supported by one solution. To survive the administration configuration hell of the repository manager and store a configuration as code within my git, I opted towards writing multiple terraform modules to configure my nexus. The following blog post shall give you a broad overview how I approached the issue and explain how to use my modules for this specific usecase.

Let’s survive the configuration hell!
Nexus provides an integration api for scripted setups which is reachable under ${*YOUR_NEXUS_URL*}/service/rest.
A swagger documentation is available under ${*YOUR_NEXUS_URL*}/service/rest/swagger.json.

Happily enough amartin provides a nexus provider for configuring nexus via terraform including a good documentation how to work with this provider.

To store credentials generated within the module I’ll use the bitwarden provider to access my vault and store them there accordingly.
There is a good documentation and well written provider available under the following link

As first step lets configure all used providers in this blog post within the main.tf.
We are using the nexus provider, a random provider for password generation and as already mention within the note a bitwarden provider to store the generated passwords.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

terraform {
  required_providers {
    nexus = {
      source = "datadrivers/nexus"
      version = "2.1.0"
    }
    random = {
      source = "hashicorp/random"
      version = "3.6.0"
    }
    bitwarden = {
      source  = "maxlaverse/bitwarden"
      version = ">= 0.7.0"
    }
  }
}

Following the declaration of all required providers is the instanciation of these with the relevant variables to configure them accordingly.

This guide references the nexus directly within a local network. There is no trusted certificate available for nexus at this point. If you are planning to run this approach against a publically available nexus please configure the insecure flag within the nexus provider accordingly.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

provider "random" {
  
}

provider "nexus" {
  insecure = true
  username = var.nexus.username
  password = var.nexus.password
  url      = var.nexus.url
}

provider "bitwarden" {
  email           = var.bitwarden.email
  master_password = var.bitwarden.master_password
  client_id       = var.bitwarden.client_id
  client_secret   = var.bitwarden.client_secret
  server          = var.bitwarden.server
}

To externalize all sensetive credentials it is advised to create a specific tfvars for each environment. The general variables.tf which follows the shown example may look like the following block.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

variable "nexus" {
  type = object({
    username = string
    password = string
    url = string
  })
  sensitive = true
}

variable "bitwarden" {
  type = object({
    email = string
    master_password = string
    server = string
    client_id = string
    client_secret = string
  })
  sensitive = true
}

An according development.tfvars may look like.

1
2
3
4
5
6
7
8
9
10
11
12
13

nexus={
    username="local-admin"
    password="awesome#Super.Password!6576"
    url="https://nexus.local.lan"
}

bitwarden={
    email="svc.user.nexus@local.lan"
    master_password="my#Awesome.Master!Password"
    client_id="user.1233-123123-123123-123"
    client_secret="1K{....}}zB"
    server="https://keyvault.local.lan"
}

With these base steps done you are now good to go for the implementation of your configuration.

So lets start with implementing a hosted docker repository, shall we?

Create a new directory called modules in the root of your project and create a new file called providers.tf inside it.
It would also be possible to reuse the provider from your base terraform code but if you want to externalize the module it may be usefule to also externalize the providers.

Within the providers.tf file add the following content:

1
2
3
4
5
6
7
8
9
10
11
12

terraform {
  required_providers {
    nexus = {
      source = "datadrivers/nexus"
      version = "2.1.0"
    }
    random = {
      source = "hashicorp/random"
      version = "3.6.0"
    }
  }
}

As next step we need to create a variables.tf to configure our required variables for this setup.
Each registry requires a name, a port and an isOnline flag.
A blobStoreName is required to configure final storage environment that is used on your host.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

variable "name" {
    type = string
    description = "Name of the docker registry"

}

variable "isOnline" {
    type = bool
    default = true
    description = "Toggle switch to enable or disable online usage of this repository"

}

variable "port" {
    type = string
    description = "Port to announce service on"

}

variable "blobStoreName" {
    type = string
    default = "default"
    description = "Blob Storage wihin nexus to use"

}

After a successful deployment we want to extract some configured values like the username of the read user and required password.
For this please add and configure the outputs.tf file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

output "pull-user" {
    value = nexus_security_user.pull-user.userid
}

output "pull-user-pw" {
    value = random_password.pull-user-password.result
}

output "push-user" {
    value = nexus_security_user.push-user.userid
}

output "push-user-pw" {
    value = random_password.push-user-password.result
}

With everything done configuration-wise it is now required to configure the actual repository that hosts the files.
The following listing creates a hosted docker repository in your nexus environment with the configuration you’ve set in your variables.
If you like you could easily extend the configuration with the currently pre-defined values in this registry.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

resource "nexus_repository_docker_hosted" "registry" {
  name   = "${var.name}"
  online = var.isOnline

  docker {
    force_basic_auth = false
    v1_enabled       = false
    http_port        = "${var.port}"
  }

  storage {
    blob_store_name                = "${var.blobStoreName}"
    strict_content_type_validation = true
    write_policy                   = "ALLOW"
  }
}

To access this newly created registry we need to create as last step new accounts. This can also be done via terraform.
The following code-blocks create random passwords for a user designated to access the registry via read only rules and one password for a user with write permission.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

resource "random_password" "pull-user-password" {
  length           = 16
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

resource "random_password" "push-user-password" {
  length           = 16
  special          = true
  override_special = "!#$%&*()-_=+[]{}<>:?"
}

resource "nexus_security_user" "pull-user" {
  userid     = "docker-${var.name}-pull"
  firstname  = "Docker Pull"
  lastname   = "${var.name}"
  email      = "svc.docker.${var.name}-pull@local.lan"
  password   = random_password.pull-user-password.result
  roles      = ["docker-${var.name}-pull-role"]
  status     = "active"
  depends_on = [nexus_repository_docker_hosted.registry, nexus_security_role.security-role-pull]
}

resource "nexus_security_user" "push-user" {
  userid     = "docker-${var.name}-push"
  firstname  = "Docker Push"
  lastname   = "${var.name}"
  email      = "svc.docker.${var.name}-push@local.lan"
  password   = random_password.push-user-password.result
  roles      = ["docker-${var.name}-push-role"]
  status     = "active"
  depends_on = [nexus_repository_docker_hosted.registry, nexus_security_role.security-role-push]
}

As you may have seen these users reference their specific security roles that we are currently not providing.
As last step we need to set those up.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

resource "nexus_security_role" "security-role-pull" {
  description = "Docker Pull Role for ${var.name}"
  name        = "docker-${var.name}-pull-role"
  privileges = [
    "nx-repository-view-docker-${var.name}-read",
    "nx-repository-view-docker-${var.name}-browse",
  ]
  roleid = "docker-${var.name}-pull-role"
  depends_on = [nexus_repository_docker_hosted.registry]
}

resource "nexus_security_role" "security-role-push" {
  description = "Docker Pull Role for ${var.name}"
  name        = "docker-${var.name}-push-role"
  privileges = [
    "nx-repository-view-docker-${var.name}-read",
    "nx-repository-view-docker-${var.name}-browse",
    "nx-repository-view-docker-${var.name}-add",
  ]
  roleid = "docker-${var.name}-push-role"
  depends_on = [nexus_repository_docker_hosted.registry]
}

When everything works together you should be able to create repositories easily with close to zero configuration overhead due to the flexibility of terraform.
This setup allows you to create multiple repositories at once.
For example if you are using the newly created module in your main terraform structure you could easily wrap it with a for_each call.

1
2
3
4
5
6
7
8
9
10

module "docker-registry" {
  source = "github.com/deB4SH/terraform-nexus-docker-module?ref=1.0.0"

  for_each = { for dr in var.docker_repository : dr.name => dr}

  name = each.key
  isOnline = each.value.isOnline
  port = each.value.port
  blobStoreName = each.value.blobStoreName
}

Based on the given information in the following block this will create two repositories with dedicated read and write users with close nearly no configuration from your end.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

docker_repository=[
    {
        name="test1"
        isOnline=true
        port="61000"
        blobStoreName="default"
    },
    {
        name="test2"
        isOnline=true
        port="61001"
        blobStoreName="default"
    }
]

I hope this guide will help you to get an introduction towards managing your infrastructure with terraform.

Sources

• Terraform Provider by amartingarcia
• Terraform Nexus Hosted Docker Module
• Terraform Nexus Hosted APT Module

Hi all,
in my current role as platform engineer for the Hamburg Port Authority we need to keep the velocity high to provide a good service for our internal clients. As result most of our infrastructure is created through templates in any way.

To get things up and running we developed a simple templater written in python with all required bindings towards jinja and kept expanding it. Naturally we expanded it without a direct scope and focus on reusability. Things needed to be done and it should work.
With growing struggle the need of a new templater arose. The idea of a new clean approach came up.

Let me present you: YART - yet another random templater

A simple but extensive templating tool for multiple needs.
YART provides you a useful schema validation for your input data to keep it free of issues and also an option to create a dynamic template structure based on your input.

Most specific for this tool may be the dynamic generation of folders based on your template.
It allows you to duplicate a specific structure multiple times.
Lets assume the following usecase.
You need to configure multiple clusters at once within an repository.
Each cluster contains multiple services which require a value configuration.
To ease up the installation you split all dedicated values within a folder named after the service.
The resulting folder structure may look like the following listing.

1
2
3
4

clusters
└── $name
    ├── externaldns
    └── mailhog

As configuration we are going to assume you want to template the following values onto your manifests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

clusters:
  - name: happy-path-cluster
    defaults:
      storageClass: "storageclass-happy-path-cluster"
      serviceUri: "happy-path-cluster.corp.local"
    serviceCatalog:
      externalDns:
        powerdnsApiKey: "myAwesomeKey"
        domainFilter:
          - uri: "*.subdomain.corp.local"
          - uri: "*.sub.corp.local"
      mailhog:
        uri: "mailhog"
  - name: happier-path-cluster
    defaults:
      storageClass: "storageclass-cluster"
      serviceUri: "happier-path-cluster.corp.local"
    serviceCatalog:
      externalDns:
        powerdnsApiKey: "myAwesomeKey"
        domainFilter:
          - uri: "*.subdomain.corp.local"
          - uri: "*.sub.corp.local"
      mailhog:
        uri: "mailhog-awesomesauce"

Based on this configuration the result looks like the following listing

1
2
3
4
5
6
7

clusters
├── happier-path-cluster
│   ├── externaldns
│   └── mailhog
└── happy-path-cluster
    ├── externaldns
    └── mailhog

The templated files could now easily picked up by your gitops tools and applied to the individual clusters.
This example is also available as code block wihtin the YART Repository.

A wider exaplaination of all features is also available within the README.MD within the repository.

Through the implementation of YART we were able to solve multiple issues with our current template approach and installed a flexable and dynamic way to create our clusters and deployment configuration with ease.

Hi all,
some projects and clusters may enter a maintenance mode in their lifetime and dont receive any updates, changes or even patches for a long time. If something like this happens it may be neccessary to rotate the certificates used by control planes. The control planes of vmware tanzu provide this functionality via kubeadm.

The following script automates this rotation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#!/bin/bash
# CONFIGURATION AREA
# PLEASE CONFIGURE THE SUPERVISOR NAMESPACE
export SUPERVISOR="tanzu-supervisor-one"

# PLEASE NO CHANGES BEYOND THIS POINT
kubectl config use-context $SUPERVISOR
# get available tkcs
tkcs=$(kubectl get tkc --no-headers -o custom-columns=":metadata.name") 
# iterate over each tkc and rotate certs on control planes
while IFS= read -r tkc; do 
echo "next tkc: $tkc"
SSHPASS=$(kubectl get secret $tkc-ssh-password -o jsonpath='{.data.ssh-passwordkey}' | base64 -d)
echo "aquired sshpass - getting control-plane ips now"
IPS=$(kubectl get vm -owide | grep ^$tkc-control-plane | awk '{print $5}')
	echo "aquired ips - running commands now"
	while IFS= read -r CPIP; do
		echo "rotate certs on node with ip: $CPIP"
		sshpass -p $SSHPASS ssh -o "StrictHostKeyChecking=no" -q vmware-system-user@$CPIP sudo kubeadm certs check-expiration < /dev/null
		sshpass -p $SSHPASS ssh -o "StrictHostKeyChecking=no" -q vmware-system-user@$CPIP sudo kubeadm certs renew all < /dev/null
		sshpass -p $SSHPASS ssh -o "StrictHostKeyChecking=no" -q vmware-system-user@$CPIP sudo reboot now < /dev/null
		echo "done with node ip: $CPIP"
	done <<< "$IPS"
done <<< "$tkcs"

This script helped us to rotate multiple clusters at once. May it help you too.

NOTE: This writeup is a knowledge dump of me to write down learnings from getting into astrophotography. Read everything with a grain of salt and please consider doing own research in this topic if my calculations may seem to be wrong. Thanks!

Hi there!
Since some of you may already know, I bought an entry-level telescope to take a dive into astrophotography.
As my first telescope and tripod I went with an EQ3-2 Sky-Watcher and a 750/150 Omegon telescope.
The first steps to get the feet wet are always to simply take a look through an ocular and watch the stars.
Most bundles come with a set of small oculars, which should serve fine for the first months of observation.

The next step may be to buy bigger and better oculars for a more detailed observation.
Before buying new expensive equipment it may be useful to calculate the range of possible magnification for your telescope to not waste money.

To start with the minimal possible magnification you could apply the following formular.

Based on my telescope I need to take the 150mm of the primary mirror and divide it with the 7mm of a human pupil which results in a of 21.42. Which means the minimal senseful magnification is 21 for my 750/150 Omegon.

Based on this calculation, it is now possible to calculate the focal length of an ocular with this equation.

For my specific usecase this results into the following equation.

This means a 35mm ocular is biggest senseful focal length that can be applied to my telescope.

Next step is to calculate the optimal focal length for an ocular.

Replacing the variables within the equation for the focal length of an ocular results in 3.5mm focal for an optimal magnification.

At last it may be useful to calculate the maximal magnification with the following equation.

The maximum resulting magnification for my telescope is around 300 which would result in a focal length of 2.5mm

Conclusion

With these values in mind, it is easier to search for good oculars and don’t waste money on higher focal lengths that may not help get a clear and sharp image of the observed sky section.

Hi all,
with the version 0.13.2 of metallb comes a change in regard to layer2 ip announcements.
Therefor it is now required to switch from the old configmap setup to a custom resource setup.
To document my upgrade steps - here is a small write-up of things required to get metallb running again.

Upgrade Guide

This guide assumes that you had been running an older version of metallb with a configmap for address pool configuration similar to the following listing:

1
2
3
4
5
6
7
8
9
10
11
12
13

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.1.100-192.168.1.195
      avoid-buggy-ips: true

This configuration can be removed from the deployment and will be replaced with the following two custom resource manifests.

At first, it is important to create an IPAddressPool manifest which provides the address range accordingly to the previous configmap configuration.

1
2
3
4
5
6
7
8

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name:  default-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.100-192.168.1.240

NOTE: Keep in mind that you need top change the addresses range accordingly to your setup.

Next, it is required to set up an L2Advertisement to announce all used IP to your local network.
This advertisement references the ipAddressPools directly within it specification and can be named something different.
To keep things in line, I named it accordingly to the IPAddressPool.

1
2
3
4
5
6
7
8

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  ipAddressPools:
  - default-pool

After checking in these manifests, metallb should pick them up and announce all used IP accordingly.

Hi all,
as some of you may know, I’m interested in homelabbing and are hosting my own kubernetes cluster at home. As part of a good homelab it is essential to keep track of logs and metrics.
The number one goto application for this usecase is often the kube Prometheus Stack, which is in my humble opinion a bit to big for my homelab in regard to memory, compute and storage footprint.
While looking for alternatives I stumbled upon Victoria Metrics which seems to be a perfect fit for my usecase.

It is build in a distributed fashion with a time-series storage ind mind. From my first view at the architectural view it looks quite fitting for my usecase and could be a nice general drop-in replacement for the Prometheus stack.

Source: Docs Victoria Metrics

So let’s get started in jump into the deep water and build our deployments.

A prepared example deployment is available here

Victoria Metrics Cluster

The cluster deployment is composed of the official helm-chart made available and contains the three root components vmselect, vminsert and vmstorage.

VMStorage is the data backend for all stored metrics and is the single golden trough for your queryable data in a time range. Due to the fact that the vmstorage component manages raw data it becomes a stateful part of your cluster, which is requiring some sort of special care.
VMInset and VMSelect are both stateless components in this stack and provide your third party applications access towards the raw data you are collecting in your cluster.

Installing the metrics cluster is rather easy due to the provided helm chart, which is easiest to view via ArtifactHub.
At the time of writing this blog post version 0.9.60 is the newest and everything is based on this.

To reduce the tool dependency, I’m going to use the HelmChartInflationGenerator for kustomize to keep everything in one universe.

First, we need to set up the inflation generator for this specific helm chart.

./base/victoria-metrics-cluster/helmrelease.yaml

1
2
3
4
5
6
7
8
9
10
11

apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: victoria-metrics-cluster
releaseName: victoria-metrics-cluster
name: victoria-metrics-cluster
version: 0.9.60
repo: https://victoriametrics.github.io/helm-charts/
valuesInline: {}
IncludeCRDs: true
namespace: victoria-metrics

./base/victoria-metrics-cluster/kustomization.yaml

1
2
3
4
5

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - helmrelease.yaml

The general information is pretty straight forward if you are already familiar with the helm-way to install prepared packages.
You may notice that the valuesInline are empty.
Due to the fact that I wanted to set up this deployment in a patch-able manor, the value overwrites are added with the next step.

./env/homelab/patches/patch-victoria-metrics-cluster.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: victoria-metrics-cluster
valuesInline: 
  rbac:
    create: true
    pspEnabled: false
  vmselect:
    replicaCount: 1
    podAnnotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8481"

  vminsert:
    replicaCount: 1
    podAnnotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8480"
    extraArgs:
      envflag.enable: "true"
      envflag.prefix: VM_
      loggerFormat: json

  vmstorage:
    replicaCount: 1
    persistentVolume:
      storageClass: nfs-client
    podAnnotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8482"

This patch applies modifications towards the helm chart which are generally available via the preconfigured values within it.
To elaborate my specific patch.
I wanted to create specific RBAC rules for my environment but was required to disable the pod security policies, due to the fact that these were removed in Kubernetes 1.25.
Besides that, I have set for each component a replication count of one to reduce the load on my environment and configured the pod annotations so that metrics are collected afterward.

NOTE: If you are going to use my example configuration. Please consider changing the storageClass, which may or may not be available in your infrastructure.

To collect both manifests, it is required to add an another kustomization with the following content.

./env/homelab/kustomization.yaml

1
2
3
4
5
6
7
8
9
10

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: victoria-metrics

resources:
  - ../../base/victoria-metrics-cluster

patchesStrategicMerge:
  - patches/patch-victoria-metrics-cluster.yaml

Using the HelmChartInflationGenerator within kustomize is currently a bit tricky and requires a special third kustomization which loads the second kustomization as generator module.

./generators/homelab/kustomization.yaml

1
2
3
4
5
6
7

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: victoria-metrics

generators:
  - ../../env/homelab/

With this setup, you are able to deploy the cluster deployment with any cicd approach or even a gitops approach.

If you are working with argocd to deploy this kustomization you need to add a plugin within your argocd-cm configmap and reference it within the plugin block in your application.

1
2
3
4
5
6
7
8
9
10

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  configManagementPlugins: |
    - name: kustomize-build-with-helm
      generate:
        command: [ "sh", "-c" ]
        args: [ "kustomize build --enable-helm" ]

Victoria Metrics Agent

Now with the cluster running, it is time to collect the first metrics from within the Kubernetes cluster. For this, it is possible to install the victoria metrics agent, which is also provided by a helm chart.

The agent is a tiny software that collects metrics from various sources and writes them towards the configure remote address.

Source: Victoria Metrics Documentation - VMagent

As first step, it is required to configure the helm inflation generator again.

./base/victoria-metrics-agent/helmrelease.yaml

1
2
3
4
5
6
7
8
9
10
11

apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: victoria-metrics-agent
releaseName: victoria-metrics-agent 
name: victoria-metrics-agent 
version: 0.8.29
repo: https://victoriametrics.github.io/helm-charts/
valuesInline: {}
IncludeCRDs: true
namespace: victoria-metrics

Equally, to the cluster deployment, a initial kustomization is required to collect all manifest together and prepare them for patches.

As next step, the patch configuration is required to configure the agent with this deployment.

NOTE: This is a rather big patch and will be partly explained afterward

./env/homelab/patches/patch-victoria-metrics-agent.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236

apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: victoria-metrics-agent
valuesInline: 
  rbac:
    pspEnabled: false
  
  deployment:
    enabled: false

  statefulset:
    enabled: true

  remoteWriteUrls:
   - http://victoria-metrics-cluster-vminsert.victoria-metrics:8480/insert/0/prometheus/
  
  config:
    global:
      scrape_interval: 10s

    scrape_configs:
      - job_name: vmagent
        static_configs:
          - targets: ["localhost:8429"]
      - job_name: "kubernetes-apiservers"
        kubernetes_sd_configs:
          - role: endpoints
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
          - source_labels:
              [
                __meta_kubernetes_namespace,
                __meta_kubernetes_service_name,
                __meta_kubernetes_endpoint_port_name,
              ]
            action: keep
            regex: default;kubernetes;https
      - job_name: "kubernetes-nodes"
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/$1/proxy/metrics
      - job_name: "kubernetes-nodes-cadvisor"
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
        metric_relabel_configs:
          - action: replace
            source_labels: [pod]
            regex: '(.+)'
            target_label: pod_name
            replacement: '${1}'
          - action: replace
            source_labels: [container]
            regex: '(.+)'
            target_label: container_name
            replacement: '${1}'
          - action: replace
            target_label: name
            replacement: k8s_stub
          - action: replace
            source_labels: [id]
            regex: '^/system\.slice/(.+)\.service$'
            target_label: systemd_service_name
            replacement: '${1}'
      - job_name: "kubernetes-service-endpoints"
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
          - action: drop
            source_labels: [__meta_kubernetes_pod_container_init]
            regex: true
          - action: keep_if_equal
            source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels:
              [
                __address__,
                __meta_kubernetes_service_annotation_prometheus_io_port,
              ]
            action: replace
            target_label: __address__
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: kubernetes_name
          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: kubernetes_node
      - job_name: "kubernetes-service-endpoints-slow"
        scrape_interval: 5m
        scrape_timeout: 30s
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
          - action: drop
            source_labels: [__meta_kubernetes_pod_container_init]
            regex: true
          - action: keep_if_equal
            source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_scrape_slow]
            action: keep
            regex: true
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels:
              [
                __address__,
                __meta_kubernetes_service_annotation_prometheus_io_port,
              ]
            action: replace
            target_label: __address__
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: kubernetes_name
          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: kubernetes_node
      - job_name: "kubernetes-services"
        metrics_path: /probe
        params:
          module: [http_2xx]
        kubernetes_sd_configs:
          - role: service
        relabel_configs:
          - source_labels:
              [__meta_kubernetes_service_annotation_prometheus_io_probe]
            action: keep
            regex: true
          - source_labels: [__address__]
            target_label: __param_target
          - target_label: __address__
            replacement: blackbox
          - source_labels: [__param_target]
            target_label: instance
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            target_label: kubernetes_name
      - job_name: "kubernetes-pods"
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - action: drop
            source_labels: [__meta_kubernetes_pod_container_init]
            regex: true
          - action: keep_if_equal
            source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels:
              [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name

Most of the beforehand patch is the configuration for the agent to scrape targets and can be ignored or copied. The important parts are the first few lines.
With the remoteWriteUrls an external data source is configured. Due to the fact that both services are running side-by-side in a single cluster, it is possible to use the cluster ip to route this traffic internally.

Both manifest locations added towards the environment overlay kustomization and the cicd environment should automatically install the agent.

Grafana

Building a collection of metrics is just one side of the medallion. The other side is displaying and reacting to changing metrics.
As always, start with a helm chart inflation.

./base/grafana/helmrelease.yaml

1
2
3
4
5
6
7
8
9
10
11

apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: grafana
releaseName: grafana
name: grafana
version: 6.50.5
repo: https://grafana.github.io/helm-charts
valuesInline: {}
IncludeCRDs: true
namespace: victoria-metrics

The next step is to add the patch for Grafana.
With the following patch, the deployment will be configured to the desired environment. For example, the ingress configuration provides all required information to access Grafana afterward.

The important part is the datasource configuration that provides the link between Grafana and the installed victoria metrics cluster.
The VMSelect application provides a dropin replacement Prometheus endpoint for Grafana to be consumed.

One downside of this used helm chart is that there is currently no support for a configuration reload sidecar container that refreshes the dashboards and configuration located in Kubernetes. Therefor, it is required to configure the default available dashboards within the dashboards block.

./env/homelab/patches/patch-grafana.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: grafana
valuesInline: 
  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources:
        - name: victoriametrics
          type: prometheus
          orgId: 1
          url: http://victoria-metrics-cluster-vmselect.victoria-metrics:8481/select/0/prometheus/
          access: proxy
          isDefault: true
          updateIntervalSeconds: 10
          editable: true

  dashboardProviders:
   dashboardproviders.yaml:
     apiVersion: 1
     providers:
     - name: 'default'
       orgId: 1
       folder: ''
       type: file
       disableDeletion: true
       editable: true
       options:
         path: /var/lib/grafana/dashboards/default

  dashboards:
    default:
      victoriametrics:
        gnetId: 11176
        revision: 18
        datasource: victoriametrics
      vmagent:
        gnetId: 12683
        revision: 7
        datasource: victoriametrics
      kubernetes:
        gnetId: 14205
        revision: 1
        datasource: victoriametrics

  ingress:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: selfsigned-ca-issuer
      kubernetes.io/ingress.class: traefik
      traefik.ingress.kubernetes.io/router.entrypoints: web, websecure
      traefik.ingress.kubernetes.io/router.tls: 'true'
      ingress.kubernetes.io/ssl-force-host: "true"
      ingress.kubernetes.io/ssl-redirect: "true"
    hosts:
      - grafana.lan
    tls:
     - secretName: grafana.lan
       hosts:
        - grafana.lan
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
      requests:
        cpu: 100m
        memory: 128Mi

Adding all folder and resources to their relevant kustomizations, and you should be welcomed with a semi-complete monitoring stack for your Kubernetes environment. Missing components like the node-exporter could easily be added to the same deployment process with the already shown approach.

As a small reminder: the complete deployment is described within the prepared repository under the following url https://github.com/deB4SH/Kustomize-Victoria-Metrics.

Hi all,
I’m currently working on refactoring the way to set up kubernetes clusters within the infrastructure of my current employer. (Role: Platform Engineer)
Due to growing configuration requirements and time-consuming decisions we’ve decided within our team that it is time to refactor the stack and try out something new.
The current setup is based on flux-cd with a self-written templating software to render manifests based on a single configuration file.
This configuration file is called config.yaml, who would have guessed that, and contains all critical information to bootstrap and deploy a new cluster environment.
Basic manifests are provided from an internal kubernetes service catalog which is version pinned for a cluster.
The rendered manifests are stored within a dedicated kubernetes-clusters repository (${cluster.name}/cluster.generated/${service.name}) and are initially deployed with a ci/cd approach to apply the tanzu kubernetes cluster and kickstart flux-cd on it.
After the initial setup: flux-cd picks up the stored manifest files within the kubernetes cluster repository and installs everything.

A catalog deployment from our kubernetes service catalog may look like:

NOTE: I will focus on the cluster part here. The service catalog is a collection of typical flux manifests (HelmRepository, HelmRelease) with a default configuration in it.

For the reference the following code snippet provides the Git Repository for flux-cd to pull manifests from.

1
2
3
4
5
6
7
8
9
10
11
12
13

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: service-catalog
  namespace: service-catalog
spec:
  interval: 10m0s
  ref:
    semver: 2.5.*
  secretRef:
    name: service-catalog-pat
  url: https://corporage-git-repository.corporate.tld/_git/kubernetes-service-catalog
  gitImplementation: libgit2

As starting point is always a flux-cd kustomization that picks up the provided manifests within the service catalog.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: external-dns
  namespace: service-catalog
spec:
  timeout: 5m
  interval: 10m0s
  retryInterval: 15s
  path: ./dns/external-dns
  prune: true
  sourceRef:
    kind: GitRepository
    name: service-catalog
    namespace: service-catalog
  validation: client
  patches:
  - placeholder-patch

As you may already see we’re adding a (not so valid) blank placeholder patch that is required for the next steps.
To change values for the helm release we are appending a patch to the flux kustomization.
Due to the limitation that flux doesn’t have any filesystem at this point of deployment you can’t use something like strategic merges with files.

The following listing shows an example patch that is appended on the flux kustomization at the last position within the patches list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

- op: add
  path: "/spec/patches/-"
  value:
    target:
      kind: HelmRelease
      name: external-dns
    patch: |-
      apiVersion: helm.toolkit.fluxcd.io/v2beta1
      kind: HelmRelease
      metadata:
        name: external-dns
        namespace: external-dns
      spec:
        values:
          provider: pdns
          pdns:
            apiUrl: "https://powerdns.corporate.tld"
            apiPort: "443"
            secretName: "external-dns"
          domainFilters:
            - devops-test-cluster.k8s.corporate.tld
            - devops-test-cluster.corporate.tld
          txtOwnerId: "devops-test-cluster"
          extraArgs:
            pdns-tls-enabled: false

With this mechanism we are able to change the provided default configuration for each cluster environment and render those patches dynamically base on the configuration within the config.yaml.

Everything is hooked together with a simple kustomization.yaml which is controlled by a top-level kustomization.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- namespace.yaml
- release-external-dns.yaml


patches:
  - path: patches/settings.yaml
    target:
      group: kustomize.toolkit.fluxcd.io
      kind: Kustomization
      name: external-dns
  - path: patches/remove-placeholder.yaml
    target:
      group: kustomize.toolkit.fluxcd.io
      kind: Kustomization

The remove placeholder patch is nothing special. It just removed the first element.

1
2

- op: remove
  path: "/spec/patches/0"

Deployments of our customers are located within a different directory (${cluster.name}/cluster.custom/*) that is picked up by flux or a completly different repository.

This approach provides us a highly customizable and secured state of deployment.
Due to the reproducable rendered templates we are able to throw away the cluster.generated folder for each upgrade and can work in a fire and forget environment.

The downside is that upgrades on our infrastructure or changes to it are time consuming, due to the fact that everything is maintained within it’s dedicated reposiotory. There is no centralization.
Every change needs to be done within the golden configuration file for each environment, which results in a render of the templates, which also results in a validation of the rendered manifests and a checkin of every required change.
Each deployment has multiple files for patches and changes to the base-configuration which can’t be loaded from a file or merged with strategic merges.

NOTE: The process sounds fun when it’s done for a single environment.. if there are twenty.. good luck with that.

The scalability of this approach lacks behind and each new cluster creates a new context for our team-members to keep track of.
Without any action against this: it is going to be a big pain-point in daily operation.

To reduce the costs of maintenance for every environment we want to create a cockpit from which we can deploy manifests remotly.
We also want to aggregate the configuration files for each environment within one repository, so you dont need to switch contexts all the time.

Within a small timeframe and some proof-of-concept work we decided to go with argo-cd and it’s capabilities to deploy manifests into different clusters and maintain a smaller footprint git repository.

Proposed Proof of Concept Setup

To tackle the presented issues from the beforehand chapter I like to take a deeper dive into the structure of our current argocd stack and the proof of concept stack before that.

Removing context switches was one of our primary goals. Achieving this resulted in a complete restructuring of our kubernetes service catalog and deployment strategy. Everything from manifests to deployment configuration should be located in a single repository. The result of this restructuring is shown in the following tree view listing.

1
2
3
4
5
6

./
├── argocd-applicationsets
├── helm
├── kustomize
├── README.md
└── cluster-environments

As always it is necessary to question your own concepts and dicisions.
The initial setup was split into two repositories, one which contains all manifests, one that hold the deployment configuration.
The advantage of collecting everything is kind of obvious but we reduced the amount of context changes with tremendously. The workflow of editing and adding new software to our general deployment is now streamlined.

On of the next migration steps was to ease up the usage of helm.

Setting up Helm Subcharts

Many of our applications we are providing to our internal customers are provided via helm. Helm allows it’s users to inline values or load them directly from configuration files.

To allow extendability of one helm chart we’ve build our own helm charts ontop of the existing ones and reference the desired charts as subcharts/dependencies.

Structurally our setup for helm charts looks like the following tree view.

1
2
3
4
5
6

./
├── Chart.yaml
├── templates
│   ├── external-dns-secret.yaml
│   └── root-ca-01.yaml
└── values.yaml

The pivotal point of helm is always the Chart.yaml which contains every relevant information to install the chart on a cluster.

1
2
3
4
5
6
7
8

apiVersion: v2
name: external-dns
version: 1.0.0
description: This Chart deploys external-dns.
dependencies:
  - name: external-dns
    version: 6.13.*
    repository: https://charts.bitnami.com/bitnami

As shown within the listing we are building our chart ontop of a dependency that installs the application itself.
Configuration changes are done within the values.yaml and may look like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

pdns_api_key: overlay_me

external-dns:
  pdns:
    apiUrl: "https://powerdns.corporate.tld"
    apiPort: "443"
    secretName: "external-dns"

  txtOwnerId: "your_awesome_textowner_id"

  image:
    registry: proxy.corporate.tld/hub.docker.com
  rbac:
    pspEnabled: true
  provider: pdns

  extraVolumeMounts:
    - name: certs
      mountPath: "/etc/ssl/certs/root-ca-01.pem"
      subPath: "root-ca-01.pem"

  extraVolumes:
    - name: certs
      configMap:
        name: root-ca-01

  extraArgs:
    pdns-tls-enabled: false
    tls-ca: /etc/ssl/certs/root-ca-01.pem

The important point is that values of your dependencies require a correct indentation under the same name as configured within the chart.

Additional manifests, like the root-ca, are provided from within the template directory. With this approach you can easily provide additional manifests with the default installation.

1
2
3
4
5
6
7
8
9
10

apiVersion: v1
kind: ConfigMap
metadata:
  name: root-ca-01
data:
  root-ca-01.pem: |
    -----BEGIN CERTIFICATE-----
    MII[....redacted....]Bg==
    -----END CERTIFICATE-----

Based on this approach it is also possible to repack multiple helm charts into an single installation. Keep in mind that building a single collection of multiple charts in one single chart may bring additional code complexity and increases the hurdles to maintain.

As next step I want to deploy the newly created helm chart onto multiple clusters without referencing it everywhere.
ArgoCD provides a nice approach to create applications in a dynamic fashion.
Within the next chapter I like to present the application set of argocd and our current approach to provision the charts on each maintained cluster.

ArgoCD Application Set

The argocd application set provides the functionality to generate automatically applications based on set generators. To keep the focus on the general structure I removed a lot of moving parts from the next ApplicationSet and reduced it to the critical components.

In our current infrastructure we are following the general consense to provide all projects a development cluster for their daily-task and a seperate production cluster for the actualy live service.
With the following ApplicationSet we are generating for each cluster, which is labeld with dev or prd, an application to rollout the external dns.

NOTE: The cluster connection is done beforehand in a seperate task. To add clusters to a argocd instance please use the argocd cli or create the service accounts on your own.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: external-dns
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            env: dev
    - clusters:
        selector:
          matchLabels:
            env: prd
  template:
    metadata:
      name: "{{name}}-external-dns"
      annotations:
        argocd.argoproj.io/manifest-generate-paths: ".;.."
    spec:
      project: bootstrap
      source:
        repoURL: https://corporate-repository-argocd.corporate.tld/kubernetes-service-catalog
        targetRevision: main
        path: "./helm/dns/external-dns"
        helm:
          releaseName: "external-dns"
          valueFiles:
            - "values.yaml"
            - "../../../values/{{name}}/dns/external-dns/values.yaml"
      destination:
        name: "{{name}}"
        namespace: "external-dns"
      syncPolicy:
        automated:
          prune: false
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
        retry:
          limit: 2

Through this deployment configuration we achieve that every connected cluster receives the same artifacts defined within our kubernetes service catalog.
Overlaying cluster specific information is done via a second helm value file that merges into the first on.

1
2
3
4
5
6

pdns_api_key: T[...redacted...]c=

external-dns:
  domainFilters:
    - example-cluster.corporate.tld
  txtOwnerId: "example-cluster"

Through this method we are reducing the required configuration tremendously. Only cluster specific configuration resides within the second values file. Generic corporate related configuration like proxy configuration and generic purpose configuration resides inside the first.
We also achieved with this approach that all values are directly filebased and not embedded within any kustomize-alike patch file.

Conclusion

With our switch from fluxcd to argocd we were able to streamline our tasks as platform team and provision our client clusters directly without the hassle of managing multiple cluster configuration within several places. We were able to reduce the code complexity that was building up with more and more services running per default on our platform. We were able to reduce the required time to maintain numerous clusters with a small team and deliver updates quick to each environment.
We were able to scale our platform accordingly to our needs without the hassle of maintain a collection of different scopes. We were able to reduce our time to market by hours due to a smaller configuration footprint.
We were able to onboard new collegues easily due to the reduced complexity.

As a clarification: Fluxcd is not a bad tool. Do not get me wrong. It simply did not cater our needs.

I hope through this devops story you get a small glimpse into my daily business. If there are any open questions: please feel free to contact me.

Hi all,
it’s been some time since I wrote my last article here. I switched jobs, started reading a lot more, worked on different projects so that blogging came way to short.
With the new year I want to try to write at least a monthly entry with one new thing I learned and want to share.
The following post describes the helm chart capabilities of kustomize and how to use it in your workflow.

This article is not focused on the topic if templates are a good thing or how to template with kustomize. Feel free to leave a comment over in mastodon: @deb4sh@hachyderm.io

Motivation

From time to time it is nessecary to edit and helm-chart further than the estimated approach of the original author of one helmchart.
A common usecase is editing the service afterwards to fit your needs. I often need to patch those after a deployment to append loadbalancer configuration or the ip that should be used. This is often a two step approach by installing the helm chart first and apply the manifest with patches afterwards.
An another example could be the overlaying of values for different environments.

With the helmChart and the Helm Chart Inflation Generator you can unpack the helm chart withing kustomize and handle the resulting manifests directly.

How to do

For demonstation purposes I’ve set up a kustomized helm installation for victoria metrics under the following link.

Everything starts with a HelmChartInflationGenerator which is found under /base/*/helmrelease.yaml.

1
2
3
4
5
6
7
8
9
10
11

apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: victoria-metrics-cluster
releaseName: victoria-metrics-cluster
name: victoria-metrics-cluster
version: 0.9.52
repo: https://victoriametrics.github.io/helm-charts/
valuesInline: {}
IncludeCRDs: true
namespace: victoria-metrics

The generation describes the location of the victoria metrics helm chart and every relevant metadata with it like the name, version, values and namespace.
The next step is to set up an aggregator for all your Helm Charts and patch all values for installation.
This is done within the /env/*/kustomization.yaml.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: victoria-metrics

resources:
  - ../../base/victoria-metrics-cluster
  - ../../base/victoria-metrics-agent
  - ../../base/grafana

patchesStrategicMerge:
  - patches/patch-cluster-values.yaml
  - patches/patch-agent-values.yaml
  - patches/patch-grafana-values.yaml

The kustomization collects all configured bases and patches their values accordingly.
Inside the following listing is the patch described for the victoria metrics cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: victoria-metrics-cluster
valuesInline: 
  rbac:
    create: true
    pspEnabled: false
  vmselect:
    replicaCount: 1
    podAnnotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8481"

  vminsert:
    replicaCount: 1
    podAnnotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8480"
    extraArgs:
      envflag.enable: "true"
      envflag.prefix: VM_
      loggerFormat: json

  vmstorage:
    replicaCount: 1
    persistentVolume:
      storageClass: longhorn
    podAnnotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8482"

Inline values provide an overlay for the default values that are available withing the helm chart.
This patching mechanism can be used to overlay the values even further if installations share the same base.

As last step we need to instruct kustomize to generate the ressources from the configured helmchart and values.
This could be done with the following listing.

1
2
3
4
5
6
7

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: victoria-metrics

generators:
  - ../../env/homelab/

The generator points towards the previously described environment configuration.
When run with kustomize build --enable-helm you should receive the rendered helm chart as kubernetes manifests.

Conclusion

With the helmchart and HelmChartInflationGenerator we are able to render helmcharts nativly in kustomize.
Due to the general patching mechanism with kustomize we can manipulate the resulting manifests with ease before everything gets deployed towards the cluster.

The generator needs some work… for example is the --enable-helm command line dependency something that a lot of people are going to script away.
There are also some heafty changes located within the milestone v5.0.0 for kustomize.

IMHO: It’s a nice progress for kustomize to allow helm charts directly within my workflow. I can remove one application from build-stack to rollout services to my clusters.

Hi all,
due to the awesome progress with proton and the integration in steam.. lets be honest… there is no need for Windows if you are not playing games that are competetiv nor secured via kernel-level anticheat. I’m also working most of my time with containers and kubernetes environments. Integrations of those two in windows are more or less not existing. Due to the wsl it gets better but it’s not quite native yet. Switching to an open operating system was the next logical step in my mind sooo.. here we are.

Due to the being of debian the packages are a bit dated and considered stable, which is fine for everyday software but not that amazing if you want to get your new gpu working. Debian is currently provinding on all branches a 470.x.y version for nvidia-driver, which is a bit dated. On date of writing this we are currently by 495.x.y.

With the following steps you can update your graphic card drivers from 470.x.y to 495.x.y.
Keep in mind.. READ BEFORE COPY PASTING.

Download the latest drivers:

Downloading the latest is easiest via an existing browser and the unix driver archive of nvidia.
Simply head towards: https://www.nvidia.com/en-us/drivers/unix/ and checkout the latest feature branch version for your system. In most cases x86/amd64.

Store it somewhere you are able to remember. For example: /home/your_user/nvidia/NVIDIA-Linux-x86_64-495.46.run

Uninstall existing drivers (if available)

Depending if this is a system you’ve already used and installed drivers from any debian repository uninstall them. If you keep them installed the kernel-modules are still available and loaded after an reboot, which makes installing the driver directly from nvidia impossible.

1

apt-get remove --purge '^nvidia-.*'

Preparing for reboot!

Next on the list: preparing the system for the reboot into the multi-user.target.
To install the nvidia driver directly you need to setup serveral things.

Install headers for your current kernel, build-essentials, libglvnd-dev and pkg-config.

1

apt install linux-headers-$(uname -r) build-essential libglvnd-dev pkg-config

Also create, if not existing, a new file under /etc/modprobe.d/blacklist-nouveau.conf with following content.

1
2

blacklist nouveau
options nouveau modeset=0

With this you are blacklisting nouveau drivers.

Next we need to update kernel-initramfs.

1

update-initramfs -u

At last we need to setup the default target at boot and reboot the system.

1
2

systemctl set-default multi-user.target
reboot now

Install Nvidia Drivers

After a reboot you should be greeted with login prompt. Enter your credentials.
Next head towards your created folder with the driver inside. Execute the following inside.

1

bash NVIDIA-Linux-x86_64-495.46.run

If executed properly, you should see and loading bar growing. After a short while your should be greeted with questions.

• Install NVIDIA’s 32-bit compatibility libraries?
• Would you like to run the nvidia-xconfig utility to automatically update your X configuration file so that the NVIDIA X driver will be used when you restart X?

Both questions should be answered with yes.

You should see a process bar that indicates the status of building your new kernel with the nvidia driver bundled. After it finished everything should be set up and ready to go.
You’ve installed the driver manually.

The last step is to return to an graphical interface after boot, which is acomplished by executing the following command.

1

systemctl set-default graphical.target

After an fresh reboot you should be greeted by your desktop environment / login interface.

You can check if the correct driver is running with nvidia-smi

1
2
3
4
5
6
7
8
9
10
11
12
13
14

/home/b4sh [core@debian] [13:55]
> nvidia-smi
Tue Dec 21 13:55:38 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.46       Driver Version: 495.46       CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:1C:00.0  On |                  N/A |
|  0%   46C    P0    41W / 260W |    962MiB /  7979MiB |      6%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+