Writing a ClusterClass
A ClusterClass becomes more useful and valuable when it can be used to create many Cluster of a similar shape. The goal of this document is to explain how ClusterClasses can be written in a way that they are flexible enough to be used in as many Clusters as possible by supporting variants of the same base Cluster shape.
Table of Contents
- Basic ClusterClass
- ClusterClass with MachineHealthChecks
- ClusterClass with patches
- Advanced features of ClusterClass with patches
- JSON patches tips & tricks
Basic ClusterClass
The following example shows a basic ClusterClass. It contains templates to shape the control plane, infrastructure and workers of a Cluster. When a Cluster is using this ClusterClass, the templates are used to generate the objects of the managed topology of the Cluster.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: docker-clusterclass-v0.1.0
spec:
controlPlane:
ref:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlaneTemplate
name: docker-clusterclass-v0.1.0
namespace: default
machineInfrastructure:
ref:
kind: DockerMachineTemplate
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
name: docker-clusterclass-v0.1.0
namespace: default
infrastructure:
ref:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerClusterTemplate
name: docker-clusterclass-v0.1.0-control-plane
namespace: default
workers:
machineDeployments:
- class: default-worker
template:
bootstrap:
ref:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
name: docker-clusterclass-v0.1.0-default-worker
namespace: default
infrastructure:
ref:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
name: docker-clusterclass-v0.1.0-default-worker
namespace: default
The following example shows a Cluster using this ClusterClass. In this case a KubeadmControlPlane
with the corresponding DockerMachineTemplate
, a DockerCluster
and a MachineDeployment
with
the corresponding KubeadmConfigTemplate
and DockerMachineTemplate
will be created. This basic
ClusterClass is already very flexible. Via the topology on the Cluster the following can be configured:
.spec.topology.version
: the Kubernetes version of the Cluster.spec.topology.controlPlane
: ControlPlane replicas and their metadata.spec.topology.workers
: MachineDeployments and their replicas, metadata and failure domain
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: my-docker-cluster
spec:
topology:
class: docker-clusterclass
version: v1.22.4
controlPlane:
replicas: 3
metadata:
labels:
cpLabel: cpLabelValue
annotations:
cpAnnotation: cpAnnotationValue
workers:
machineDeployments:
- class: default-worker
name: md-0
replicas: 4
metadata:
labels:
mdLabel: mdLabelValue
annotations:
mdAnnotation: mdAnnotationValue
failureDomain: region
Best practices:
- The ClusterClass name should be generic enough to make sense across multiple clusters, i.e. a name which corresponds to a single Cluster, e.g. “my-cluster”, is not recommended.
- Try to keep the ClusterClass names short and consistent (if you publish multiple ClusterClasses).
- As a ClusterClass usually evolves over time and you might want to rebase Clusters from one version of a ClusterClass to another, consider including a version suffix in the ClusterClass name. For more information about changing a ClusterClass please see: Changing a ClusterClass.
- Prefix the templates used in a ClusterClass with the name of the ClusterClass.
- Don’t reuse the same template in multiple ClusterClasses. This is automatically taken care of by prefixing the templates with the name of the ClusterClass.
ClusterClass with MachineHealthChecks
MachineHealthChecks
can be configured in the ClusterClass for the control plane and for a
MachineDeployment class. The following configuration makes sure a MachineHealthCheck
is
created for the control plane and for every MachineDeployment
using the default-worker
class.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: docker-clusterclass-v0.1.0
spec:
controlPlane:
...
machineHealthCheck:
maxUnhealthy: 33%
nodeStartupTimeout: 15m
unhealthyConditions:
- type: Ready
status: Unknown
timeout: 300s
- type: Ready
status: "False"
timeout: 300s
workers:
machineDeployments:
- class: default-worker
...
machineHealthCheck:
unhealthyRange: "[0-2]"
nodeStartupTimeout: 10m
unhealthyConditions:
- type: Ready
status: Unknown
timeout: 300s
- type: Ready
status: "False"
timeout: 300s
ClusterClass with patches
As shown above, basic ClusterClasses are already very powerful. But there are cases where more powerful mechanisms are required. Let’s assume you want to manage multiple Clusters with the same ClusterClass, but they require different values for a field in one of the referenced templates of a ClusterClass.
A concrete example would be to deploy Clusters with different registries. In this case,
every cluster needs a Cluster-specific value for .spec.kubeadmConfigSpec.clusterConfiguration.imageRepository
in KubeadmControlPlane
. Use cases like this can be implemented with ClusterClass patches.
Defining variables in the ClusterClass
The following example shows how variables can be defined in the ClusterClass. A variable definition specifies the name and the schema of a variable and if it is required. The schema defines how a variable is defaulted and validated. It supports a subset of the schema of CRDs. For more information please see the godoc.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: docker-clusterclass-v0.1.0
spec:
...
variables:
- name: imageRepository
required: true
schema:
openAPIV3Schema:
type: string
description: ImageRepository is the container registry to pull images from.
default: registry.k8s.io
example: registry.k8s.io
Defining patches in the ClusterClass
The variable can then be used in a patch to set a field on a template referenced in the ClusterClass.
The selector
specifies on which template the patch should be applied. jsonPatches
specifies which JSON
patches should be applied to that template. In this case we set the imageRepository
field of the
KubeadmControlPlaneTemplate
to the value of the variable imageRepository
. For more information
please see the godoc.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: docker-clusterclass-v0.1.0
spec:
...
patches:
- name: imageRepository
definitions:
- selector:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlaneTemplate
matchResources:
controlPlane: true
jsonPatches:
- op: add
path: /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/imageRepository
valueFrom:
variable: imageRepository
Setting variable values in the Cluster
After creating a ClusterClass with a variable definition, the user can now provide a value for the variable in the Cluster as in the example below.
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: my-docker-cluster
spec:
topology:
...
variables:
- name: imageRepository
value: my.custom.registry
Advanced features of ClusterClass with patches
This section will explain more advanced features of ClusterClass patches.
MachineDeployment variable overrides
If you want to use many variations of MachineDeployments in Clusters, you can either define a MachineDeployment class for every variation or you can define patches and variables to make a single MachineDeployment class more flexible.
In the following example we make the instanceType
of a AWSMachineTemplate
customizable.
First we define the workerMachineType
variable and the corresponding patch:
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: aws-clusterclass-v0.1.0
spec:
...
variables:
- name: workerMachineType
required: true
schema:
openAPIV3Schema:
type: string
default: t3.large
patches:
- name: workerMachineType
definitions:
- selector:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSMachineTemplate
matchResources:
machineDeploymentClass:
names:
- default-worker
jsonPatches:
- op: add
path: /spec/template/spec/instanceType
valueFrom:
variable: workerMachineType
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSMachineTemplate
metadata:
name: aws-clusterclass-v0.1.0-default-worker
spec:
template:
spec:
# instanceType: workerMachineType will be set by the patch.
iamInstanceProfile: "nodes.cluster-api-provider-aws.sigs.k8s.io"
---
...
In the Cluster resource the workerMachineType
variable can then be set cluster-wide and
it can also be overridden for an individual MachineDeployment.
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: my-aws-cluster
spec:
...
topology:
class: aws-clusterclass-v0.1.0
version: v1.22.0
controlPlane:
replicas: 3
workers:
machineDeployments:
- class: "default-worker"
name: "md-small-workers"
replicas: 3
variables:
overrides:
# Overrides the cluster-wide value with t3.small.
- name: workerMachineType
value: t3.small
# Uses the cluster-wide value t3.large.
- class: "default-worker"
name: "md-large-workers"
replicas: 3
variables:
- name: workerMachineType
value: t3.large
Builtin variables
In addition to variables specified in the ClusterClass, the following builtin variables can be referenced in patches:
builtin.cluster.{name,namespace}
builtin.cluster.topology.{version,class}
builtin.cluster.network.{serviceDomain,services,pods,ipFamily}
builtin.controlPlane.{replicas,version,name}
- Please note, these variables are only available when patching control plane or control plane machine templates.
builtin.controlPlane.machineTemplate.infrastructureRef.name
- Please note, these variables are only available when using a control plane with machines and when patching control plane or control plane machine templates.
builtin.machineDeployment.{replicas,version,class,name,topologyName}
- Please note, these variables are only available when patching the templates of a MachineDeployment
and contain the values of the current
MachineDeployment
topology.
- Please note, these variables are only available when patching the templates of a MachineDeployment
and contain the values of the current
builtin.machineDeployment.{infrastructureRef.name,bootstrap.configRef.name}
- Please note, these variables are only available when patching the templates of a MachineDeployment
and contain the values of the current
MachineDeployment
topology.
- Please note, these variables are only available when patching the templates of a MachineDeployment
and contain the values of the current
Builtin variables can be referenced just like regular variables, e.g.:
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: docker-clusterclass-v0.1.0
spec:
...
patches:
- name: clusterName
definitions:
- selector:
...
jsonPatches:
- op: add
path: /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/controllerManager/extraArgs/cluster-name
valueFrom:
variable: builtin.cluster.name
Tips & Tricks
Builtin variables can be used to dynamically calculate image names. The version used in the patch
will always be the same as the one we set in the corresponding MachineDeployment (works the same way
with .builtin.controlPlane.version
).
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: docker-clusterclass-v0.1.0
spec:
...
patches:
- name: customImage
description: "Sets the container image that is used for running dockerMachines."
definitions:
- selector:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
matchResources:
machineDeploymentClass:
names:
- default-worker
jsonPatches:
- op: add
path: /spec/template/spec/customImage
valueFrom:
template: |
kindest/node:{{ .builtin.machineDeployment.version }}
Complex variable types
Variables can also be objects, maps and arrays. An object is specified with the type object
and
by the schemas of the fields of the object. A map is specified with the type object
and the schema
of the map values. An array is specified via the type array
and the schema of the array items.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: docker-clusterclass-v0.1.0
spec:
...
variables:
- name: httpProxy
schema:
openAPIV3Schema:
type: object
properties:
# Schema of the url field.
url:
type: string
# Schema of the noProxy field.
noProxy:
type: string
- name: mdConfig
schema:
openAPIV3Schema:
type: object
additionalProperties:
# Schema of the map values.
type: object
properties:
osImage:
type: string
- name: dnsServers
schema:
openAPIV3Schema:
type: array
items:
# Schema of the array items.
type: string
Objects, maps and arrays can be used in patches either directly by referencing the variable name, or by accessing individual fields. For example:
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: docker-clusterclass-v0.1.0
spec:
...
jsonPatches:
- op: add
path: /spec/template/spec/httpProxy/url
valueFrom:
# Use the url field of the httpProxy variable.
variable: httpProxy.url
- op: add
path: /spec/template/spec/customImage
valueFrom:
# Use the osImage field of the mdConfig variable for the current MD class.
template: "{{ (index .mdConfig .builtin.machineDeployment.class).osImage }}"
- op: add
path: /spec/template/spec/dnsServers
valueFrom:
# Use the entire dnsServers array.
variable: dnsServers
- op: add
path: /spec/template/spec/dnsServer
valueFrom:
# Use the first item of the dnsServers array.
variable: dnsServers[0]
Tips & Tricks
Complex variables can be used to make references in templates configurable, e.g. the identityRef
used in AzureCluster
.
Of course it’s also possible to only make the name of the reference configurable, including restricting the valid values
to a pre-defined enum.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: azure-clusterclass-v0.1.0
spec:
...
variables:
- name: clusterIdentityRef
schema:
openAPIV3Schema:
type: object
properties:
kind:
type: string
name:
type: string
Even if OpenAPI schema allows defining free form objects, e.g.
variables:
- name: freeFormObject
schema:
openAPIV3Schema:
type: object
User should be aware that the lack of the validation of users provided data could lead to problems when those values are used in patch or when the generated templates are created (see e.g. 6135).
As a consequence we recommend avoiding this practice while we are considering alternatives to make it explicit for the ClusterClass authors to opt-in in this feature, thus accepting the implied risks.
Using variable values in JSON patches
We already saw above that it’s possible to use variable values in JSON patches. It’s also possible to calculate values via Go templating or to use hard-coded values.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: docker-clusterclass-v0.1.0
spec:
...
patches:
- name: etcdImageTag
definitions:
- selector:
...
jsonPatches:
- op: add
path: /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/etcd
valueFrom:
# This template is first rendered with Go templating, then parsed by
# a YAML/JSON parser and then used as value of the JSON patch.
# For example, if the variable etcdImageTag is set to `3.5.1-0` the
# .../clusterConfiguration/etcd field will be set to:
# {"local": {"imageTag": "3.5.1-0"}}
template: |
local:
imageTag: {{ .etcdImageTag }}
- name: imageRepository
definitions:
- selector:
...
jsonPatches:
- op: add
path: /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/imageRepository
# This hard-coded value is used directly as value of the JSON patch.
value: "my.custom.registry"
Tips & Tricks
Templates can be used to implement defaulting behavior during JSON patch value calculation. This can be used if the simple constant default value which can be specified in the schema is not enough.
valueFrom:
# If .vnetName is set, it is used. Otherwise, we will use `{{.builtin.cluster.name}}-vnet`.
template: "{{ if .vnetName }}{{.vnetName}}{{else}}{{.builtin.cluster.name}}-vnet{{end}}"
When writing templates, a subset of functions from the Sprig library can be used to
write expressions, e.g., {{ .name | upper }}
. Only functions that are guaranteed to evaluate to the same result
for a given input are allowed (e.g. upper
or max
can be used, while now
or randAlpha
cannot be used).
Optional patches
Patches can also be conditionally enabled. This can be done by configuring a Go template via enabledIf
.
The patch is then only applied if the Go template evaluates to true
. In the following example the httpProxy
patch is only applied if the httpProxy
variable is set (and not empty).
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: docker-clusterclass-v0.1.0
spec:
...
variables:
- name: httpProxy
schema:
openAPIV3Schema:
type: string
patches:
- name: httpProxy
enabledIf: "{{ if .httpProxy }}true{{end}}"
definitions:
...
Tips & Tricks:
Hard-coded values can be used to test the impact of a patch during development, gradually roll out patches, etc. .
enabledIf: false
A boolean variable can be used to enable/disable a patch (or “feature”). This can have opt-in or opt-out behavior depending on the default value of the variable.
enabledIf: "{{ .httpProxyEnabled }}"
Of course the same is possible by adding a boolean variable to a configuration object.
enabledIf: "{{ .httpProxy.enabled }}"
Builtin variables can be leveraged to apply a patch only for a specific Kubernetes version.
enabledIf: '{{ semverCompare "1.21.1" .builtin.controlPlane.version }}'
With semverCompare
and coalesce
a feature can be enabled in newer versions of Kubernetes for both KubeadmConfigTemplate and KubeadmControlPlane.
enabledIf: '{{ semverCompare "^1.22.0" (coalesce .builtin.controlPlane.version .builtin.machineDeployment.version )}}'
Version-aware patches
In some cases the ClusterClass authors want a patch to be computed according to the Kubernetes version in use.
While this is not a problem “per se” and it does not differ from writing any other patch, it is important to keep in mind that there could be different Kubernetes version in a Cluster at any time, all of them accessible via built in variables:
builtin.cluster.topology.version
defines the Kubernetes version fromcluster.topology
, and it acts as the desired Kubernetes version for the entire cluster. However, during an upgrade workflow it could happen that some objects in the Cluster are still at the older version.builtin.controlPlane.version
, represent the desired version for the control plane object; usually this version changes immediately aftercluster.topology.version
is updated (unless there are other operations in progress preventing the upgrade to start).builtin.machineDeployment.version
, represent the desired version for each specific MachineDeployment object; this version changes only after the upgrade for the control plane is completed, and in case of many MachineDeployments in the same cluster, they are upgraded sequentially.
This info should provide the bases for developing version-aware patches, allowing the patch author to determine when a patch should adapt to the new Kubernetes version by choosing one of the above variables. In practice the following rules applies to the most common use cases:
- When developing a version-aware patch for the control plane,
builtin.controlPlane.version
must be used. - When developing a version-aware patch for MachineDeployments,
builtin.machineDeployment.version
must be used.
Tips & Tricks:
Sometimes users need to define variables to be used by version-aware patches, and in this case it is important to keep in mind that there could be different Kubernetes versions in a Cluster at any time.
A simple approach to solve this problem is to define a map of version-aware variables, with the key of each item being the Kubernetes version. Patch could then use the proper builtin variables as a lookup entry to fetch the corresponding values for the Kubernetes version in use by each object.
JSON patches tips & tricks
JSON patches specification RFC6902 requires that the target of add operation must exist.
As a consequence ClusterClass authors should pay special attention when the following conditions apply in order to prevent errors when a patch is applied:
- the patch tries to
add
a value to an array (which is a slice in the corresponding go struct) - the slice was defined with
omitempty
- the slice currently does not exist
A workaround in this particular case is to create the array in the patch instead of adding to the non-existing one. When creating the slice, existing values would be overwritten so this should only be used when it does not exist.
The following example shows both cases to consider while writing a patch for adding a value to a slice.
This patch targets to add a file to the files
slice of a KubeadmConfigTemplate
which has omitempty set.
This patch requires the key .spec.template.spec.files
to exist to succeed.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: my-clusterclass
spec:
...
patches:
- name: add file
definitions:
- selector:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
jsonPatches:
- op: add
path: /spec/template/spec/files/-
value:
content: Some content.
path: /some/file
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
name: "quick-start-default-worker-bootstraptemplate"
spec:
template:
spec:
...
files:
- content: Some other content
path: /some/other/file
This patch would overwrite an existing slice at .spec.template.spec.files
.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
name: my-clusterclass
spec:
...
patches:
- name: add file
definitions:
- selector:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
jsonPatches:
- op: add
path: /spec/template/spec/files
value:
- content: Some content.
path: /some/file
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
name: "quick-start-default-worker-bootstraptemplate"
spec:
template:
spec:
...