apiVersion: apache.org/v1alpha1 kind: Tomcat metadata: name: example-app spec: replicaCount: 2
This guide outlines Helm chart support in the Operator SDK and walks through an
example of building and running an Nginx Operator with the operator-sdk CLI
tool that uses an existing Helm chart.
The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way. This framework includes the Operator SDK, which assists developers in bootstrapping and building an Operator based on their expertise without requiring knowledge of Kubernetes API complexities.
One of the Operator SDK’s options for generating an Operator project includes leveraging an existing Helm chart to deploy Kubernetes resources as a unified application, without having to write any Go code. Such Helm-based Operators are designed to excel at stateless applications that require very little logic when rolled out, because changes should be applied to the Kubernetes objects that are generated as part of the chart. This may sound limiting, but can be sufficient for a surprising amount of use-cases as shown by the proliferation of Helm charts built by the Kubernetes community.
The main function of an Operator is to read from a custom object that represents
your application instance and have its desired state match what is running. In
the case of a Helm-based Operator, the object’s spec field is a list of
configuration options that are typically described in Helm’s values.yaml file.
Instead of setting these values with flags using the Helm CLI (e.g., helm
install -f values.yaml), you can express them within a Custom Resource (CR),
which, as a native Kubernetes object, enables the benefits of RBAC applied to it
and an audit trail.
For an example of a simple CR called Tomcat:
apiVersion: apache.org/v1alpha1 kind: Tomcat metadata: name: example-app spec: replicaCount: 2
The replicaCount value, 2 in this case, is propagated into the chart’s
templates where following is used:
{{ .Values.replicaCount }}
After an Operator is built and deployed, you can deploy a new instance of an app
by creating a new instance of a CR, or list the different instances running in
all environments using a kubectl or oc command:
$ kubectl get Tomcats --all-namespaces
There is no need to use the Helm CLI or install Tiller; Helm-based Operators import code from the Helm project. All you need to do is have an instance of the Operator running and register the CR with a Custom Resource Definition (CRD). And because it obeys RBAC, you can more easily prevent production changes.
See the following sections for instructions on installing the Operator SDK to build and run your own Helm-based Operator.
The Operator SDK has a CLI tool that assists developers in creating, building, and deploying a new Operator project. You can install the SDK CLI on your workstation so you are prepared to start authoring your own Operators.
Clone an operator-sdk repository:
$ mkdir -p $GOPATH/src/github.com/operator-framework $ cd $GOPATH/src/github.com/operator-framework $ git clone https://github.com/operator-framework/operator-sdk $ cd operator-sdk
Check out the desired release branch:
$ git checkout master
Install the SDK CLI tool:
$ make dep $ make install
This installs the CLI binary operator-sdk at $GOPATH/bin.
Verify that the CLI tool was installed correctly:
$ operator-sdk -h
This procedure walks through an example of building a simple Nginx Operator powered by a Helm chart using tools and libraries provided by the Operator SDK.
|
It is best practice to build a new Operator for each chart. This can allow for
more native-behaving Kubernetes APIs (e.g., |
Operator SDK CLI installed on the development workstation
Access to a Kubernetes-based cluster v1.11.3+ (for example OKD 4.0)
using an account with cluster-admin permissions
kubectl v1.11.3+
(can alternatively use oc)
Create a new project.
To create a new Helm-based, namespace-scoped nginx-operator project, use the
operator-sdk new command:
$ operator-sdk new nginx-operator \ --api-version=example.com/v1alpha1 --kind=Nginx --type=helm $ cd nginx-operator
|
See Appendices to learn about the project directory structure created by the previous commands. |
This creates the nginx-operator project specifically for watching the Nginx
resource with APIVersion example.com/v1apha1 and Kind Nginx.
A namespace-scoped Operator (the default) watches and manages resources in a single namespace, whereas a cluster-scoped operator watches and manages resources cluster-wide. Namespace-scoped operators are preferred because of their flexibility. They enable decoupled upgrades, namespace isolation for failures and monitoring, and differing API definitions.
However, there are use cases where a cluster-scoped operator may make sense. For
example, the cert-manager operator is often deployed with cluster-scoped
permissions and watches so that it can manage issuing certificates for an entire
cluster.
If you would like to create your nginx-operator project to be cluster-scoped,
use the following operator-sdk new command instead:
$ operator-sdk new nginx-operator \
--cluster-scoped --api-version=example.com/v1alpha1 \
--kind=Nginx --type=helm
Using the --cluster-scoped flag will scaffold the new Operator with the
following modifications:
deploy/operator.yaml: Set WATCH_NAMESPACE="" instead of setting it to the
Pod’s namespace.
deploy/role.yaml: Use ClusterRole instead of Role.
deploy/role_binding.yaml:
Use ClusterRoleBinding instead of RoleBinding.
Set the subject namespace to REPLACE_NAMESPACE. This must be changed to the
namespace in which the Operator is deployed.
Customize the Operator logic.
For this example, the nginx-operator executes the following reconciliation
logic for each Nginx Custom Resource (CR):
Create a Nginx Deployment if it does not exist.
Create a Nginx Service if it does not exist.
Create a Nginx Ingress if it is enabled and does not exist.
Ensure that the Deployment, Service, and optional Ingress match the desired configuration (e.g., replica count, image, service type) as specified by the Nginx CR.
By default, the nginx-operator watches Nginx resource events as shown in the
watches.yaml file and executes Helm releases using the specified chart:
--- - version: v1alpha1 group: example.com kind: Nginx chart: /opt/helm/helm-charts/nginx
Review the Nginx Helm chart.
When a Helm Operator project is created, the Operator SDK creates an example Helm chart that contains a set of templates for a simple Nginx release.
For this example, templates are available for Deployment, Service, and Ingress
resources, along with a NOTES.txt template, which Helm chart developers use to
convey helpful information about a release.
If you are not already familiar with Helm Charts, take a moment to review the Helm Chart developer documentation.
Understand the Nginx CR spec.
Helm uses a concept called
values
to provide customizations to a Helm chart’s defaults, which are defined in the
Helm chart’s values.yaml file.
Override these defaults by setting the desired values in the CR spec. You can use the number of replicas as an example:
First, inspect the helm-charts/nginx/values.yaml file to find that the chart
has a value called replicaCount and it is set to 1 by default. To have 2
Nginx instances in your deployment, your CR spec must contain replicaCount: 2.
Update the deploy/crds/example_v1alpha1_nginx_cr.yaml file to look like the
following:
apiVersion: example.com/v1alpha1 kind: Nginx metadata: name: example-nginx spec: replicaCount: 2
Similarly, the default service port is set to 80. To instead use 8080,
update the deploy/crds/example_v1alpha1_nginx_cr.yaml file again by adding the
service port override:
apiVersion: example.com/v1alpha1
kind: Nginx
metadata:
name: example-nginx
spec:
replicaCount: 2
service:
port: 8080
The Helm Operator applies the entire spec as if it was the contents of a values
file, just like the helm install -f ./overrides.yaml command works.
Deploy the CRD.
Before running the Operator, Kubernetes needs to know about the new custom resource definition (CRD) the operator will be watching. Deploy the following CRD:
$ kubectl create -f deploy/crds/example_v1alpha1_nginx_crd.yaml
Build and run the Operator.
There are two ways to build and run the Operator:
As a Pod inside a Kubernetes cluster.
As a Go program outside the cluster using the operator-sdk up command.
Choose one of the following methods.
Option 1: Run as a Pod inside a Kubernetes cluster. This is the preferred method for production use.
Build the nginx-operator image and push it to a registry:
$ operator-sdk build quay.io/example/nginx-operator:v0.0.1 $ docker push quay.io/example/nginx-operator:v0.0.1
Kubernetes deployment manifests are generated in the deploy/operator.yaml
file. The deployment image in this file needs to be modified from the
placeholder REPLACE_IMAGE to the previous built image. To do this, run:
$ sed -i 's|REPLACE_IMAGE|quay.io/example/nginx-operator:v0.0.1|g' deploy/operator.yaml
If you created your Operator using the --cluster-scoped=true flag, update the
service account namespace in the generated ClusterRoleBinding to match where
you are deploying your Operator:
$ export OPERATOR_NAMESPACE=$(kubectl config view --minify -o jsonpath='{.contexts[0].context.namespace}')
$ sed -i "s|REPLACE_NAMESPACE|$OPERATOR_NAMESPACE|g" deploy/role_binding.yaml
|
If you are performing these steps on OSX, use the following commands instead: $ sed -i "" 's|REPLACE_IMAGE|quay.io/example/nginx-operator:v0.0.1|g' deploy/operator.yaml $ sed -i "" "s|REPLACE_NAMESPACE|$OPERATOR_NAMESPACE|g" deploy/role_binding.yaml |
Deploy the nginx-operator:
$ kubectl create -f deploy/service_account.yaml $ kubectl create -f deploy/role.yaml $ kubectl create -f deploy/role_binding.yaml $ kubectl create -f deploy/operator.yaml
Verify that the nginx-operator is up and running:
$ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE nginx-operator 1 1 1 1 1m
Option 2: Run outside the cluster. This method is preferred during the development cycle to speed up deployment and testing.
It is important that the chart path referenced in the watches.yaml file exists
on your machine. By default, the watches.yaml file is scaffolded to work with
an Operator image built with the operator-sdk build command. When developing
and testing your operator with the operator-sdk up local command, the SDK
looks in your local file system for this path.
It is recommend to create a symlink at this location to point to your Helm chart’s path:
$ sudo mkdir -p /opt/helm/helm-charts $ sudo ln -s $PWD/helm-charts/nginx /opt/helm/helm-charts/nginx
To run the Operator locally with the default Kubernetes configuration file
present at $HOME/.kube/config:
$ operator-sdk up local INFO[0000] Go Version: go1.10.3 INFO[0000] Go OS/Arch: linux/amd64 INFO[0000] operator-sdk Version: v0.3.0+git
To run the Operator locally with a provided Kubernetes configuration file:
$ operator-sdk up local --kubeconfig=<path_to_config> INFO[0000] Go Version: go1.10.3 INFO[0000] Go OS/Arch: linux/amd64 INFO[0000] operator-sdk Version: v0.3.0+git
Deploy the Nginx custom resource.
Apply the Nginx CR that you modified earlier:
$ kubectl apply -f deploy/crds/example_v1alpha1_nginx_cr.yaml
Ensure that the nginx-operator creates the Deployment for the CR:
$ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 2 2 2 2 1m
Check the Pods to confirm two replicas were created:
$ kubectl get pods NAME READY STATUS RESTARTS AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1-f8f9c875d-fjcr9 1/1 Running 0 1m example-nginx-b9phnoz9spckcrua7ihrbkrt1-f8f9c875d-ljbzl 1/1 Running 0 1m
Check that the Service port is set to 8080:
$ kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 ClusterIP 10.96.26.3 <none> 8080/TCP 1m
Update the replicaCount and remove the port.
Change the spec.replicaCount field from 2 to 3, remove the spec.service
field, and apply the change:
$ cat deploy/crds/example_v1alpha1_nginx_cr.yaml apiVersion: "example.com/v1alpha1" kind: "Nginx" metadata: name: "example-nginx" spec: replicaCount: 3 $ kubectl apply -f deploy/crds/example_v1alpha1_nginx_cr.yaml
Confirm that the Operator changes the Deployment size:
$ kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 3 3 3 3 1m
Check that the Service port is set to the default 80:
$ kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-nginx-b9phnoz9spckcrua7ihrbkrt1 ClusterIP 10.96.26.3 <none> 80/TCP 1m
Clean up the resources:
$ kubectl delete -f deploy/crds/example_v1alpha1_nginx_cr.yaml $ kubectl delete -f deploy/operator.yaml $ kubectl delete -f deploy/role_binding.yaml $ kubectl delete -f deploy/role.yaml $ kubectl delete -f deploy/service_account.yaml $ kubectl delete -f deploy/crds/example_v1alpha1_nginx_cr.yaml