Deploying a Customized HAProxy Router - Setting up a Router | Configuring Clusters

Overview
Obtaining the Router Configuration Template
Modifying the Router Configuration Template
Using a ConfigMap to Replace the Router Configuration Template
Using Stick Tables
Rebuilding Your Router

Overview

The default HAProxy router is intended to satisfy the needs of most users. However, it does not expose all of the capability of HAProxy. Therefore, users may need to modify the router for their own needs.

You may need to implement new features within the application back-ends, or modify the current operation. The router plug-in provides all the facilities necessary to make this customization.

The router pod uses a template file to create the needed HAProxy configuration file. The template file is a golang template. When processing the template, the router has access to OKD information, including the router’s deployment configuration, the set of admitted routes, and some helper functions.

When the router pod starts, and every time it reloads, it creates an HAProxy configuration file, and then it starts HAProxy. The HAProxy configuration manual describes all of the features of HAProxy and how to construct a valid configuration file.

A configMap can be used to add the new template to the router pod. With this approach, the router deployment configuration is modified to mount the configMap as a volume in the router pod. The TEMPLATE_FILE environment variable is set to the full path name of the template file in the router pod.

It is not guaranteed that router template customizations will still work after you upgrade OKD.

Also, router template customizations must be applied to the template version of the router that is running.

Alternatively, you can build a custom router image and use it when deploying some or all of your routers. There is no need for all routers to run the same image. To do this, modify the haproxy-template.config file, and rebuild the router image. The new image is pushed to the cluster’s Docker repository, and the router’s deployment configuration image: field is updated with the new name. When the cluster is updated, the image needs to be rebuilt and pushed.

In either case, the router pod starts with the template file.

Obtaining the Router Configuration Template

The HAProxy template file is fairly large and complex. For some changes, it may be easier to modify the existing template rather than writing a complete replacement. You can obtain a haproxy-config.template file from a running router by running this on master, referencing the router pod:

# oc get po
NAME                       READY     STATUS    RESTARTS   AGE
router-2-40fc3             1/1       Running   0          11d
# oc exec router-2-40fc3 cat haproxy-config.template > haproxy-config.template
# oc exec router-2-40fc3 cat haproxy.config > haproxy.config

Alternatively, you can log onto the node that is running the router:

# docker run --rm --interactive=true --tty --entrypoint=cat \
    openshift/origin-haproxy-router haproxy-config.template

The image name is from container images.

Save this content to a file for use as the basis of your customized template. The saved haproxy.config shows what is actually running.

Modifying the Router Configuration Template

Background

The template is based on the golang template. It can reference any of the environment variables in the router’s deployment configuration, any configuration information that is described below, and router provided helper functions.

The structure of the template file mirrors the resulting HAProxy configuration file. As the template is processed, anything not surrounded by {{" something "}} is directly copied to the configuration file. Passages that are surrounded by {{" something "}} are evaluated. The resulting text, if any, is copied to the configuration file.

Go Template Actions

The define action names the file that will contain the processed template.

{{define "/var/lib/haproxy/conf/haproxy.config"}}pipeline{{end}}

Table 1. Template Router Functions
Function	Meaning
`processEndpointsForAlias(alias ServiceAliasConfig, svc ServiceUnit, action string) []Endpoint`	Returns the list of valid endpoints. When action is "shuffle", the order of endpoints is randomized.
`env(variable, default …string) string`	Tries to get the named environment variable from the pod. If it is not defined or empty, it returns the optional second argument. Otherwise, it returns an empty string.
`matchPattern(pattern, s string) bool`	The first argument is a string that contains the regular expression, the second argument is the variable to test. Returns a Boolean value indicating whether the regular expression provided as the first argument matches the string provided as the second argument.
`isInteger(s string) bool`	Determines if a given variable is an integer.
`firstMatch(s string, allowedValues …string) bool`	Compares a given string to a list of allowed strings. Returns first match scanning left to right through the list.
`matchValues(s string, allowedValues …string) bool`	Compares a given string to a list of allowed strings. Returns "true" if the string is an allowed value, otherwise returns false.
`generateRouteRegexp(hostname, path string, wildcard bool) string`	Generates a regular expression matching the route hosts (and paths). The first argument is the host name, the second is the path, and the third is a wildcard Boolean.
`genCertificateHostName(hostname string, wildcard bool) string`	Generates host name to use for serving/matching certificates. First argument is the host name and the second is the wildcard Boolean.
`isTrue(s string) bool`	Determines if a given variable contains "true".

These functions are provided by the HAProxy template router plug-in.

Router Provided Information

This section reviews the OKD information that the router makes available to the template. The router configuration parameters are the set of data that the HAProxy router plug-in is given. The fields are accessed by (dot) .Fieldname.

The tables below the Router Configuration Parameters expand on the definitions of the various fields. In particular, .State has the set of admitted routes.

Table 2. Router Configuration Parameters
Field	Type	Description
`WorkingDir`	string	The directory that files will be written to, defaults to */var/lib/containers/router*
`State`	`map[string](ServiceAliasConfig)`	The routes.
`ServiceUnits`	`map[string]ServiceUnit`	The service lookup.
`DefaultCertificate`	string	Full path name to the default certificate in pem format.
`PeerEndpoints`	`[]Endpoint`	Peers.
`StatsUser`	string	User name to expose stats with (if the template supports it).
`StatsPassword`	string	Password to expose stats with (if the template supports it).
`StatsPort`	int	Port to expose stats with (if the template supports it).
`BindPorts`	bool	Whether the router should bind the default ports.

Table 3. Router ServiceAliasConfig (A Route)
Field	Type	Description
`Name`	string	The user-specified name of the route.
`Namespace`	string	The namespace of the route.
`Host`	string	The host name. For example, `www.example.com`.
`Path`	string	Optional path. For example, `www.example.com/myservice` where `myservice` is the path.
`TLSTermination`	`routeapi.TLSTerminationType`	The termination policy for this back-end; drives the mapping files and router configuration.
`Certificates`	`map[string]Certificate`	Certificates used for securing this back-end. Keyed by the certificate ID.
`Status`	`ServiceAliasConfigStatus`	Indicates the status of configuration that needs to be persisted.
`PreferPort`	string	Indicates the port the user wants to expose. If empty, a port will be selected for the service.
`InsecureEdgeTerminationPolicy`	`routeapi.InsecureEdgeTerminationPolicyType`	Indicates desired behavior for insecure connections to an edge-terminated route: `none` (or `disable`), `allow`, or `redirect`.
`RoutingKeyName`	string	Hash of the route + namespace name used to obscure the cookie ID.
`IsWildcard`	bool	Indicates this service unit needing wildcard support.
`Annotations`	`map[string]string`	Annotations attached to this route.
`ServiceUnitNames`	`map[string]int32`	Collection of services that support this route, keyed by service name and valued on the weight attached to it with respect to other entries in the map.
`ActiveServiceUnits`	int	Count of the `ServiceUnitNames` with a non-zero weight.

The ServiceAliasConfig is a route for a service. Uniquely identified by host + path. The default template iterates over routes using {{range $cfgIdx, $cfg := .State }}. Within such a {{range}} block, the template can refer to any field of the current ServiceAliasConfig using $cfg.Field.

Table 4. Router ServiceUnit
Field	Type	Description
`Name`	string	Name corresponds to a service name + namespace. Uniquely identifies the `ServiceUnit`.
`EndpointTable`	`[]Endpoint`	Endpoints that back the service. This translates into a final back-end implementation for routers.

ServiceUnit is an encapsulation of a service, the endpoints that back that service, and the routes that point to the service. This is the data that drives the creation of the router configuration files

Table 5. Router Endpoint
Field	Type
`ID`	string
`IP`	string
`Port`	string
`TargetName`	string
`PortName`	string
`IdHash`	string
`NoHealthCheck`	bool

Endpoint is an internal representation of a Kubernetes endpoint.

Table 6. Router Certificate, ServiceAliasConfigStatus
Field	Type	Description
`Certificate`	string	Represents a public/private key pair. It is identified by an ID, which will become the file name. A CA certificate will not have a `PrivateKey` set.
`ServiceAliasConfigStatus`	string	Indicates that the necessary files for this configuration have been persisted to disk. Valid values: "saved", "".

Table 7. Router Certificate Type
Field	Type	Description
ID	string
Contents	string	The certificate.
PrivateKey	string	The private key.

Table 8. Router TLSTerminationType
Field	Type	Description
`TLSTerminationType`	string	Dictates where the secure communication will stop.
`InsecureEdgeTerminationPolicyType`	string	Indicates the desired behavior for insecure connections to a route. While each router may make its own decisions on which ports to expose, this is normally port 80.

TLSTerminationType and InsecureEdgeTerminationPolicyType dictate where the secure communication will stop.

Table 9. Router TLSTerminationType Values
Constant	Value	Meaning
`TLSTerminationEdge`	`edge`	Terminate encryption at the edge router.
`TLSTerminationPassthrough`	`passthrough`	Terminate encryption at the destination, the destination is responsible for decrypting traffic.
`TLSTerminationReencrypt`	`reencrypt`	Terminate encryption at the edge router and re-encrypt it with a new certificate supplied by the destination.

Table 10. Router InsecureEdgeTerminationPolicyType Values
Type	Meaning
`Allow`	Traffic is sent to the server on the insecure port (default).
`Disable`	No traffic is allowed on the insecure port.
`Redirect`	Clients are redirected to the secure port.

None ("") is the same as Disable.

Annotations

Each route can have annotations attached. Each annotation is just a name and a value.

apiVersion: v1
kind: Route
metadata:
  annotations:
    haproxy.router.openshift.io/timeout: 5500ms
[...]

The name can be anything that does not conflict with existing Annotations. The value is any string. The string can have multiple tokens separated by a space. For example, aa bb cc. The template uses {{index}} to extract the value of an annotation. For example:

{{$balanceAlgo := index $cfg.Annotations "haproxy.router.openshift.io/balance"}}

This is an example of how this could be used for mutual client authorization.

{{ with $cnList := index $cfg.Annotations "whiteListCertCommonName" }}
  {{   if ne $cnList "" }}
    acl test ssl_c_s_dn(CN) -m str {{ $cnList }}
    http-request deny if !test
  {{   end }}
{{ end }}

Then, you can handle the white-listed CNs with this command.

$ oc annotate route <route-name> --overwrite whiteListCertCommonName="CN1 CN2 CN3"

See Route-specific Annotations for more information.

Environment Variables

The template can use any environment variables that exist in the router pod. The environment variables can be set in the deployment configuration. New environment variables can be added.

They are referenced by the env function:

{{env "ROUTER_MAX_CONNECTIONS" "20000"}}

The first string is the variable, and the second string is the default when the variable is missing or nil. When ROUTER_MAX_CONNECTIONS is not set or is nil, 20000 is used. Environment variables are a map where the key is the environment variable name and the content is the value of the variable.

See Route-specific Environment variables for more information.

Example Usage

Here is a simple template based on the HAProxy template file.

Start with a comment:

{{/*
  Here is a small example of how to work with templates
  taken from the HAProxy template file.
*/}}

The template can create any number of output files. Use a define construct to create an output file. The file name is specified as an argument to define, and everything inside the define block up to the matching end is written as the contents of that file.

{{ define "/var/lib/haproxy/conf/haproxy.config" }}
global
{{ end }}

The above will copy global to the /var/lib/haproxy/conf/haproxy.config file, and then close the file.

Set up logging based on environment variables.

{{ with (env "ROUTER_SYSLOG_ADDRESS" "") }}
  log {{.}} {{env "ROUTER_LOG_FACILITY" "local1"}} {{env "ROUTER_LOG_LEVEL" "warning"}}
{{ end }}

The env function extracts the value for the environment variable. If the environment variable is not defined or nil, the second argument is returned.

The with construct sets the value of "." (dot) within the with block to whatever value is provided as an argument to with. The with action tests Dot for nil. If not nil, the clause is processed up to the end. In the above, assume ROUTER_SYSLOG_ADDRESS contains /var/log/msg, ROUTER_LOG_FACILITY is not defined, and ROUTER_LOG_LEVEL contains info. The following will be copied to the output file:

  log /var/log/msg local1 info

Each admitted route ends up generating lines in the configuration file. Use range to go through the admitted routes:

{{ range $cfgIdx, $cfg := .State }}
  backend be_http_{{$cfgIdx}}
{{end}}

.State is a map of ServiceAliasConfig, where the key is the route name. range steps through the map and, for each pass, it sets $cfgIdx with the key, and sets $cfg to point to the ServiceAliasConfig that describes the route. If there are two routes named myroute and hisroute, the above will copy the following to the output file:

  backend be_http_myroute
  backend be_http_hisroute

Route Annotations, $cfg.Annotations, is also a map with the annotation name as the key and the content string as the value. The route can have as many annotations as desired and the use is defined by the template author. The user codes the annotation into the route and the template author customized the HAProxy template to handle the annotation.

The common usage is to index the annotation to get the value.

{{$balanceAlgo := index $cfg.Annotations "haproxy.router.openshift.io/balance"}}

The index extracts the value for the given annotation, if any. Therefore, $balanceAlgo will contain the string associated with the annotation or nil. As above, you can test for a non-nil string and act on it with the with construct.

{{ with $balanceAlgo }}
  balance $balanceAlgo
{{ end }}

Here when $balanceAlgo is not nil, balance $balanceAlgo is copied to the output file.

In a second example, you want to set a server timeout based on a timeout value set in an annotation.

$value := index $cfg.Annotations "haproxy.router.openshift.io/timeout"

The $value can now be evaluated to make sure it contains a properly constructed string. The matchPattern function accepts a regular expression and returns true if the argument satisfies the expression.

matchPattern "[1-9][0-9]*(us\|ms\|s\|m\|h\|d)?" $value

This would accept 5000ms but not 7y. The results can be used in a test.

{{if (matchPattern "[1-9][0-9]*(us\|ms\|s\|m\|h\|d)?" $value) }}
  timeout server  {{$value}}
{{ end }}

It can also be used to match tokens:

matchPattern "roundrobin|leastconn|source" $balanceAlgo

Alternatively matchValues can be used to match tokens:

matchValues $balanceAlgo "roundrobin" "leastconn" "source"

Using a ConfigMap to Replace the Router Configuration Template

You can use a ConfigMap to customize the router instance without rebuilding the router image. The haproxy-config.template, reload-haproxy, and other scripts can be modified as well as creating and modifying router environment variables.

Copy the haproxy-config.template that you want to modify as described above. Modify it as desired.
Create a ConfigMap:
```
$ oc create configmap customrouter --from-file=haproxy-config.template
```
The customrouter ConfigMap now contains a copy of the modified haproxy-config.template file.

Modify the router deployment configuration to mount the ConfigMap as a file and point the TEMPLATE_FILE environment variable to it. This can be done via oc set env and oc set volume commands, or alternatively by editing the router deployment configuration.

Using oc commands

$ oc set volume dc/router --add --overwrite \
    --name=config-volume \
    --mount-path=/var/lib/haproxy/conf/custom \
    --source='{"configMap": { "name": "customrouter"}}'
$ oc set env dc/router \
    TEMPLATE_FILE=/var/lib/haproxy/conf/custom/haproxy-config.template

Editing the Router Deployment Configuration

Use oc edit dc router to edit the router deployment configuration with a text editor.

...
        - name: STATS_USERNAME
          value: admin
        - name: TEMPLATE_FILE  (1)
          value: /var/lib/haproxy/conf/custom/haproxy-config.template
        image: openshift/origin-haproxy-routerp
...
        terminationMessagePath: /dev/termination-log
        volumeMounts: (2)
        - mountPath: /var/lib/haproxy/conf/custom
          name: config-volume
      dnsPolicy: ClusterFirst
...
      terminationGracePeriodSeconds: 30
      volumes: (3)
      - configMap:
          name: customrouter
        name: config-volume
...

1	In the `spec.container.env` field, add the `TEMPLATE_FILE` environment variable to point to the mounted *haproxy-config.template* file.
2	Add the `spec.container.volumeMounts` field to create the mount point.
3	Add a new `spec.volumes` field to mention the ConfigMap.

Save the changes and exit the editor. This restarts the router.

Using Stick Tables

The following example customization can be used in a highly-available routing setup to use stick-tables that synchronize between peers.

Adding a Peer Section

In order to synchronize stick-tables amongst peers you must a define a peers section in your HAProxy configuration. This section determines how HAProxy will identify and connect to peers. The plug-in provides data to the template under the .PeerEndpoints variable to allow you to easily identify members of the router service. You may add a peer section to the haproxy-config.template file inside the router image by adding:

{{ if (len .PeerEndpoints) gt 0 }}
peers openshift_peers
  {{ range $endpointID, $endpoint := .PeerEndpoints }}
    peer {{$endpoint.TargetName}} {{$endpoint.IP}}:1937
  {{ end }}
{{ end }}

Changing the Reload Script

When using stick-tables, you have the option of telling HAProxy what it should consider the name of the local host in the peer section. When creating endpoints, the plug-in attempts to set the TargetName to the value of the endpoint’s TargetRef.Name. If TargetRef is not set, it will set the TargetName to the IP address. The TargetRef.Name corresponds with the Kubernetes host name, therefore you can add the -L option to the reload-haproxy script to identify the local host in the peer section.

peer_name=$HOSTNAME (1)

if [ -n "$old_pid" ]; then
  /usr/sbin/haproxy -f $config_file -p $pid_file -L $peer_name -sf $old_pid
else
  /usr/sbin/haproxy -f $config_file -p $pid_file -L $peer_name
fi

1	Must match an endpoint target name that is used in the peer section.

Modifying Back Ends

Finally, to use the stick-tables within back ends, you can modify the HAProxy configuration to use the stick-tables and peer set. The following is an example of changing the existing back end for TCP connections to use stick-tables:

            {{ if eq $cfg.TLSTermination "passthrough" }}
backend be_tcp_{{$cfgIdx}}
  balance leastconn
  timeout check 5000ms
  stick-table type ip size 1m expire 5m{{ if (len $.PeerEndpoints) gt 0 }} peers openshift_peers {{ end }}
  stick on src
                {{ range $endpointID, $endpoint := $serviceUnit.EndpointTable }}
  server {{$endpointID}} {{$endpoint.IP}}:{{$endpoint.Port}} check inter 5000ms
                {{ end }}
            {{ end }}

After this modification, you can rebuild your router.

Rebuilding Your Router

In order to rebuild the router, you need copies of several files that are present on a running router. Make a work directory and copy the files from the router:

# mkdir - myrouter/conf
# cd myrouter
# oc get po
NAME                       READY     STATUS    RESTARTS   AGE
router-2-40fc3             1/1       Running   0          11d
# oc exec router-2-40fc3 cat haproxy-config.template > conf/haproxy-config.template
# oc exec router-2-40fc3 cat error-page-503.http > conf/error-page-503.http
# oc exec router-2-40fc3 cat default_pub_keys.pem > conf/default_pub_keys.pem
# oc exec router-2-40fc3 cat ../Dockerfile > Dockerfile
# oc exec router-2-40fc3 cat ../reload-haproxy > reload-haproxy

You can edit or replace any of these files. However, conf/haproxy-config.template and reload-haproxy are the most likely to be modified.

After updating the files:

# docker build -t openshift/origin-haproxy-router-myversion .
# docker tag openshift/origin-haproxy-router-myversion 172.30.243.98:5000/openshift/haproxy-router-myversion (1)
# docker push 172.30.243.98:5000/openshift/origin-haproxy-router-pc:latest (2)

1	Tag the version with the repository. In this case the repository is `172.30.243.98:5000`.
2	Push the tagged version to the repository. It may be necessary to `docker login` to the repository first.

To use the new router, edit the router deployment configuration either by changing the image: string or by adding the --images=<repo>/<image>:<tag> flag to the oc adm router command.

When debugging the changes, it is helpful to set imagePullPolicy: Always in the deployment configuration to force an image pull on each pod creation. When debugging is complete, you can change it back to imagePullPolicy: IfNotPresent to avoid the pull on each pod start.