×

Overview

An OKD route exposes a service at a host name, like www.example.com, so that external clients can reach it by name.

DNS resolution for a host name is handled separately from routing. Your administrator may have configured a DNS wildcard entry that will resolve to the OKD node that is running the OKD router. If you are using a different host name you may need to modify its DNS records independently to resolve to the node that is running the router.

Each route consists of a name (limited to 63 characters), a service selector, and an optional security configuration.

Routers

An OKD administrator can deploy routers to nodes in an OKD cluster, which enable routes created by developers to be used by external clients. The routing layer in OKD is pluggable, and two available router plug-ins are provided and supported by default.

See the Installation and Configuration guide for information on deploying a router.

A router uses the service selector to find the service and the endpoints backing the service. When both router and service provide load balancing, OKD uses the router load balancing. A router detects relevant changes in the IP addresses of its services and adapts its configuration accordingly. This is useful for custom routers to communicate modifications of API objects to an external routing solution.

The path of a request starts with the DNS resolution of a host name to one or more routers. The suggested method is to define a cloud domain with a wildcard DNS entry pointing to one or more virtual IP (VIP) addresses backed by multiple router instances. Routes using names and addresses outside the cloud domain require configuration of individual DNS entries.

When there are fewer VIP addresses than routers, the routers corresponding to the number of addresses are active and the rest are passive. A passive router is also known as a hot-standby router. For example, with two VIP addresses and three routers, you have an "active-active-passive" configuration. See High Availability for more information on router VIP configuration.

Routes can be sharded among the set of routers. Administrators can set up sharding on a cluster-wide basis and users can set up sharding for the namespace in their project. Sharding allows the operator to define multiple router groups. Each router in the group serves only a subset of traffic.

OKD routers provide external host name mapping and load balancing of service end points over protocols that pass distinguishing information directly to the router; the host name must be present in the protocol in order for the router to determine where to send it.

Router plug-ins assume they can bind to host ports 80 (HTTP) and 443 (HTTPS), by default. This means that routers must be placed on nodes where those ports are not otherwise in use. Alternatively, a router can be configured to listen on other ports by setting the ROUTER_SERVICE_HTTP_PORT and ROUTER_SERVICE_HTTPS_PORT environment variables.

Because a router binds to ports on the host node, only one router listening on those ports can be on each node if the router uses host networking (the default). Cluster networking is configured such that all routers can access all pods in the cluster.

Routers support the following protocols:

  • HTTP

  • HTTPS (with SNI)

  • WebSockets

  • TLS with SNI

WebSocket traffic uses the same route conventions and supports the same TLS termination types as other traffic.

For a secure connection to be established, a cipher common to the client and server must be negotiated. As time goes on, new, more secure ciphers become available and are integrated into client software. As older clients become obsolete, the older, less secure ciphers can be dropped. By default, the router supports a broad range of commonly available clients. The router can be configured to use a selected set of ciphers that support desired clients and do not include the less secure ciphers.

Template Routers

A template router is a type of router that provides certain infrastructure information to the underlying router implementation, such as:

  • A wrapper that watches endpoints and routes.

  • Endpoint and route data, which is saved into a consumable form.

  • Passing the internal state to a configurable template and executing the template.

  • Calling a reload script.

Available Router Plug-ins

The following router plug-ins are provided and supported in OKD.

Instructions on deploying these routers are available in Deploying a Router.

HAProxy Template Router

The HAProxy template router implementation is the reference implementation for a template router plug-in. It uses the openshift/origin-haproxy-router repository to run an HAProxy instance alongside the template router plug-in.

The following diagram illustrates how data flows from the master through the plug-in and finally into an HAProxy configuration:

HAProxy Router Data Flow
Figure 1. HAProxy Router Data Flow

Sticky Sessions

Implementing sticky sessions is up to the underlying router configuration. The template router plug-in provides the service name and namespace to the underlying implementation. This can be used for more advanced configuration, such as implementing stick-tables that synchronize between a set of peers.

Sticky sessions ensure that all traffic from a user’s session go to the same pod, creating a better user experience. While satisfying the user’s requests, the pod caches data, which can be used in subsequent requests. For example, for a cluster with five back-end pods and two load-balanced routers, you can ensure that the same pod receives the web traffic from the same web browser regardless of the router that handles it.

While returning routing traffic to the same pod is desired, it cannot be guaranteed. However, you can use HTTP headers to set a cookie to determine the pod used in the last connection. When the user sends another request to the application the browser re-sends the cookie and the router knows where to send the traffic.

Cluster administrators can turn off stickiness for passthrough routes separately from other connections, or turn off stickiness entirely.

By default, sticky sessions for passthrough routes are implemented using the source load-balancing strategy. However, the roundrobin load-balancing strategy is the default when there are active services with weights greater than 1. You can change the default for all passthrough routes by using the ROUTER_LOAD_BALANCE_ALGORITHM environment variable, and for individual routes by using the haproxy.router.openshift.io/balance route specific annotation.

Other types of routes use the leastconn load-balancing strategy by default, which can be changed by using the ROUTER_LOAD_BALANCE_ALGORITHM environment variable. It can be changed for individual routes by using the haproxy.router.openshift.io/balance route specific annotation.

Cookies cannot be set on passthrough routes, because the HTTP traffic cannot be seen. Instead, a number is calculated based on the source IP address, which determines the back-end.

If back-ends change, the traffic could head to the wrong server, making it less sticky, and if you are using a load-balancer (which hides the source IP) the same number is set for all connections and traffic is sent to the same pod.

In addition, the template router plug-in provides the service name and namespace to the underlying implementation. This can be used for more advanced configuration such as implementing stick-tables that synchronize between a set of peers.

Specific configuration for this router implementation is stored in the haproxy-config.template file located in the /var/lib/haproxy/conf directory of the router container.

The file may be customized

The source load-balancing strategy does not distinguish between external client IP addresses; because of the NAT configuration, the originating IP address (HAProxy remote) is the same. Unless the HAProxy router is running with hostNetwork: true, all external clients will be routed to a single pod.

Router Environment Variables

For all the items outlined in this section, you can set environment variables in the deployment config for the router to alter its configuration, or use the oc set env command:

$ oc set env <object_type>/<object_name> KEY1=VALUE1 KEY2=VALUE2

For example:

$ oc set env dc/router ROUTER_SYSLOG_ADDRESS=127.0.0.1 ROUTER_LOG_LEVEL=debug
Table 1. Router Environment Variables
Variable Default Description

DEFAULT_CERTIFICATE

The contents of a default certificate to use for routes that don’t expose a TLS server cert; in PEM format.

DEFAULT_CERTIFICATE_DIR

A path to a directory that contains a file named tls.crt. If tls.crt is not a PEM file which also contains a private key, it is first combined with a file named tls.key in the same directory. The PEM-format contents are then used as the default certificate. Only used if DEFAULT_CERTIFICATE or DEFAULT_CERTIFICATE_PATH are not specified.

DEFAULT_CERTIFICATE_PATH

A path to default certificate to use for routes that don’t expose a TLS server cert; in PEM format. Only used if DEFAULT_CERTIFICATE is not specified.

DROP_SYN_DURING_RESTART

false

When set to true, enables an iptables rule to drop certain packets during a restart to provide a seamless restart. See the install guide for details.

EXTENDED_VALIDATION

true

If true, the router confirms that the certificate is structurally correct. It does not verify the certificate against any CA. Set false to turn off the tests.

NAMESPACE_LABELS

A label selector to apply to namespaces to watch, empty means all.

PROJECT_LABELS

A label selector to apply to projects to watch, emtpy means all.

RELOAD_SCRIPT

The path to the reload script to use to reload the router.

ROUTER_ALLOWED_DOMAINS

A comma-separated list of domains that the host name in a route can only be part of. Any subdomain in the domain can be used. Option ROUTER_DENIED_DOMAINS overrides any values given in this option. If set, everything outside of the allowed domains will be rejected.

ROUTER_BACKEND_CHECK_INTERVAL

5000ms

Length of time between subsequent "liveness" checks on backends. (TimeUnits)

ROUTER_BACKEND_PROCESS_ENDPOINTS

String to specify how the endpoints should be processed while using the template function processEndpointsForAlias. Valid values are ["shuffle", ""]. "shuffle" will randomize the elements upon every call. Default behavior returns in pre-determined order.

ROUTER_BIND_PORTS_AFTER_SYNC

false

If set to true or TRUE, then the router does not bind to any ports until it has completely synchronized state. If not set to 'true' or 'TRUE', the router will bind to ports and start processing requests immediately, but there may be routes that are not loaded.

ROUTER_CLIENT_FIN_TIMEOUT

1s

Controls the TCP FIN timeout period for the client connecting to the route. If the FIN sent to close the connection is not answered within the given time, HAProxy will close the connection anyway. This is harmless if set to a low value and uses fewer resources on the router. (TimeUnits)

ROUTER_COOKIE_NAME

Specifies cookie name to override the internally generated default name. The name must consist of any combination of upper and lower case letters, digits, "_", and "-". The default is the hashed internal key name for the route.

ROUTER_COMPRESSION_MIME

"text/html text/plain text/css"

A space separated list of mime types to compress.

ROUTER_DEFAULT_CLIENT_TIMEOUT

30s

Length of time within which a client has to acknowledge or send data. (TimeUnits)

ROUTER_DEFAULT_CONNECT_TIMEOUT

5s

The maximum connect time. (TimeUnits)

ROUTER_DEFAULT_SERVER_FIN_TIMEOUT

1s

Controls the TCP FIN timeout from the router to the pod backing the route. (TimeUnits)

ROUTER_DEFAULT_SERVER_TIMEOUT

30s

Length of time within which a server has to acknowledge or send data. (TimeUnits)

ROUTER_DEFAULT_TUNNEL_TIMEOUT

1h

Length of time till which TCP or WebSocket connections will remain open. If you have websockets/tcp connections (and any time HAProxy is reloaded), the old HAProxy processes will "linger" around for that period. (TimeUnits)

ROUTER_DENIED_DOMAINS

A comma-separated list of domains that the host name in a route can not be part of. No subdomain in the domain can be used either. Overrides option ROUTER_ALLOWED_DOMAINS.

ROUTER_ENABLE_COMPRESSION

If true or TRUE, compress responses when possible.

ROUTER_LISTEN_ADDR

0.0.0.0:1936

Sets the listening address for router metrics.

ROUTER_LOG_LEVEL

warning

The log level to send to the syslog server.

ROUTER_MAX_CONNECTIONS

20000

Maximum number of concurrent connections.

ROUTER_METRICS_HAPROXY_SERVER_THRESHOLD

500

ROUTER_METRICS_HAPROXY_EXPORTED

Metrics collected in CSV format. For example, defaultSelectedMetrics = []int{2, 4, 5, 7, 8, 9, 13, 14, 17, 21, 24, 33, 35, 40, 43, 60}

ROUTER_METRICS_HAPROXY_BASE_SCRAPE_INTERVAL

5s

ROUTER_METRICS_HAPROXY_TIMEOUT

5s

ROUTER_METRICS_TYPE

haproxy

Generate metrics for the HAProxy router. (haproxy is the only supported value)

ROUTER_OVERRIDE_HOSTNAME

If set true, override the spec.host value for a route with the template in ROUTER_SUBDOMAIN.

ROUTER_SERVICE_HTTPS_PORT

443

Port to listen for HTTPS requests.

ROUTER_SERVICE_HTTP_PORT

80

Port to listen for HTTP requests.

ROUTER_SERVICE_NAME

public

The name that the router identifies itself in the in route status.

ROUTER_CANONICAL_HOSTNAME

The (optional) host name of the router shown in the in route status.

ROUTER_SERVICE_NAMESPACE

The namespace the router identifies itself in the in route status. Required if ROUTER_SERVICE_NAME is used.

ROUTER_SERVICE_NO_SNI_PORT

10443

Internal port for some front-end to back-end communication (see note below).

ROUTER_SERVICE_SNI_PORT

10444

Internal port for some front-end to back-end communication (see note below).

ROUTER_SLOWLORIS_HTTP_KEEPALIVE

300s

Set the maximum time to wait for a new HTTP request to appear. If this is set too low, it can confuse browsers and applications not expecting a small keepalive value. (TimeUnits)

ROUTER_SLOWLORIS_TIMEOUT

10s

Length of time the transmission of an HTTP request can take. (TimeUnits)

ROUTER_SUBDOMAIN

The template that should be used to generate the host name for a route without spec.host (e.g. ${name}-${namespace}.myapps.mycompany.com).

ROUTER_SYSLOG_ADDRESS

Address to send log messages. Disabled if empty.

ROUTER_SYSLOG_FORMAT

If set, override the default log format used by underlying router implementation. Its value should conform with underlying router implementation’s specification.

ROUTER_TCP_BALANCE_SCHEME

source

Load-balancing strategy for multiple endpoints for pass-through routes. Available options are source, roundrobin, or leastconn.

ROUTER_LOAD_BALANCE_ALGORITHM

leastconn

Load-balancing strategy for routes with multiple endpoints. Available options are source, roundrobin, and leastconn.

ROUTE_LABELS

A label selector to apply to the routes to watch, empty means all.

STATS_PASSWORD

The password needed to access router stats (if the router implementation supports it).

STATS_PORT

Port to expose statistics on (if the router implementation supports it). If not set, stats are not exposed.

STATS_USERNAME

The user name needed to access router stats (if the router implementation supports it).

TEMPLATE_FILE

/var/lib/haproxy/conf/custom/ haproxy-config-custom.template

The path to the HAProxy template file (in the container image).

RELOAD_INTERVAL

5s

The minimum frequency the router is allowed to reload to accept new changes. (TimeUnits)

ROUTER_USE_PROXY_PROTOCOL

When set to true or TRUE, HAProxy expects incoming connections to use the PROXY protocol on port 80 or port 443. The source IP address can pass through a load balancer if the load balancer supports the protocol, for example Amazon ELB.

ROUTER_ALLOW_WILDCARD_ROUTES

When set to true or TRUE, any routes with a wildcard policy of Subdomain that pass the router admission checks will be serviced by the HAProxy router.

ROUTER_DISABLE_NAMESPACE_OWNERSHIP_CHECK

Set to true to relax the namespace ownership policy.

ROUTER_STRICT_SNI

strict-sni

ROUTER_CIPHERS

intermediate

Specify the set of ciphers supported by bind.

If you want to run multiple routers on the same machine, you must change the ports that the router is listening on, ROUTER_SERVICE_SNI_PORT and ROUTER_SERVICE_NO_SNI_PORT. These ports can be anything you want as long as they are unique on the machine. These ports will not be exposed externally.

Timeouts

TimeUnits are represented by a number followed by the unit: us *(microseconds), ms (milliseconds, default), s (seconds), m (minutes), h *(hours), d (days).

The regular expression is: [1-9][0-9]*(us\|ms\|s\|m\|h\|d)

Load-balancing Strategy

When a route has multiple endpoints, HAProxy distributes requests to the route among the endpoints based on the selected load-balancing strategy. This applies when no persistence information is available, such as on the first request in a session.

The strategy can be one of the following:

  • roundrobin: Each endpoint is used in turn, according to its weight. This is the smoothest and fairest algorithm when the server’s processing time remains equally distributed.

  • leastconn: The endpoint with the lowest number of connections receives the request. Round-robin is performed when multiple endpoints have the same lowest number of connections. Use this algorithm when very long sessions are expected, such as LDAP, SQL, TSE, or others. Not intended to be used with protocols that typically use short sessions such as HTTP.

  • source: The source IP address is hashed and divided by the total weight of the running servers to designate which server will receive the request. This ensures that the same client IP address will always reach the same server as long as no server goes down or up. If the hash result changes due to the number of running servers changing, many clients will be directed to different servers. This algorithm is generally used with passthrough routes.

The ROUTER_TCP_BALANCE_SCHEME environment variable sets the default strategy for passthorugh routes. The ROUTER_LOAD_BALANCE_ALGORITHM environment variable sets the default strategy for the router for the remaining routes. A route specific annotation, haproxy.router.openshift.io/balance, can be used to control specific routes.

HAProxy Strict SNI

By default, when a host does not resolve to a route in a HTTPS or TLS SNI request, the default certificate is returned to the caller as part of the 503 response. This exposes the default certificate and can pose security concerns because the wrong certificate is served for a site. The HAProxy strict-sni option to bind suppresses use of the default certificate.

The ROUTER_STRICT_SNI environment variable controls bind processing. When set to true or TRUE, strict-sni is added to the HAProxy bind. The default setting is false.

The option can be set when the router is created or added later.

$ oc adm router --strict-sni

This sets ROUTER_STRICT_SNI=true.

Router Cipher Suite

Each client (for example, Chrome 30, or Java8) includes a suite of ciphers used to securely connect with the router. The router must have at least one of the ciphers for the connection to be complete:

Table 2. Router Cipher Profiles
Profile Oldest compatible client

modern

Firefox 27, Chrome 30, IE 11 on Windows 7, Edge, Opera 17, Safari 9, Android 5.0, Java 8

intermediate

Firefox 1, Chrome 1, IE 7, Opera 5, Safari 1, Windows XP IE8, Android 2.3, Java 7

old

Windows XP IE6, Java 6

See the Security/Server Side TLS reference guide for more information.

The router defaults to the intermediate profile. You can select a different profile using the --ciphers option when creating a route, or by changing the ROUTER_CIPHERS environment variable with the values modern, intermediate, or old for an existing router. Alternatively, a set of ":" separated ciphers can be provided. The ciphers must be from the set displayed by:

openssl ciphers

Route Host Names

In order for services to be exposed externally, an OKD route allows you to associate a service with an externally-reachable host name. This edge host name is then used to route traffic to the service.

When multiple routes from different namespaces claim the same host, the oldest route wins and claims it for the namespace. If additional routes with different path fields are defined in the same namespace, those paths are added. If multiple routes with the same path are used, the oldest takes priority.

A consequence of this behavior is that if you have two routes for a host name: an older one and a newer one. If someone else has a route for the same host name that they created between when you created the other two routes, then if you delete your older route, your claim to the host name will no longer be in effect. The other namespace now claims the host name and your claim is lost.

Example 1. A Route with a Specified Host:
apiVersion: v1
kind: Route
metadata:
  name: host-route
spec:
  host: www.example.com  (1)
  to:
    kind: Service
    name: service-name
1 Specifies the externally-reachable host name used to expose a service.
Example 2. A Route Without a Host:
apiVersion: v1
kind: Route
metadata:
  name: no-route-hostname
spec:
  to:
    kind: Service
    name: service-name

If a host name is not provided as part of the route definition, then OKD automatically generates one for you. The generated host name is of the form:

<route-name>[-<namespace>].<suffix>

The following example shows the OKD-generated host name for the above configuration of a route without a host added to a namespace mynamespace:

Example 3. Generated Host Name
no-route-hostname-mynamespace.router.default.svc.cluster.local (1)
1 The generated host name suffix is the default routing subdomain router.default.svc.cluster.local.

A cluster administrator can also customize the suffix used as the default routing subdomain for their environment.

Route Types

Routes can be either secured or unsecured. Secure routes provide the ability to use several types of TLS termination to serve certificates to the client. Routers support edge, passthrough, and re-encryption termination.

Example 4. Unsecured Route Object YAML Definition
apiVersion: v1
kind: Route
metadata:
  name: route-unsecured
spec:
  host: www.example.com
  to:
    kind: Service
    name: service-name

Unsecured routes are simplest to configure, as they require no key or certificates, but secured routes offer security for connections to remain private.

A secured route is one that specifies the TLS termination of the route. The available types of termination are described below.

Path Based Routes

Path based routes specify a path component that can be compared against a URL (which requires that the traffic for the route be HTTP based) such that multiple routes can be served using the same host name, each with a different path. Routers should match routes based on the most specific path to the least; however, this depends on the router implementation. The host name and path are passed through to the backend server so it should be able to successfully answer requests for them. For example: a request to http://example.com/foo/ that goes to the router will result in a pod seeing a request to http://example.com/foo/.

The following table shows example routes and their accessibility:

Table 3. Route Availability
Route When Compared to Accessible

www.example.com/test

www.example.com/test

Yes

www.example.com

No

www.example.com/test and www.example.com

www.example.com/test

Yes

www.example.com

Yes

www.example.com

www.example.com/test

Yes (Matched by the host, not the route)

www.example.com

Yes

Example 5. An Unsecured Route with a Path:
apiVersion: v1
kind: Route
metadata:
  name: route-unsecured
spec:
  host: www.example.com
  path: "/test"   (1)
  to:
    kind: Service
    name: service-name
1 The path is the only added attribute for a path-based route.

Path-based routing is not available when using passthrough TLS, as the router does not terminate TLS in that case and cannot read the contents of the request.

Secured Routes

Secured routes specify the TLS termination of the route and, optionally, provide a key and certificate(s).

TLS termination in OKD relies on SNI for serving custom certificates. Any non-SNI traffic received on port 443 is handled with TLS termination and a default certificate (which may not match the requested host name, resulting in validation errors).

Secured routes can use any of the following three types of secure TLS termination.

Edge Termination

With edge termination, TLS termination occurs at the router, prior to proxying traffic to its destination. TLS certificates are served by the front end of the router, so they must be configured into the route, otherwise the router’s default certificate will be used for TLS termination.

Example 6. A Secured Route Using Edge Termination
apiVersion: v1
kind: Route
metadata:
  name: route-edge-secured (1)
spec:
  host: www.example.com
  to:
    kind: Service
    name: service-name (1)
  tls:
    termination: edge            (2)
    key: |-                      (3)
      -----BEGIN PRIVATE KEY-----
      [...]
      -----END PRIVATE KEY-----
    certificate: |-              (4)
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----
    caCertificate: |-            (5)
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----
1 The name of the object, which is limited to 63 characters.
2 The termination field is edge for edge termination.
3 The key field is the contents of the PEM format key file.
4 The certificate field is the contents of the PEM format certificate file.
5 An optional CA certificate may be required to establish a certificate chain for validation.

Because TLS is terminated at the router, connections from the router to the endpoints over the internal network are not encrypted.

Edge-terminated routes can specify an insecureEdgeTerminationPolicy that enables traffic on insecure schemes (HTTP) to be disabled, allowed or redirected. The allowed values for insecureEdgeTerminationPolicy are: None or empty (for disabled), Allow or Redirect. The default insecureEdgeTerminationPolicy is to disable traffic on the insecure scheme. A common use case is to allow content to be served via a secure scheme but serve the assets (example images, stylesheets and javascript) via the insecure scheme.

Example 7. A Secured Route Using Edge Termination Allowing HTTP Traffic
apiVersion: v1
kind: Route
metadata:
  name: route-edge-secured-allow-insecure (1)
spec:
  host: www.example.com
  to:
    kind: Service
    name: service-name (1)
  tls:
    termination:                   edge   (2)
    insecureEdgeTerminationPolicy: Allow  (3)
    [ ... ]
1 The name of the object, which is limited to 63 characters.
2 The termination field is edge for edge termination.
3 The insecure policy to allow requests sent on an insecure scheme HTTP.
Example 8. A Secured Route Using Edge Termination Redirecting HTTP Traffic to HTTPS
apiVersion: v1
kind: Route
metadata:
  name: route-edge-secured-redirect-insecure (1)
spec:
  host: www.example.com
  to:
    kind: Service
    name: service-name (1)
  tls:
    termination:                   edge      (2)
    insecureEdgeTerminationPolicy: Redirect  (3)
    [ ... ]
1 The name of the object, which is limited to 63 characters.
2 The termination field is edge for edge termination.
3 The insecure policy to redirect requests sent on an insecure scheme HTTP to a secure scheme HTTPS.

Passthrough Termination

With passthrough termination, encrypted traffic is sent straight to the destination without the router providing TLS termination. Therefore no key or certificate is required.

Example 9. A Secured Route Using Passthrough Termination
apiVersion: v1
kind: Route
metadata:
  name: route-passthrough-secured (1)
spec:
  host: www.example.com
  to:
    kind: Service
    name: service-name (1)
  tls:
    termination: passthrough     (2)
1 The name of the object, which is limited to 63 characters.
2 The termination field is set to passthrough. No other encryption fields are needed.

The destination pod is responsible for serving certificates for the traffic at the endpoint. This is currently the only method that can support requiring client certificates (also known as two-way authentication).

Passthrough routes can also have an insecureEdgeTerminationPolicy. The only valid values are None (or empty, for disabled) or Redirect.

Re-encryption Termination

Re-encryption is a variation on edge termination where the router terminates TLS with a certificate, then re-encrypts its connection to the endpoint which may have a different certificate. Therefore the full path of the connection is encrypted, even over the internal network. The router uses health checks to determine the authenticity of the host.

Example 10. A Secured Route Using Re-Encrypt Termination
apiVersion: v1
kind: Route
metadata:
  name: route-pt-secured (1)
spec:
  host: www.example.com
  to:
    kind: Service
    name: service-name (1)
  tls:
    termination: reencrypt        (2)
    key: [as in edge termination]
    certificate: [as in edge termination]
    caCertificate: [as in edge termination]
    destinationCACertificate: |-  (3)
      -----BEGIN CERTIFICATE-----
      [...]
      -----END CERTIFICATE-----
1 The name of the object, which is limited to 63 characters.
2 The termination field is set to reencrypt. Other fields are as in edge termination.
3 Required for re-encryption. destinationCACertificate specifies a CA certificate to validate the endpoint certificate, securing the connection from the router to the destination pods. If the service is using a service signing certificate, or the administrator has specified a default CA certificate for the router and the service has a certificate signed by that CA, this field can be omitted.

If the destinationCACertificate field is left empty, the router automatically leverages the certificate authority that is generated for service serving certificates, and is injected into every pod as /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt. This allows new routes that leverage end-to-end encryption without having to generate a certificate for the route. This is useful for custom routers or the F5 router, which might not allow the destinationCACertificate unless the administrator has allowed it.

Re-encrypt routes can have an insecureEdgeTerminationPolicy with all of the same values as edge-terminated routes.

Router Sharding

In OKD, each route can have any number of labels in its metadata field. A router uses selectors (also known as a selection expression) to select a subset of routes from the entire pool of routes to serve. A selection expression can also involve labels on the route’s namespace. The selected routes form a router shard. You can create and modify router shards independently from the routes, themselves.

This design supports traditional sharding as well as overlapped sharding. In traditional sharding, the selection results in no overlapping sets and a route belongs to exactly one shard. In overlapped sharding, the selection results in overlapping sets and a route can belong to many different shards. For example, a single route may belong to a SLA=high shard (but not SLA=medium or SLA=low shards), as well as a geo=west shard (but not a geo=east shard).

Another example of overlapped sharding is a set of routers that select based on namespace of the route:

Router Selection Namespaces

router-1

A* — J*

A*, B*, C*, D*, E*, F*, G*, H*, I*, J*

router-2

K* — T*

K*, L*, M*, N*, O*, P*, Q*, R*, S*, T*

router-3

Q* — Z*

Q*, R*, S*, T*, U*, V*, W*, X*, Y*, Z*

Both router-2 and router-3 serve routes that are in the namespaces Q*, R*, S*, T*. To change this example from overlapped to traditional sharding, we could change the selection of router-2 to K* — P*, which would eliminate the overlap.

When routers are sharded, a given route is bound to zero or more routers in the group. The route binding ensures uniqueness of the route across the shard. Uniqueness allows secure and non-secure versions of the same route to exist within a single shard. This implies that routes now have a visible life cycle that moves from created to bound to active.

In the sharded environment the first route to hit the shard reserves the right to exist there indefinitely, even across restarts.

During a green/blue deployment a route may be be selected in multiple routers. An OKD application administrator may wish to bleed traffic from one version of the application to another and then turn off the old version.

Sharding can be done by the administrator at a cluster level and by the user at a project/namespace level. When namespace labels are used, the service account for the router must have cluster-reader permission to permit the router to access the labels in the namespace.

For two or more routes that claim the same host name, the resolution order is based on the age of the route and the oldest route would win the claim to that host. In the case of sharded routers, routes are selected based on their labels matching the router’s selection criteria. There is no consistent way to determine when labels are added to a route. So if an older route claiming an existing host name is "re-labelled" to match the router’s selection criteria, it will replace the existing route based on the above mentioned resolution order (oldest route wins).

Alternate Backends and Weights

A route is usually associated with one service through the to: token with kind: Service. All of the requests to the route are handled by endpoints in the service based on the load balancing strategy.

It is possible to have as many as four services supporting the route. The portion of requests that are handled by each service is governed by the service weight.

The first service is entered using the to: token as before, and up to three additional services can be entered using the alternateBackend: token. Each service must be kind: Service which is the default.

Each service has a weight associated with it. The portion of requests handled by the service is weight / sum_of_all_weights. When a service has more than one endpoint, the service’s weight is distributed among the endpoints with each endpoint getting at least 1. If the service weight is 0 each of the service’s endpoints will get 0.

The weight must be in the range 0-256. The default is 1. When the weight is 0 no requests are passed to the service. If all services have weight 0, requests are returned with a 503 error. When a service has no endpoints, the weight is effectively 0.

When using alternateBackends also use the roundrobin load balancing strategy to ensure requests are distributed as expected to the services based on weight. roundrobin can be set for a route using a route annotation, or for the router in general using an environment variable.

The following is an example route configuration using alternate backends for A/B deployments.

A Route with alternateBackends and weights:
apiVersion: v1
kind: Route
metadata:
  name: route-alternate-service
  annotations:
    haproxy.router.openshift.io/balance: roundrobin  (1)
spec:
  host: www.example.com
  to:
    kind: Service
    name: service-name  (2)
    weight: 20          (4)
  alternateBackends:
  - kind: Service
    name: service-name2 (3)
    weight: 10          (4)
    kind: Service
    name: service-name3 (3)
    weight: 10          (4)
1 This route uses roundrobin load balancing strategy
2 The first service name is service-name which may have 0 or more pods
3 The alternateBackend services may also have 0 or more pods
4 The total weight is 40. service-name will get 20/40 or 1/2 of the requests, service-name2 and service-name3 will each get 1/4 of the requests, assuming each service has 1 or more endpoints.

Route-specific Annotations

Using environment variables, a router can set the default options for all the routes it exposes. An individual route can override some of these defaults by providing specific configurations in its annotations.

Route Annotations

For all the items outlined in this section, you can set annotations on the route definition for the route to alter its configuration

Table 4. Route Annotations
Variable Description Environment Variable Used as Default

haproxy.router.openshift.io/balance

Sets the load-balancing algorithm. Available options are source, roundrobin, and leastconn.

ROUTER_TCP_BALANCE_SCHEME for passthrough routes. Otherwise, use ROUTER_LOAD_BALANCE_ALGORITHM.

haproxy.router.openshift.io/disable_cookies

Disables the use of cookies to track related connections. If set to true or TRUE, the balance algorithm is used to choose which back-end serves connections for each incoming HTTP request.

haproxy.router.openshift.io/cookie_name

Specifies an optional cookie to be used for this route. The name must consist of any combination of upper and lower case letters, digits, "_", and "-". The default is the hashed internal key name for the route.

haproxy.router.openshift.io/rate-limit-connections

Setting true or TRUE to enables rate limiting functionality.

haproxy.router.openshift.io/rate-limit-connections.concurrent-tcp

Limits the number of concurrent TCP connections shared by an IP address.

haproxy.router.openshift.io/rate-limit-connections.rate-http

Limits the rate at which an IP address can make HTTP requests.

haproxy.router.openshift.io/rate-limit-connections.rate-tcp

Limits the rate at which an IP address can make TCP connections.

haproxy.router.openshift.io/timeout

Sets a server-side timeout for the route. (TimeUnits)

ROUTER_DEFAULT_SERVER_TIMEOUT

router.openshift.io/haproxy.health.check.interval

Sets the interval for the back-end health checks. (TimeUnits)

ROUTER_BACKEND_CHECK_INTERVAL

haproxy.router.openshift.io/ip_whitelist

Sets a whitelist for the route.

haproxy.router.openshift.io/hsts_header

Sets a Strict-Transport-Security header for the edge terminated or re-encrypt route.

Example 11. A Route Setting Custom Timeout
apiVersion: v1
kind: Route
metadata:
  annotations:
    haproxy.router.openshift.io/timeout: 5500ms (1)
[...]
1 Specifies the new timeout with HAProxy supported units (us, ms, s, m, h, d). If unit not provided, ms is the default.

Setting a server-side timeout value for passthrough routes too low can cause WebSocket connections to timeout frequently on that route.

Route-specific IP Whitelists

You can restrict access to a route to a select set of IP addresses by adding the haproxy.router.openshift.io/ip_whitelist annotation on the route. The whitelist is a space-separated list of IP addresses and/or CIDRs for the approved source addresses. Requests from IP addresses that are not in the whitelist are dropped.

Some examples:

When editing a route, add the following annotation to define the desired source IP’s. Alternatively, use oc annotate route <name>.

Allow only one specific IP address:

metadata:
  annotations:
    haproxy.router.openshift.io/ip_whitelist: 192.168.1.10

Allow several IP addresses:

metadata:
  annotations:
    haproxy.router.openshift.io/ip_whitelist: 192.168.1.10 192.168.1.11 192.168.1.12

Allow an IP CIDR network:

metadata:
  annotations:
    haproxy.router.openshift.io/ip_whitelist: 192.168.1.0/24

Allow mixed IP addresses and IP CIDR networks:

metadata:
  annotations:
    haproxy.router.openshift.io/ip_whitelist: 180.5.61.153 192.168.1.0/24 10.0.0.0/8

Creating Routes Specifying a Wildcard Subdomain Policy

A wildcard policy allows a user to define a route that covers all hosts within a domain (when the router is configured to allow it). A route can specify a wildcard policy as part of its configuration using the wildcardPolicy field. Any routers run with a policy allowing wildcard routes will expose the route appropriately based on the wildcard policy.

Example 12. A Route Specifying a Subdomain WildcardPolicy
apiVersion: v1
kind: Route
spec:
  host: wildcard.example.com  (1)
  wildcardPolicy: Subdomain   (2)
  to:
    kind: Service
    name: service-name
1 Specifies the externally reachable host name used to expose a service.
2 Specifies that the externally reachable host name should allow all hosts in the subdomain example.com. *.example.com is the subdomain for host name wildcard.example.com to reach the exposed service.

Route Status

The route status field is only set by routers. If changes are made to a route so that a router no longer serves a specific route, the status becomes stale. The routers do not clear the route status field. To remove the stale entries in the route status, use the clear-route-status script.

Denying or Allowing Certain Domains in Routes

A router can be configured to deny or allow a specific subset of domains from the host names in a route using the ROUTER_DENIED_DOMAINS and ROUTER_ALLOWED_DOMAINS environment variables.

ROUTER_DENIED_DOMAINS

Domains listed are not allowed in any indicated routes.

ROUTER_ALLOWED_DOMAINS

Only the domains listed are allowed in any indicated routes.

The domains in the list of denied domains take precedence over the list of allowed domains. Meaning OKD first checks the deny list (if applicable), and if the host name is not in the list of denied domains, it then checks the list of allowed domains. However, the list of allowed domains is more restrictive, and ensures that the router only admits routes with hosts that belong to that list.

For example, to deny the [*.]open.header.test, [*.]openshift.org and [*.]block.it routes for the myrouter route:

$ oc adm router myrouter ...
$ oc set env dc/myrouter ROUTER_DENIED_DOMAINS="open.header.test, openshift.org, block.it"

This means that myrouter will admit the following based on the route’s name:

$ oc expose service/<name> --hostname="foo.header.test"
$ oc expose service/<name> --hostname="www.allow.it"
$ oc expose service/<name> --hostname="www.openshift.test"

However, myrouter will deny the following:

$ oc expose service/<name> --hostname="open.header.test"
$ oc expose service/<name> --hostname="www.open.header.test"
$ oc expose service/<name> --hostname="block.it"
$ oc expose service/<name> --hostname="franco.baresi.block.it"
$ oc expose service/<name> --hostname="openshift.org"
$ oc expose service/<name> --hostname="api.openshift.org"

Alternatively, to block any routes where the host name is not set to [*.]stickshift.org or [*.]kates.net:

$ oc adm router myrouter ...
$ oc set env dc/myrouter ROUTER_ALLOWED_DOMAINS="stickshift.org, kates.net"

This means that the myrouter router will admit:

$ oc expose service/<name> --hostname="stickshift.org"
$ oc expose service/<name> --hostname="www.stickshift.org"
$ oc expose service/<name> --hostname="kates.net"
$ oc expose service/<name> --hostname="api.kates.net"
$ oc expose service/<name> --hostname="erno.r.kube.kates.net"

However, myrouter will deny the following:

$ oc expose service/<name> --hostname="www.open.header.test"
$ oc expose service/<name> --hostname="drive.ottomatic.org"
$ oc expose service/<name> --hostname="www.wayless.com"
$ oc expose service/<name> --hostname="www.deny.it"

To implement both scenarios, run:

$ oc adm router adrouter ...
$ oc env dc/adrouter ROUTER_ALLOWED_DOMAINS="openshift.org, kates.net" \
    ROUTER_DENIED_DOMAINS="ops.openshift.org, metrics.kates.net"

This will allow any routes where the host name is set to [*.]openshift.org or [*.]kates.net, and not allow any routes where the host name is set to [*.]ops.openshift.org or [*.]metrics.kates.net.

Therefore, the following will be denied:

$ oc expose service/<name> --hostname="www.open.header.test"
$ oc expose service/<name> --hostname="ops.openshift.org"
$ oc expose service/<name> --hostname="log.ops.openshift.org"
$ oc expose service/<name> --hostname="www.block.it"
$ oc expose service/<name> --hostname="metrics.kates.net"
$ oc expose service/<name> --hostname="int.metrics.kates.net"

However, the following will be allowed:

$ oc expose service/<name> --hostname="openshift.org"
$ oc expose service/<name> --hostname="api.openshift.org"
$ oc expose service/<name> --hostname="m.api.openshift.org"
$ oc expose service/<name> --hostname="kates.net"
$ oc expose service/<name> --hostname="api.kates.net"

Disabling the Namespace Ownership Check

Hosts and subdomains are owned by the namespace of the route that first makes the claim. Other routes created in the namespace can make claims on the subdomain. All other namespaces are prevented from making claims on the claimed hosts and subdomains. The namespace that owns the host also owns all paths associated with the host, for example www.abc.xyz/path1.

For example, if the host www.abc.xyz is not claimed by any route. Creating route r1 with host www.abc.xyz in namespace ns1 makes namespace ns1 the owner of host www.abc.xyz and subdomain abc.xyz for wildcard routes. If another namespace, ns2, tries to create a route with say a different path www.abc.xyz/path1/path2, it would fail because a route in another namespace (ns1 in this case) owns that host.

With wildcard routes the namespace that owns the subdomain owns all hosts in the subdomain. If a namespace owns subdomain abc.xyz as in the above example, another namespace cannot claim z.abc.xyz.

By disabling the namespace ownership rules, you can disable these restrictions and allow hosts (and subdomains) to be claimed across namespaces.

If you decide to disable the namespace ownership checks in your router, be aware that this allows end users to claim ownership of hosts across namespaces. While this change can be desirable in certain development environments, use this feature with caution in production environments, and ensure that your cluster policy has locked down untrusted end users from creating routes.

For example, with ROUTER_DISABLE_NAMESPACE_OWNERSHIP_CHECK=true, if namespace ns1 creates the oldest route r1 www.abc.xyz, it owns only the hostname (+ path). Another namespace can create a wildcard route even though it does not have the oldest route in that subdomain (abc.xyz) and we could potentially have other namespaces claiming other non-wildcard overlapping hosts (for example, foo.abc.xyz, bar.abc.xyz, baz.abc.xyz) and their claims would be granted.

Any other namespace (for example, ns2) can now create a route r2 www.abc.xyz/p1/p2, and it would be admitted. Similarly another namespace (ns3) can also create a route wildthing.abc.xyz with a subdomain wildcard policy and it can own the wildcard.

As this example demonstrates, the policy ROUTER_DISABLE_NAMESPACE_OWNERSHIP_CHECK=true is more lax and allows claims across namespaces. The only time the router would reject a route with the namespace ownership disabled is if the host+path is already claimed.

For example, if a new route rx tries to claim www.abc.xyz/p1/p2, it would be rejected as route r2 owns that host+path combination. This is true whether route rx is in the same namespace or other namespace since the exact host+path is already claimed.

This feature can be set during router creation or by setting an environment variable in the router’s deployment configuration.

$ oc adm router ... --disable-namespace-ownership-check=true
$ oc env dc/router ROUTER_DISABLE_NAMESPACE_OWNERSHIP_CHECK=true