apiVersion: config.openshift.io/v1
kind: Ingress
metadata:
name: cluster
spec:
domain: apps.openshiftdemos.com
When you create your OKD cluster, pods and services running on the cluster are each allocated their own IP addresses. The IP addresses are accessible to other pods and services running nearby but are not accessible to outside clients. The Ingress Operator implements the IngressController
API and is the component responsible for enabling external access to OKD cluster services.
The Ingress Operator makes it possible for external clients to access your service by deploying and managing one or more HAProxy-based
Ingress Controllers to handle routing. You can use the Ingress Operator to route traffic by specifying OKD Route
and Kubernetes Ingress
resources. Configurations within the Ingress Controller, such as the ability to define endpointPublishingStrategy
type and internal load balancing, provide ways to publish Ingress Controller endpoints.
The installation program generates an asset with an Ingress
resource in the config.openshift.io
API group, cluster-ingress-02-config.yml
.
Ingress
resourceapiVersion: config.openshift.io/v1
kind: Ingress
metadata:
name: cluster
spec:
domain: apps.openshiftdemos.com
The installation program stores this asset in the cluster-ingress-02-config.yml
file in the manifests/
directory. This Ingress
resource defines the cluster-wide configuration for Ingress. This Ingress configuration is used as follows:
The Ingress Operator uses the domain from the cluster Ingress configuration as the domain for the default Ingress Controller.
The OpenShift API Server Operator uses the domain from the cluster Ingress configuration. This domain is also used when generating a default host for a Route
resource that does not specify an explicit host.
The ingresscontrollers.operator.openshift.io
resource offers the following
configuration parameters.
Parameter | Description | ||||
---|---|---|---|---|---|
|
The If empty, the default value is |
||||
|
|
||||
|
On GCP, AWS, and Azure you can configure the following
If not set, the default value is based on
For most platforms, the
|
||||
|
The The secret must contain the following keys and data:
* If not set, a wildcard certificate is automatically generated and used. The certificate is valid for the Ingress Controller The in-use certificate, whether generated or user-specified, is automatically integrated with OKD built-in OAuth server. |
||||
|
|
||||
|
|
||||
|
If not set, the defaults values are used.
|
||||
|
If not set, the default value is based on the When using the The minimum TLS version for Ingress Controllers is
|
||||
|
The |
||||
|
|
||||
|
|
||||
|
By setting the By default, the policy is set to
By setting These adjustments are only applied to cleartext, edge-terminated, and re-encrypt routes, and only when using HTTP/1. For request headers, these adjustments are applied only for routes that have the |
||||
|
|
||||
|
|
||||
|
For any cookie that you want to capture, the following parameters must be in your
For example:
|
||||
|
|
||||
|
|
||||
|
The
|
||||
|
The
These connections come from load balancer health probes or web browser speculative connections (preconnect) and can be safely ignored. However, these requests can be caused by network errors, so setting this field to |
All parameters are optional. |
TLS security profiles provide a way for servers to regulate which ciphers a connecting client can use when connecting to the server.
You can use a TLS (Transport Layer Security) security profile to define which TLS ciphers are required by various OKD components. The OKD TLS security profiles are based on Mozilla recommended configurations.
You can specify one of the following TLS security profiles for each component:
Profile | Description | ||
---|---|---|---|
|
This profile is intended for use with legacy clients or libraries. The profile is based on the Old backward compatibility recommended configuration. The
|
||
|
This profile is the recommended configuration for the majority of clients. It is the default TLS security profile for the Ingress Controller, kubelet, and control plane. The profile is based on the Intermediate compatibility recommended configuration. The |
||
|
This profile is intended for use with modern clients that have no need for backwards compatibility. This profile is based on the Modern compatibility recommended configuration. The
|
||
|
This profile allows you to define the TLS version and ciphers to use.
|
When using one of the predefined profile types, the effective profile configuration is subject to change between releases. For example, given a specification to use the Intermediate profile deployed on release X.Y.Z, an upgrade to release X.Y.Z+1 might cause a new profile configuration to be applied, resulting in a rollout. |
To configure a TLS security profile for an Ingress Controller, edit the IngressController
custom resource (CR) to specify a predefined or custom TLS security profile. If a TLS security profile is not configured, the default value is based on the TLS security profile set for the API server.
IngressController
CR that configures the Old
TLS security profileapiVersion: operator.openshift.io/v1
kind: IngressController
...
spec:
tlsSecurityProfile:
old: {}
type: Old
...
The TLS security profile defines the minimum TLS version and the TLS ciphers for TLS connections for Ingress Controllers.
You can see the ciphers and the minimum TLS version of the configured TLS security profile in the IngressController
custom resource (CR) under Status.Tls Profile
and the configured TLS security profile under Spec.Tls Security Profile
. For the Custom
TLS security profile, the specific ciphers and minimum TLS version are listed under both parameters.
The HAProxy Ingress Controller image supports TLS The Ingress Operator also converts the TLS |
You have access to the cluster as a user with the cluster-admin
role.
Edit the IngressController
CR in the openshift-ingress-operator
project to configure the TLS security profile:
$ oc edit IngressController default -n openshift-ingress-operator
Add the spec.tlsSecurityProfile
field:
IngressController
CR for a Custom
profileapiVersion: operator.openshift.io/v1
kind: IngressController
...
spec:
tlsSecurityProfile:
type: Custom (1)
custom: (2)
ciphers: (3)
- ECDHE-ECDSA-CHACHA20-POLY1305
- ECDHE-RSA-CHACHA20-POLY1305
- ECDHE-RSA-AES128-GCM-SHA256
- ECDHE-ECDSA-AES128-GCM-SHA256
minTLSVersion: VersionTLS11
...
1 | Specify the TLS security profile type (Old , Intermediate , or Custom ). The default is Intermediate . |
2 | Specify the appropriate field for the selected type:
|
3 | For the custom type, specify a list of TLS ciphers and minimum accepted TLS version. |
Save the file to apply the changes.
Verify that the profile is set in the IngressController
CR:
$ oc describe IngressController default -n openshift-ingress-operator
Name: default
Namespace: openshift-ingress-operator
Labels: <none>
Annotations: <none>
API Version: operator.openshift.io/v1
Kind: IngressController
...
Spec:
...
Tls Security Profile:
Custom:
Ciphers:
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-GCM-SHA256
Min TLS Version: VersionTLS11
Type: Custom
...
You can configure the Ingress Controller to enable mutual TLS (mTLS) authentication by setting a spec.clientTLS
value. The clientTLS
value configures the Ingress Controller to verify client certificates. This configuration includes setting a clientCA
value, which is a reference to a config map. The config map contains the PEM-encoded CA certificate bundle that is used to verify a client’s certificate. Optionally, you can configure a list of certificate subject filters.
If the clientCA
value specifies an X509v3 certificate revocation list (CRL) distribution point, the Ingress Operator downloads the CRL and configures the Ingress Controller to acknowledge it. Requests that do not provide valid certificates are rejected.
You have access to the cluster as a user with the cluster-admin
role.
Create a config map that is in the openshift-config
namespace:
$ oc create configmap router-ca-certs-default --from-file=ca-bundle.pem=client-ca.crt -n openshift-config
The config map data key must be |
Edit the IngressController
resource in the openshift-ingress-operator
project:
$ oc edit IngressController default -n openshift-ingress-operator
Add the spec.clientTLS field and subfields to configure mutual TLS:
IngressController
CR for a clientTLS
profile that specifies filtering patterns apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
clientTLS:
clientCertificatePolicy: Required
clientCA:
name: router-ca-certs-default
allowedSubjectPatterns:
- "^/CN=example.com/ST=NC/C=US/O=Security/OU=OpenShift$"
The Ingress Operator is a core feature of OKD and is enabled out of the box.
Every new OKD installation has an ingresscontroller
named default. It
can be supplemented with additional Ingress Controllers. If the default
ingresscontroller
is deleted, the Ingress Operator will automatically recreate it
within a minute.
View the default Ingress Controller:
$ oc describe --namespace=openshift-ingress-operator ingresscontroller/default
You can view and inspect the status of your Ingress Operator.
View your Ingress Operator status:
$ oc describe clusteroperators/ingress
You can view your Ingress Controller logs.
View your Ingress Controller logs:
$ oc logs --namespace=openshift-ingress-operator deployments/ingress-operator -c <container_name>
Your can view the status of a particular Ingress Controller.
View the status of an Ingress Controller:
$ oc describe --namespace=openshift-ingress-operator ingresscontroller/<name>
As an administrator, you can configure an Ingress Controller to use a custom
certificate by creating a Secret resource and editing the IngressController
custom resource (CR).
You must have a certificate/key pair in PEM-encoded files, where the certificate is signed by a trusted certificate authority or by a private trusted certificate authority that you configured in a custom PKI.
Your certificate meets the following requirements:
The certificate is valid for the ingress domain.
The certificate uses the subjectAltName
extension to specify a wildcard domain, such as *.apps.ocp4.example.com
.
You must have an IngressController
CR. You may use the default one:
$ oc --namespace openshift-ingress-operator get ingresscontrollers
NAME AGE
default 10m
If you have intermediate certificates, they must be included in the |
The following assumes that the custom certificate and key pair are in the
tls.crt
and tls.key
files in the current working directory. Substitute the
actual path names for tls.crt
and tls.key
. You also may substitute another
name for custom-certs-default
when creating the Secret resource and
referencing it in the IngressController CR.
This action will cause the Ingress Controller to be redeployed, using a rolling deployment strategy. |
Create a Secret resource containing the custom certificate in the
openshift-ingress
namespace using the tls.crt
and tls.key
files.
$ oc --namespace openshift-ingress create secret tls custom-certs-default --cert=tls.crt --key=tls.key
Update the IngressController CR to reference the new certificate secret:
$ oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/default \
--patch '{"spec":{"defaultCertificate":{"name":"custom-certs-default"}}}'
Verify the update was effective:
$ echo Q |\
openssl s_client -connect console-openshift-console.apps.<domain>:443 -showcerts 2>/dev/null |\
openssl x509 -noout -subject -issuer -enddate
where:
<domain>
Specifies the base domain name for your cluster.
subject=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = *.apps.example.com
issuer=C = US, ST = NC, L = Raleigh, O = RH, OU = OCP4, CN = example.com
notAfter=May 10 08:32:45 2022 GM
You can alternatively apply the following YAML to set a custom default certificate:
|
The certificate secret name should match the value used to update the CR.
Once the IngressController CR has been modified, the Ingress Operator updates the Ingress Controller’s deployment to use the custom certificate.
As an administrator, you can remove a custom certificate that you configured an Ingress Controller to use.
You have access to the cluster as a user with the cluster-admin
role.
You have installed the OpenShift CLI (oc
).
You previously configured a custom default certificate for the Ingress Controller.
To remove the custom certificate and restore the certificate that ships with OKD, enter the following command:
$ oc patch -n openshift-ingress-operator ingresscontrollers/default \
--type json -p $'- op: remove\n path: /spec/defaultCertificate'
There can be a delay while the cluster reconciles the new certificate configuration.
To confirm that the original cluster certificate is restored, enter the following command:
$ echo Q | \
openssl s_client -connect console-openshift-console.apps.<domain>:443 -showcerts 2>/dev/null | \
openssl x509 -noout -subject -issuer -enddate
where:
<domain>
Specifies the base domain name for your cluster.
subject=CN = *.apps.<domain>
issuer=CN = ingress-operator@1620633373
notAfter=May 10 10:44:36 2023 GMT
Automatically scale an Ingress Controller to dynamically meet routing performance or availability requirements such as the requirement to increase throughput. The following procedure provides an example for scaling up the default IngressController
.
The Custom Metrics Autoscaler is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope. |
You have the OpenShift CLI (oc
) installed.
You have access to an OKD cluster as a user with the cluster-admin
role.
You have the Custom Metrics Autoscaler Operator installed.
Create a project in the openshift-ingress-operator
namespace by running the following command:
$ oc project openshift-ingress-operator
Enable OpenShift monitoring for user-defined projects by creating and applying a config map:
Create a new ConfigMap
object, cluster-monitoring-config.yaml
:
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
enableUserWorkload: true (1)
1 | When set to true , the enableUserWorkload parameter enables monitoring for user-defined projects in a cluster. |
Apply the config map by running the following command:
$ oc apply -f cluster-monitoring-config.yaml
Create a service account to authenticate with Thanos by running the following command:
$ oc create serviceaccount thanos && oc describe serviceaccount thanos
Name: thanos
Namespace: openshift-ingress-operator
Labels: <none>
Annotations: <none>
Image pull secrets: thanos-dockercfg-b4l9s
Mountable secrets: thanos-dockercfg-b4l9s
Tokens: thanos-token-c422q
Events: <none>
Define a TriggerAuthentication
object within the openshift-ingress-operator
namespace using the service account’s token.
Define the variable secret
that contains the secret by running the following command:
$ secret=$(oc get secret | grep thanos-token | head -n 1 | awk '{ print $1 }')
Create the TriggerAuthentication
object and pass the value of the secret
variable to the TOKEN
parameter:
$ oc process TOKEN="$secret" -f - <<EOF | oc apply -f -
apiVersion: template.openshift.io/v1
kind: Template
parameters:
- name: TOKEN
objects:
- apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: keda-trigger-auth-prometheus
spec:
secretTargetRef:
- parameter: bearerToken
name: \${TOKEN}
key: token
- parameter: ca
name: \${TOKEN}
key: ca.crt
EOF
Create and apply a role for reading metrics from Thanos:
Create a new role, thanos-metrics-reader.yaml
, that reads metrics from pods and nodes:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: thanos-metrics-reader
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
verbs:
- get
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
Apply the new role by running the following command:
$ oc apply -f thanos-metrics-reader.yaml
Add the new role to the service account by entering the following commands:
$ oc adm policy add-role-to-user thanos-metrics-reader -z thanos --role=namespace=openshift-ingress-operator
$ oc adm policy -n openshift-ingress-operator add-cluster-role-to-user cluster-monitoring-view -z thanos
The argument |
Create a new ScaledObject
YAML file, ingress-autoscaler.yaml
, that targets the default Ingress Controller deployment:
ScaledObject
definitionapiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: ingress-scaler
spec:
scaleTargetRef: (1)
apiVersion: operator.openshift.io/v1
kind: IngressController
name: default
envSourceContainerName: ingress-operator
minReplicaCount: 1
maxReplicaCount: 20 (2)
cooldownPeriod: 1
pollingInterval: 1
triggers:
- type: prometheus
metricType: AverageValue
metadata:
serverAddress: https://<example-cluster>:9091 (3)
namespace: openshift-ingress-operator (4)
metricName: 'kube-node-role'
threshold: '1'
query: 'sum(kube_node_role{role="worker",service="kube-state-metrics"})' (5)
authModes: "bearer"
authenticationRef:
name: keda-trigger-auth-prometheus
1 | The custom resource that you are targeting. In this case, the Ingress Controller. |
2 | Optional: The maximum number of replicas. If you omit this field, the default maximum is set to 100 replicas. |
3 | The cluster address and port. |
4 | The Ingress Operator namespace. |
5 | This expression evaluates to however many worker nodes are present in the deployed cluster. |
If you are using cross-namespace queries, you must target port 9091 and not port 9092 in the |
Apply the custom resource definition by running the following command:
$ oc apply -f ingress-autoscaler.yaml
Verify that the default Ingress Controller is scaled out to match the value returned by the kube-state-metrics
query by running the following commands:
Use the grep
command to search the Ingress Controller YAML file for replicas:
$ oc get ingresscontroller/default -o yaml | grep replicas:
replicas: 3
Get the pods in the openshift-ingress
project:
$ oc get pods -n openshift-ingress
NAME READY STATUS RESTARTS AGE
router-default-7b5df44ff-l9pmm 2/2 Running 0 17h
router-default-7b5df44ff-s5sl5 2/2 Running 0 3d22h
router-default-7b5df44ff-wwsth 2/2 Running 0 66s
Manually scale an Ingress Controller to meeting routing performance or
availability requirements such as the requirement to increase throughput. oc
commands are used to scale the IngressController
resource. The following
procedure provides an example for scaling up the default IngressController
.
Scaling is not an immediate action, as it takes time to create the desired number of replicas. |
View the current number of available replicas for the default IngressController
:
$ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'
2
Scale the default IngressController
to the desired number of replicas using
the oc patch
command. The following example scales the default IngressController
to 3 replicas:
$ oc patch -n openshift-ingress-operator ingresscontroller/default --patch '{"spec":{"replicas": 3}}' --type=merge
ingresscontroller.operator.openshift.io/default patched
Verify that the default IngressController
scaled to the number of replicas
that you specified:
$ oc get -n openshift-ingress-operator ingresscontrollers/default -o jsonpath='{$.status.availableReplicas}'
3
You can alternatively apply the following YAML to scale an Ingress Controller to three replicas:
|
1 | If you need a different amount of replicas, change the replicas value. |
You can configure the Ingress Controller to enable access logs. If you have clusters that do not receive much traffic, then you can log to a sidecar. If you have high traffic clusters, to avoid exceeding the capacity of the logging stack or to integrate with a logging infrastructure outside of OKD, you can forward logs to a custom syslog endpoint. You can also specify the format for access logs.
Container logging is useful to enable access logs on low-traffic clusters when there is no existing Syslog logging infrastructure, or for short-term use while diagnosing problems with the Ingress Controller.
Syslog is needed for high-traffic clusters where access logs could exceed the OpenShift Logging stack’s capacity, or for environments where any logging solution needs to integrate with an existing Syslog logging infrastructure. The Syslog use-cases can overlap.
Log in as a user with cluster-admin
privileges.
Configure Ingress access logging to a sidecar.
To configure Ingress access logging, you must specify a destination using spec.logging.access.destination
. To specify logging to a sidecar container, you must specify Container
spec.logging.access.destination.type
. The following example is an Ingress Controller definition that logs to a Container
destination:
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
replicas: 2
logging:
access:
destination:
type: Container
When you configure the Ingress Controller to log to a sidecar, the operator creates a container named logs
inside the Ingress Controller Pod:
$ oc -n openshift-ingress logs deployment.apps/router-default -c logs
2020-05-11T19:11:50.135710+00:00 router-default-57dfc6cd95-bpmk6 router-default-57dfc6cd95-bpmk6 haproxy[108]: 174.19.21.82:39654 [11/May/2020:19:11:50.133] public be_http:hello-openshift:hello-openshift/pod:hello-openshift:hello-openshift:10.128.2.12:8080 0/0/1/0/1 200 142 - - --NI 1/1/0/0/0 0/0 "GET / HTTP/1.1"
Configure Ingress access logging to a Syslog endpoint.
To configure Ingress access logging, you must specify a destination using spec.logging.access.destination
. To specify logging to a Syslog endpoint destination, you must specify Syslog
for spec.logging.access.destination.type
. If the destination type is Syslog
, you must also specify a destination endpoint using spec.logging.access.destination.syslog.endpoint
and you can specify a facility using spec.logging.access.destination.syslog.facility
. The following example is an Ingress Controller definition that logs to a Syslog
destination:
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
replicas: 2
logging:
access:
destination:
type: Syslog
syslog:
address: 1.2.3.4
port: 10514
The |
Configure Ingress access logging with a specific log format.
You can specify spec.logging.access.httpLogFormat
to customize the log format. The following example is an Ingress Controller definition that logs to a syslog
endpoint with IP address 1.2.3.4 and port 10514:
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
replicas: 2
logging:
access:
destination:
type: Syslog
syslog:
address: 1.2.3.4
port: 10514
httpLogFormat: '%ci:%cp [%t] %ft %b/%s %B %bq %HM %HU %HV'
Disable Ingress access logging.
To disable Ingress access logging, leave spec.logging
or spec.logging.access
empty:
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
replicas: 2
logging:
access: null
A cluster administrator can set the thread count to increase the amount of incoming connections a cluster can handle. You can patch an existing Ingress Controller to increase the amount of threads.
The following assumes that you already created an Ingress Controller.
Update the Ingress Controller to increase the number of threads:
$ oc -n openshift-ingress-operator patch ingresscontroller/default --type=merge -p '{"spec":{"tuningOptions": {"threadCount": 8}}}'
If you have a node that is capable of running large amounts of resources, you can configure |
When creating an Ingress Controller on cloud platforms, the Ingress Controller is published by a public cloud load balancer by default. As an administrator, you can create an Ingress Controller that uses an internal cloud load balancer.
If your cloud provider is Microsoft Azure, you must have at least one public load balancer that points to your nodes. If you do not, all of your nodes will lose egress connectivity to the internet. |
If you want to change the |