1. Documentation
×

About route advertisements

Advertise cluster network routes with Border Gateway Protocol

With route advertisements enabled, the OVN-Kubernetes network plugin supports advertising network routes for the default pod network and cluster user-defined (CUDN) networks to the provider network, including EgressIPs, and importing routes from the provider network to the default pod network and CUDNs. From the provider network, IP addresses advertised from the default pod network and CUDNs can be reached directly.

For example, you can import routes to the default pod network so you no longer need to manually configure routes on each node. Previously, you might have been using local gateway mode (RoutingViaHost=true) and manually configuring routes on each node to approximate a similar configuration. With route advertisements you can accomplish this seamlessly and you can use shared gateway mode (RoutingViaHost=false) as well.

Route reflectors on the provider network are supported and can reduce the number of BGP connections required to advertise routes on large networks.

If you use EgressIPs with route advertisements enabled, the layer 3 provider network is aware of EgressIP failovers. This allows you to locate cluster nodes that host EgressIPs on different layer 2 segments whereas before only the layer 2 provider network was aware so that required all the egress nodes to be on the same layer 2 segment.

Supported platforms

Advertising routes with border gateway protocol (BGP) is supported on the following infrastructure types:

  • Bare-metal

Infrastructure requirements

To use route advertisements, you must have configured BGP for your network infrastructure. Outages or misconfigurations of your network infrastructure might cause disruptions to your cluster network.

Compatibility with other networking features

Route advertisements support the following OKD Networking features:

Multiple external gateways (MEG)

MEG is not supported with this feature.

EgressIPs

Supports the use and advertisement of EgressIPs. The node where an egress IP address resides advertises the EgressIP. An egress IP address must be on the same layer 2 network subnet as the egress node. The following limitations apply:

  • Advertising EgressIPs from a user-defined network (CUDN) operating in layer 2 mode are not supported.

  • Advertising EgressIPs for a network that has both egress IP addresses assigned to the primary network interface and egress IP addresses assigned to additional network interfaces is impractical. All EgressIPs are advertised on all of the BGP sessions of the selected FRRConfiguration instances, regardless of whether these sessions are established over the same interface that the EgressIP is assigned to or not, potentially leading to unwanted advertisements.

Services

Works with the MetalLB Operator to advertise services to the provider network.

Egress service

Full support.

Egress firewall

Full support.

Egress QoS

Full support.

Network policies

Full support.

Direct pod ingress

Full support for the default cluster network and cluster user-defined (CUDN) networks.

Considerations for use with the MetalLB Operator

The MetalLB Operator is installed as an add-on to the cluster. Deployment of the MetalLB Operator automatically enables FRR-K8s as an additional routing capability provider. This feature and the MetalLB Operator use the same FRR-K8s deployment.

Considerations for naming cluster user-defined networks (CUDNs)

When referencing a VRF device in a FRRConfiguration CR, the VRF name is the same as the CUDN name for VRF names that are less than or equal to 15 characters. It is recommended to use a VRF name no longer than 15 characters so that the VRF name can be inferred from the CUDN name.

BGP routing custom resources

The following custom resources (CRs) are used to configure route advertisements with BGP:

RouteAdvertisements

This CR defines the advertisements for the BGP routing. From this CR, the OVN-Kubernetes controller generates a FRRConfiguration object that configures the FRR daemon to advertise cluster network routes. This CR is cluster scoped.

FRRConfiguration

This CR is used to define BGP peers and to configure route imports from the provider network into the cluster network. Before applying RouteAdvertisements objects, at least one FRRConfiguration object must be initially defined to configure the BGP peers. This CR is namespaced.

OVN-Kubernetes controller generation of FRRConfiguration objects

An FRRConfiguration object is generated for each network and node selected by a RouteAdvertisements CR with the appropriate advertised prefixes that apply to each node. The OVN-Kubernetes controller checks whether the RouteAdvertisements-CR-selected nodes are a subset of the nodes that are selected by the RouteAdvertisements-CR-selected FRR configurations.

Any filtering or selection of prefixes to receive are not considered in FRRConfiguration objects that are generated from the RouteAdvertisement CRs. Configure any prefixes to receive on other FRRConfiguration objects. OVN-Kubernetes imports routes from the VRF into the appropriate network.

Cluster Network Operator configuration

The Cluster Network Operator (CNO) API exposes several fields to configure route advertisements:

  • spec.additionalRoutingCapabilities.providers: Specifies an additional routing provider, which is required to advertise routes. The only supported value is FRR, which enables deployment of the FRR-K8S daemon for the cluster. When enabled, the FRR-K8S daemon is deployed on all nodes.

  • spec.defaultNetwork.ovnKubernetesConfig.routeAdvertisements: Enables route advertisements for the default cluster network and CUDN networks. The spec.additionalRoutingCapabilities field must be set to FRR to enable this feature.

RouteAdvertisements object configuration

You can define an RouteAdvertisements object, which is cluster scoped, with the following properties.

The fields for the RouteAdvertisements custom resource (CR) are described in the following table:

Table 1. RouteAdvertisements object
Field Type Description

metadata.name

string

Specifies the name of the RouteAdvertisements object.

advertisements

array

Specifies an array that can contain a list of different types of networks to advertise. Supports only the "PodNetwork" and "EgressIP" values.

frrConfigurationSelector

object

Determines which FRRConfiguration CR the OVN-Kubernetes driven FRRConfiguration CR is based on.

networkSelector

object

Specifies which networks to advertise among default cluster network and cluster user defined networks (CUDNs).

nodeSelector

object

Limits the advertisements to selected nodes. When advertisements="PodNetwork" is selected, all nodes must be selected. When advertisements="EgressIP" is selected, only the egress IP addresses assigned to the selected nodes are advertised.

targetVRF

string

Determines which router to advertise the routes in. Routes are advertised on the routers associated with this virtual routing and forwarding (VRF) target, as specified on the selected FRRConfiguration CR. When omitted, the default VRF is used as the target. When specified as auto, a VRF with the same name as the network name is used as the target.

Examples advertising pod IP addresses with BGP

The following examples describe several configurations for advertising pod IP addresses and EgressIPs with Border Gateway Protocol (BGP). The external network border router has the 172.18.0.5 IP address. These configures assume that you have configured an external route reflector that can relay routes to all nodes on the cluster network.

Advertising the default cluster network

In this scenario, the default cluster network is exposed to the external network so that pod IP addresses and EgressIPs are advertised to the provider network.

This scenario relies upon the following FRRConfiguration object:

FRRConfiguration CR
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
  name: default
spec:
  advertisements:
  - PodNetwork
  - EgressIP
  networkSelectors:
  - networkSelectionType: DefaultNetwork
  frrConfigurationSelector:
    matchLabels:
      routeAdvertisements: receive-all
  nodeSelector: {}

When the OVN-Kubernetes controller sees this RouteAdvertisements CR, it generates further FRRConfiguration objects based on the selected ones that configure the FRR daemon to advertise the routes for the default cluster network.

An example of a FRRConfiguration CR generated by OVN-Kubernetes
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: ovnk-generated-abcdef
  namespace: openshift-frr-k8s
spec:
  bgp:
    routers:
    - asn: 64512
      neighbors:
        - address: 172.18.0.5
          asn: 64512
          toReceive:
            allowed:
              mode: filtered
          toAdvertise:
            allowed:
              prefixes:
              - <default_network_host_subnet>
      prefixes:
      - <default_network_host_subnet>
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: ovn-worker

In the example generated FRRConfiguration object, <default_network_host_subnet> is the subnet of the default cluster network that is advertised to the provider network.

Advertising pod IPs from a cluster user-defined network over BGP

In this scenario, the blue cluster user-defined network (CUDN) is exposed to the external network so that the network’s pod IP addresses and EgressIPs are advertised to the provider network.

This scenario relies upon the following FRRConfiguration object:

FRRConfiguration CR
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: receive-all
  namespace: openshift-frr-k8s
  labels:
    routeAdvertisements: receive-all
spec:
  bgp:
    routers:
    - asn: 64512
      neighbors:
      - address: 172.18.0.5
        asn: 64512
        disableMP: true
        toReceive:
          allowed:
            mode: all

With this FRRConfiguration object, routes will be imported from neighbor 172.18.0.5 into the default VRF and are available to the default cluster network.

The CUDNs are advertised over the default VRF as illustrated in the following diagram:

Advertising pod IPs from a cluster user-defined network over BGP
Red CUDN

  • A VRF named red associated with a CUDN named red

  • A subnet of 10.0.0.0/24

Blue CUDN

  • A VRF named blue associated with a CUDN named blue

  • A subnet of 10.0.1.0/24

In this configuration, two separate CUDNs are defined. The red network covers the 10.0.0.0/24 subnet and the blue network covers the 10.0.1.0/24 subnet. The red and blue networks are labeled as export: true.

The following RouteAdvertisements CR describes the configuration for the red and blue tenants:

RouteAdvertisements CR for the red and blue tenants
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
  name: advertise-cudns
spec:
  advertisements:
  - PodNetwork
  - EgressIP
  networkSelectors:
  - networkSelectionType: ClusterUserDefinedNetworks
    clusterUserDefinedNetworkSelector:
      networkSelector:
        matchLabels:
          export: "true"
  frrConfigurationSelector:
    matchLabels:
      routeAdvertisements: receive-all
  nodeSelector: {}

When the OVN-Kubernetes controller sees this RouteAdvertisements CR, it generates generates further FRRConfiguration objects based on the selected ones that configure the FRR daemon to advertise the routes. The following example is of one such configuration object, with the number of FRRConfiguration objects created depending on the node and networks selected.

An example of a FRRConfiguration CR generated by OVN-Kubernetes
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: ovnk-generated-abcdef
  namespace: openshift-frr-k8s
spec:
  bgp:
    routers:
    - asn: 64512
      vrf: blue
      imports:
      - vrf: default
    - asn: 64512
      neighbors:
        - address: 172.18.0.5
          asn: 64512
          toReceive:
            allowed:
              mode: filtered
          toAdvertise:
            allowed:
              prefixes:
              - 10.0.1.0/24
      prefixes:
      - 10.0.1.0/24
      imports:
      - vrf: blue
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: ovn-worker

The generated FRRConfiguration object configures the subnet 10.0.1.0/24, which belongs to network blue, to be imported into the default VRF and advertised to the 172.18.0.5 neighbor. An FRRConfiguration object is generated for each network and nodes selected by a RouteAdvertisements CR with the appropriate prefixes that apply to each node.

When the targetVRF field is omitted, the routes are leaked and advertised over the default VRF. Additionally, routes that were imported to the default VRF after the definition of the initial FRRConfiguration object are also imported into the blue VRF.

Advertising pod IPs from a cluster user-defined network over BGP with VPN

In this scenario, a VLAN interface is attached to the VRF device associated with the blue network. This setup provides a VRF lite design, where FRR-K8S is used to advertise the blue network only over the corresponding BGP session on the blue network VRF/VLAN link to the next hop Provide Edge (PE) router. The red tenant uses the same configuration. The blue and red networks are labeled as export: true.

This scenario does not support the use of EgressIPs.

The following diagram illustrates this configuration:

Advertising pod IPs from a cluster user-defined network over BGP with VPN
Red CUDN

  • A VRF named red associated with a CUDN named red

  • A VLAN interface attached to the VRF device and connected to the external PE router

  • An assigned subnet of 10.0.2.0/24

Blue CUDN

  • A VRF named blue associated with a CUDN named blue

  • A VLAN interface attached to the VRF device and connected to the external PE router

  • An assigned subnet of 10.0.1.0/24

This approach is available only when you use OVN-Kubernetes in local gateway mode by setting routingViaHost=true.

In the following configuration, an additional FRRConfiguration CR configures peering with the PE router on the blue and red VLANs:

FRRConfiguration CR manually configured for BGP VPN setup
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: vpn-blue-red
  namespace: openshift-frr-k8s
  labels:
    routeAdvertisements: vpn-blue-red
spec:
  bgp:
    routers:
    - asn: 64512
      vrf: blue
      neighbors:
      - address: 182.18.0.5
        asn: 64512
        toReceive:
          allowed:
            mode: filtered
    - asn: 64512
      vrf: red
      neighbors:
      - address: 192.18.0.5
        asn: 64512
        toReceive:
          allowed:
            mode: filtered

The following RouteAdvertisements CR describes the configuration for the blue and red tenants:

RouteAdvertisements CR for the blue and red tenants
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
  name: advertise-vrf-lite
spec:
  targetVRF: auto
  advertisements:
  - "PodNetwork"
  nodeSelector: {}
  frrConfigurationSelector:
    matchLabels:
      routeAdvertisements: vpn-blue-red
  networkSelectors:
  - networkSelectionType: ClusterUserDefinedNetworks
    clusterUserDefinedNetworkSelector:
      networkSelector:
        matchLabels:
          export: "true"

In the RouteAdvertisements CR, the targetVRF is set to auto so that advertisements occur within the VRF device that corresponds to the individual networks that are selected. In this scenario, the pod subnet for blue is advertised over the blue VRF device, and the pod subnet for red is advertised over the red VRF device. Additionally, each BGP session imports routes to only the corresponding CUDN VRF as defined by the initial FRRConfiguration object.

When the OVN-Kubernetes controller sees this RouteAdvertisements CR, it generates further FRRConfiguration objects based on the selected ones that configure the FRR daemon to advertise the routes for the blue and red tenants.

FRRConfiguration CR generated by OVN-Kubernetes for blue and red tenants
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: ovnk-generated-abcde
  namespace: openshift-frr-k8s
spec:
  bgp:
    routers:
    - asn: 64512
      neighbors:
      - address: 182.18.0.5
        asn: 64512
        toReceive:
          allowed:
            mode: filtered
        toAdvertise:
          allowed:
            prefixes:
            - 10.0.1.0/24
      vrf: blue
      prefixes:
        - 10.0.1.0/24
    - asn: 64512
      neighbors:
      - address: 192.18.0.5
        asn: 64512
        toReceive:
          allowed:
            mode: filtered
        toAdvertise:
          allowed:
            prefixes:
            - 10.0.2.0/24
      vrf: red
      prefixes:
         - 10.0.2.0/24
  nodeSelector:
     matchLabels:
        kubernetes.io/hostname: ovn-worker

In this scenario, any filtering or selection of routes to receive must be done in the FRRConfiguration CR that defines peering relationships.

Additional resources