"event":"ipAllocated","ip":"172.22.0.201","msg":"IP address assigned by controller
As a cluster administrator, you can add the MetalLB Operator to your cluster so that when a service of type LoadBalancer
is added to the cluster, MetalLB can add an external IP address for the service.
The external IP address is added to the host network for your cluster.
Using MetalLB is valuable when you have a bare-metal cluster, or an infrastructure that is like bare metal, and you want fault-tolerant access to an application through an external IP address.
You must configure your networking infrastructure to ensure that network traffic for the external IP address is routed from clients to the host network for the cluster.
After deploying MetalLB with the MetalLB Operator, when you add a service of type LoadBalancer
, MetalLB provides a platform-native load balancer.
The MetalLB Operator monitors its own namespace for the following custom resources:
MetalLB
When you add a MetalLB
custom resource to the cluster, the MetalLB Operator deploys MetalLB on the cluster.
The Operator only supports a single instance of the custom resource.
If the instance is deleted, the Operator removes MetalLB from the cluster.
IPAddressPool
MetalLB requires one or more pools of IP addresses that it can assign to a service when you add a service of type LoadBalancer
.
An IPAddressPool
includes a list of IP addresses.
The list can be a single IP address that is set using a range, such as 1.1.1.1-1.1.1.1, a range specified in CIDR notation, a range specified as a starting and ending address separated by a hyphen, or a combination of the three.
An IPAddressPool
requires a name.
The documentation uses names like doc-example
, doc-example-reserved
, and doc-example-ipv6
.
An IPAddressPool
assigns IP addresses from the pool.
L2Advertisement
and BGPAdvertisement
custom resources enable the advertisement of a given IP from a given pool.
A single |
BGPPeer
The BGP peer custom resource identifies the BGP router for MetalLB to communicate with, the AS number of the router, the AS number for MetalLB, and customizations for route advertisement. MetalLB advertises the routes for service load-balancer IP addresses to one or more BGP peers.
BFDProfile
The BFD profile custom resource configures Bidirectional Forwarding Detection (BFD) for a BGP peer. BFD provides faster path failure detection than BGP alone provides.
L2Advertisement
The L2Advertisement custom resource advertises an IP coming from an IPAddressPool
using the L2 protocol.
BGPAdvertisement
The BGPAdvertisement custom resource advertises an IP coming from an IPAddressPool
using the BGP protocol.
After you add the MetalLB
custom resource to the cluster and the Operator deploys MetalLB, the controller
and speaker
MetalLB software components begin running.
MetalLB validates all relevant custom resources.
When you install the MetalLB Operator, the metallb-operator-controller-manager
deployment starts a pod. The pod is the implementation of the Operator. The pod monitors for changes to all the relevant resources.
When the Operator starts an instance of MetalLB, it starts a controller
deployment and a speaker
daemon set.
controller
The Operator starts the deployment and a single pod. When you add a service of type LoadBalancer
, Kubernetes uses the controller
to allocate an IP address from an address pool.
In case of a service failure, verify you have the following entry in your controller
pod logs:
"event":"ipAllocated","ip":"172.22.0.201","msg":"IP address assigned by controller
speaker
The Operator starts a daemon set for speaker
pods. By default, a pod is started on each node in your cluster. You can limit the pods to specific nodes by specifying a node selector in the MetalLB
custom resource when you start MetalLB. If the controller
allocated the IP address to the service and service is still unavailable, read the speaker
pod logs. If the speaker
pod is unavailable, run the oc describe pod -n
command.
For layer 2 mode, after the controller
allocates an IP address for the service, the speaker
pods use an algorithm to determine which speaker
pod on which node will announce the load balancer IP address.
The algorithm involves hashing the node name and the load balancer IP address. For more information, see "MetalLB and external traffic policy".
The speaker
uses Address Resolution Protocol (ARP) to announce IPv4 addresses and Neighbor Discovery Protocol (NDP) to announce IPv6 addresses.
For Border Gateway Protocol (BGP) mode, after the controller
allocates an IP address for the service, each speaker
pod advertises the load balancer IP address with its BGP peers. You can configure which nodes start BGP sessions with BGP peers.
Requests for the load balancer IP address are routed to the node with the speaker
that announces the IP address. After the node receives the packets, the service proxy routes the packets to an endpoint for the service. The endpoint can be on the same node in the optimal case, or it can be on another node. The service proxy chooses an endpoint each time a connection is established.
With layer 2 mode, one node in your cluster receives all the traffic for the service IP address. With BGP mode, a router on the host network opens a connection to one of the nodes in the cluster for a new client connection. How your cluster handles the traffic after it enters the node is affected by the external traffic policy.
cluster
This is the default value for spec.externalTrafficPolicy
.
With the cluster
traffic policy, after the node receives the traffic, the service proxy distributes the traffic to all the pods in your service.
This policy provides uniform traffic distribution across the pods, but it obscures the client IP address and it can appear to the application in your pods that the traffic originates from the node rather than the client.
local
With the local
traffic policy, after the node receives the traffic, the service proxy only sends traffic to the pods on the same node.
For example, if the speaker
pod on node A announces the external service IP, then all traffic is sent to node A.
After the traffic enters node A, the service proxy only sends traffic to pods for the service that are also on node A.
Pods for the service that are on additional nodes do not receive any traffic from node A.
Pods for the service on additional nodes act as replicas in case failover is needed.
This policy does not affect the client IP address. Application pods can determine the client IP address from the incoming connections.
The following information is important when configuring the external traffic policy in BGP mode. Although MetalLB advertises the load balancer IP address from all the eligible nodes, the number of nodes loadbalancing the service can be limited by the capacity of the router to establish equal-cost multipath (ECMP) routes. If the number of nodes advertising the IP is greater than the ECMP group limit of the router, the router will use less nodes than the ones advertising the IP. For example, if the external traffic policy is set to |
In layer 2 mode, the speaker
pod on one node announces the external IP address for a service to the host network.
From a network perspective, the node appears to have multiple IP addresses assigned to a network interface.
Since layer 2 mode relies on ARP and NDP, the client must be on the same subnet of the nodes announcing the service in order for MetalLB to work. Additionally, the IP address assigned to the service must be on the same subnet of the network used by the client to reach the service. |
The speaker
pod responds to ARP requests for IPv4 services and NDP requests for IPv6.
In layer 2 mode, all traffic for a service IP address is routed through one node. After traffic enters the node, the service proxy for the CNI network provider distributes the traffic to all the pods for the service.
Because all traffic for a service enters through a single node in layer 2 mode, in a strict sense, MetalLB does not implement a load balancer for layer 2.
Rather, MetalLB implements a failover mechanism for layer 2 so that when a speaker
pod becomes unavailable, a speaker
pod on a different node can announce the service IP address.
When a node becomes unavailable, failover is automatic.
The speaker
pods on the other nodes detect that a node is unavailable and a new speaker
pod and node take ownership of the service IP address from the failed node.
The preceding graphic shows the following concepts related to MetalLB:
An application is available through a service that has a cluster IP on the 172.130.0.0/16
subnet.
That IP address is accessible from inside the cluster.
The service also has an external IP address that MetalLB assigned to the service, 192.168.100.200
.
Nodes 1 and 3 have a pod for the application.
The speaker
daemon set runs a pod on each node.
The MetalLB Operator starts these pods.
Each speaker
pod is a host-networked pod.
The IP address for the pod is identical to the IP address for the node on the host network.
The speaker
pod on node 1 uses ARP to announce the external IP address for the service, 192.168.100.200
.
The speaker
pod that announces the external IP address must be on the same node as an endpoint for the service and the endpoint must be in the Ready
condition.
Client traffic is routed to the host network and connects to the 192.168.100.200
IP address.
After traffic enters the node, the service proxy sends the traffic to the application pod on the same node or another node according to the external traffic policy that you set for the service.
If the external traffic policy for the service is set to cluster
, the node that advertises the 192.168.100.200
load balancer IP address is selected from the nodes where a speaker
pod is running. Only that node can receive traffic for the service.
If the external traffic policy for the service is set to local
, the node that advertises the 192.168.100.200
load balancer IP address is selected from the nodes where a speaker
pod is running and at least an endpoint of the service. Only that node can receive traffic for the service. In the preceding graphic, either node 1 or 3 would advertise 192.168.100.200
.
If node 1 becomes unavailable, the external IP address fails over to another node.
On another node that has an instance of the application pod and service endpoint, the speaker
pod begins to announce the external IP address, 192.168.100.200
and the new node receives the client traffic.
In the diagram, the only candidate is node 3.
In BGP mode, by default each speaker
pod advertises the load balancer IP address for a service to each BGP peer. It is also possible to advertise the IPs coming from a given pool to a specific set of peers by adding an optional list of BGP peers.
BGP peers are commonly network routers that are configured to use the BGP protocol.
When a router receives traffic for the load balancer IP address, the router picks one of the nodes with a speaker
pod that advertised the IP address.
The router sends the traffic to that node.
After traffic enters the node, the service proxy for the CNI network provider distributes the traffic to all the pods for the service.
The directly-connected router on the same layer 2 network segment as the cluster nodes can be configured as a BGP peer.
If the directly-connected router is not configured as a BGP peer, you need to configure your network so that packets for load balancer IP addresses are routed between the BGP peers and the cluster nodes that run the speaker
pods.
Each time a router receives new traffic for the load balancer IP address, it creates a new connection to a node. Each router manufacturer has an implementation-specific algorithm for choosing which node to initiate the connection with. However, the algorithms commonly are designed to distribute traffic across the available nodes for the purpose of balancing the network load.
If a node becomes unavailable, the router initiates a new connection with another node that has a speaker
pod that advertises the load balancer IP address.