1. Documentation
×

MetalLB logging, troubleshooting, and support

Setting the MetalLB logging levels

MetalLB uses FRRouting (FRR) in a container with the default setting of info generates a lot of logging. You can control the verbosity of the logs generated by setting the logLevel as illustrated in this example.

Gain a deeper insight into MetalLB by setting the logLevel to debug as follows:

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Create a file, such as setdebugloglevel.yaml, with content such as the following example:

    apiVersion: metallb.io/v1beta1
kind: MetalLB
metadata:
  name: metallb
  namespace: metallb-system
spec:
  logLevel: debug
  nodeSelector:
    node-role.kubernetes.io/worker: ""

    While other fields like speakerConfig can be configured here, the bgpBackend field must be omitted. It is reserved for internal use by MetalLB to manage the active BGP implementation.

  2. Apply the configuration:

    $ oc replace -f setdebugloglevel.yaml

    Use oc replace as the understanding is the metallb CR is already created and here you are changing the log level.

  3. Display the names of the speaker pods:

    $ oc get -n metallb-system pods -l component=speaker
    Example output
    NAME                    READY   STATUS    RESTARTS   AGE
speaker-2m9pm           4/4     Running   0          9m19s
speaker-7m4qw           3/4     Running   0          19s
speaker-szlmx           4/4     Running   0          9m19s

    Speaker and controller pods are recreated to ensure the updated logging level is applied. The logging level is modified for all the components of MetalLB.

  4. View the speaker logs:

    $ oc logs -n metallb-system speaker-7m4qw -c speaker
    Example output
    {"branch":"main","caller":"main.go:92","commit":"3d052535","goversion":"gc / go1.17.1 / amd64","level":"info","msg":"MetalLB speaker starting (commit 3d052535, branch main)","ts":"2022-05-17T09:55:05Z","version":""}
{"caller":"announcer.go:110","event":"createARPResponder","interface":"ens4","level":"info","msg":"created ARP responder for interface","ts":"2022-05-17T09:55:05Z"}
{"caller":"announcer.go:119","event":"createNDPResponder","interface":"ens4","level":"info","msg":"created NDP responder for interface","ts":"2022-05-17T09:55:05Z"}
{"caller":"announcer.go:110","event":"createARPResponder","interface":"tun0","level":"info","msg":"created ARP responder for interface","ts":"2022-05-17T09:55:05Z"}
{"caller":"announcer.go:119","event":"createNDPResponder","interface":"tun0","level":"info","msg":"created NDP responder for interface","ts":"2022-05-17T09:55:05Z"}
I0517 09:55:06.515686      95 request.go:665] Waited for 1.026500832s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v1alpha1?timeout=32s
{"Starting Manager":"(MISSING)","caller":"k8s.go:389","level":"info","ts":"2022-05-17T09:55:08Z"}
{"caller":"speakerlist.go:310","level":"info","msg":"node event - forcing sync","node addr":"10.0.128.4","node event":"NodeJoin","node name":"ci-ln-qb8t3mb-72292-7s7rh-worker-a-vvznj","ts":"2022-05-17T09:55:08Z"}
{"caller":"service_controller.go:113","controller":"ServiceReconciler","enqueueing":"openshift-kube-controller-manager-operator/metrics","epslice":"{\"metadata\":{\"name\":\"metrics-xtsxr\",\"generateName\":\"metrics-\",\"namespace\":\"openshift-kube-controller-manager-operator\",\"uid\":\"ac6766d7-8504-492c-9d1e-4ae8897990ad\",\"resourceVersion\":\"9041\",\"generation\":4,\"creationTimestamp\":\"2022-05-17T07:16:53Z\",\"labels\":{\"app\":\"kube-controller-manager-operator\",\"endpointslice.kubernetes.io/managed-by\":\"endpointslice-controller.k8s.io\",\"kubernetes.io/service-name\":\"metrics\"},\"annotations\":{\"endpoints.kubernetes.io/last-change-trigger-time\":\"2022-05-17T07:21:34Z\"},\"ownerReferences\":[{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"name\":\"metrics\",\"uid\":\"0518eed3-6152-42be-b566-0bd00a60faf8\",\"controller\":true,\"blockOwnerDeletion\":true}],\"managedFields\":[{\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"apiVersion\":\"discovery.k8s.io/v1\",\"time\":\"2022-05-17T07:20:02Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:addressType\":{},\"f:endpoints\":{},\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:endpoints.kubernetes.io/last-change-trigger-time\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:app\":{},\"f:endpointslice.kubernetes.io/managed-by\":{},\"f:kubernetes.io/service-name\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"0518eed3-6152-42be-b566-0bd00a60faf8\\\"}\":{}}},\"f:ports\":{}}}]},\"addressType\":\"IPv4\",\"endpoints\":[{\"addresses\":[\"10.129.0.7\"],\"conditions\":{\"ready\":true,\"serving\":true,\"terminating\":false},\"targetRef\":{\"kind\":\"Pod\",\"namespace\":\"openshift-kube-controller-manager-operator\",\"name\":\"kube-controller-manager-operator-6b98b89ddd-8d4nf\",\"uid\":\"dd5139b8-e41c-4946-a31b-1a629314e844\",\"resourceVersion\":\"9038\"},\"nodeName\":\"ci-ln-qb8t3mb-72292-7s7rh-master-0\",\"zone\":\"us-central1-a\"}],\"ports\":[{\"name\":\"https\",\"protocol\":\"TCP\",\"port\":8443}]}","level":"debug","ts":"2022-05-17T09:55:08Z"}

  5. List the FRR-K8s pods:

    $ oc get -n openshift-frr-k8s pods
    Example output
    NAME                                      READY   STATUS    RESTARTS   AGE
frr-k8s-bz2dn                            7/7     Running   0          4h
frr-k8s-statuscleaner-59cf6f5d44-9wkfr   1/1     Running   0          4h

  6. View the FRR logs by specifying the frr container in one of the frr-k8s pods:

    $ oc logs -n openshift-frr-k8s <frr_k8s_pod_name> -c frr
    Example output
    2026/03/02 09:53:09 WATCHFRR: [T83RR-8SM5G] watchfrr 8.5.3 starting: vty@0
2026/03/02 09:53:09 WATCHFRR: [ZCJ3S-SPH5S] zebra state -> down : initial connection attempt failed
2026/03/02 09:53:09 WATCHFRR: [ZCJ3S-SPH5S] bgpd state -> down : initial connection attempt failed
2026/03/02 09:53:09 WATCHFRR: [ZCJ3S-SPH5S] staticd state -> down : initial connection attempt failed
2026/03/02 09:53:09 WATCHFRR: [ZCJ3S-SPH5S] bfdd state -> down : initial connection attempt failed
2026/03/02 09:53:09 ZEBRA: [NNACN-54BDA][EC 4043309110] Disabling MPLS support (no kernel support)
2026/03/02 09:53:09 WATCHFRR: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2026/03/02 09:53:09 WATCHFRR: [QDG3Y-BY5TN] zebra state -> up : connect succeeded
2026/03/02 09:53:09 WATCHFRR: [QDG3Y-BY5TN] bgpd state -> up : connect succeeded
2026/03/02 09:53:09 WATCHFRR: [QDG3Y-BY5TN] staticd state -> up : connect succeeded
2026/03/02 09:53:09 WATCHFRR: [QDG3Y-BY5TN] bfdd state -> up : connect succeeded
2026/03/02 09:53:09 WATCHFRR: [KWE5Q-QNGFC] all daemons up, doing startup-complete notify
2026/03/02 09:53:09 ZEBRA: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00
2026/03/02 09:53:09 BGP: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00

FRRouting (FRR) log levels

The following table describes the FRR logging levels.

Table 1. Log levels
Log level Description

all

Supplies all logging information for all logging levels.

debug

Information that is diagnostically helpful to people. Set to debug to give detailed troubleshooting information.

info

Provides information that always should be logged but under normal circumstances does not require user intervention. This is the default logging level.

warn

Anything that can potentially cause inconsistent MetalLB behaviour. Usually MetalLB automatically recovers from this type of error.

error

Any unrecoverable error in MetalLB. These errors usually require administrator intervention to fix.

none

Turn off all logging.

Troubleshooting BGP issues

As a cluster administrator, if you need to troubleshoot BGP configuration issues, you need to run commands in the FRR container.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. In OKD 4.17 and later, MetalLB uses FRR-K8s as the default backend for BGP. The MetalLB Operator deploys the FRR-K8s daemon set in the metallb-system namespace. Display the names of the frr-k8s pods by running the following command:

    $ oc get pods -n metallb-system -l app=frr-k8s
    Example output
    NAME                                     READY   STATUS    RESTARTS   AGE
frr-k8s-bz2dn                            7/7     Running   0          15m
frr-k8s-statuscleaner-59cf6f5d44-9wkfr   1/1     Running   0          15m

  2. Display the running configuration for FRR by running the following command:

    $ oc exec -n metallb-system <frr-k8s-pod> -c frr -- vtysh -c "show running-config"
    Example output
    Building configuration...

Current configuration:
!
frr version 8.5.3
frr defaults traditional
hostname mysno-sno.demo.lab
log file /etc/frr/frr.log informational
log timestamp precision 3
no ip forwarding
no ipv6 forwarding
service integrated-vtysh-config
!
router bgp 64500
 bgp router-id 10.0.1.2
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 bgp graceful-restart preserve-fw-state
 no bgp network import-check
 neighbor 10.0.2.3 remote-as 64500
 neighbor 10.0.2.3 bfd profile doc-example-bfd-profile-full
 neighbor 10.0.2.3 timers 5 15
 neighbor 10.0.2.4 remote-as 64500
 neighbor 10.0.2.4 bfd profile doc-example-bfd-profile-full
 neighbor 10.0.2.4 timers 5 15
 !
 address-family ipv4 unicast
  network 203.0.113.200/30
  neighbor 10.0.2.3 activate
  neighbor 10.0.2.3 route-map 10.0.2.3-in in
  neighbor 10.0.2.4 activate
  neighbor 10.0.2.4 route-map 10.0.2.4-in in
 exit-address-family
 !
 address-family ipv6 unicast
  network fc00:f853:ccd:e799::/124
  neighbor 10.0.2.3 activate
  neighbor 10.0.2.3 route-map 10.0.2.3-in in
  neighbor 10.0.2.4 activate
  neighbor 10.0.2.4 route-map 10.0.2.4-in in
 exit-address-family
exit
!
ip prefix-list 192.168.122.12-inpl-ipv4 seq 1 deny any
ip prefix-list 192.168.122.12-allowed-ipv4 seq 1 permit 192.168.122.210/32
!
ipv6 prefix-list 192.168.122.12-allowed-ipv6 seq 1 deny any
ipv6 prefix-list 192.168.122.12-inpl-ipv4 seq 2 deny any
!
route-map 192.168.122.12-out permit 1
 match ip address prefix-list 192.168.122.12-allowed-ipv4
exit
!
route-map 192.168.122.12-out permit 2
 match ipv6 address prefix-list 192.168.122.12-allowed-ipv6
exit
!
route-map 192.168.122.12-in permit 3
 match ip address prefix-list 192.168.122.12-inpl-ipv4
exit
!
route-map 192.168.122.12-in permit 4
 match ipv6 address prefix-list 192.168.122.12-inpl-ipv4
exit
!
ip nht resolve-via-default
!
ipv6 nht resolve-via-default
!
end

    where:

    router bgp 64500

    The router bgp section indicates the ASN for MetalLB.

    neighbor 10.0.2.3 remote-as 64500

    Confirm that a neighbor <ip-address> remote-as <peer-ASN> line exists for each BGP peer custom resource that you added.

    bfd profile doc-example-bfd-profile-full

    If you configured BFD, confirm that the BFD profile is associated with the correct BGP peer and that the BFD profile appears in the command output.

    network 203.0.113.200/30

    Confirm that the network <ip-address-range> lines match the IP address ranges that you specified in address pool custom resources that you added.

  3. Display the BGP summary by running the following command:

    $ oc exec -n metallb-system <frr-k8s-pod> -c frr -- vtysh -c "show bgp summary"
    Example output
    IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.122.12, local AS number 64501 vrf-id 0
BGP table version 1
RIB entries 1, using 192 bytes of memory
Peers 1, using 725 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
10.0.2.3        4      64500       387       389        0    0    0 00:32:02            0        1
10.0.2.4        4      64500         0         0        0    0    0    never       Active        0

Total number of neighbors 1

    where:

    10.0.2.3

    Confirm that the output includes a line for each BGP peer custom resource that you added.

    10.0.2.4

    Output that shows 0 messages received and messages sent indicates a BGP peer that does not have a BGP session. Check network connectivity and the BGP configuration of the BGP peer.

  4. Display the BGP peers that received an address pool by running the following command:

    $ oc exec -n metallb-system <frr-k8s-pod> -c frr -- vtysh -c "show bgp ipv4 unicast"

    Replace ipv4 with ipv6 to display the BGP peers that received an IPv6 address pool.

    Example output
    BGP routing table entry for 203.0.113.200/30
Paths: (1 available, best #1, table default)
  Advertised to non peer-group peers:
  10.0.2.3
  Local
    0.0.0.0 from 0.0.0.0 (10.0.1.2)
      Origin IGP, metric 0, weight 32768, valid, sourced, local, best (First path received)
      Last update: Mon Jan 10 19:49:07 2022

    where:

    10.0.2.3

    Confirm that the output includes an IP address for a BGP peer.

Troubleshooting BFD issues

The Bidirectional Forwarding Detection (BFD) implementation that Red Hat supports uses FRRouting (FRR) in a container that exists in an frr-k8s pod in the metallb-system namespace. The BFD implementation relies on BFD peers also being configured as BGP peers with an established BGP session. As a cluster administrator, if you need to troubleshoot BFD configuration issues, you need to run commands in the FRR container.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

  • You have installed the OpenShift CLI (oc).

Procedure

  1. Display the names of the FRR-K8s pods by running the following command:

    $ oc get pods -n metallb-system -l app=frr-k8s
    Example output
    NAME            READY   STATUS    RESTARTS   AGE
frr-k8s-bz2dn   7/7     Running   0          106m

  2. Run the following command against the frr container in the FRR-K8s pod to display the BFD peers:

    $ oc exec -n metallb-system <frr_k8s_pod_name> -c frr -- vtysh -c "show bfd peers brief"
    Example output
    Session count: 1
SessionId  LocalAddress              PeerAddress              Status
=========  ============              ===========              ======
3909139637 10.0.1.2                  10.0.2.3                 up

    where:

    up

    Specifies that the PeerAddress column includes each BFD peer. If the output does not list a BFD peer IP address that you expected the output to include, troubleshoot BGP connectivity with the peer. If the status field indicates down, check for connectivity on the links and equipment between the node and the peer. You can determine the node name for the frr-k8s pod with a command like oc get pods -n metallb-system <frr-k8s-pod> -o jsonpath='{.spec.nodeName}'.

MetalLB metrics for BGP and BFD

OKD captures the following Prometheus metrics for MetalLB that relate to BGP peers and BFD profiles.

Table 2. MetalLB BFD metrics
Name Description

frrk8s_bfd_control_packet_input

Counts the number of BFD control packets received from each BFD peer.

frrk8s_bfd_control_packet_output

Counts the number of BFD control packets sent to each BFD peer.

frrk8s_bfd_echo_packet_input

Counts the number of BFD echo packets received from each BFD peer.

frrk8s_bfd_echo_packet_output

Counts the number of BFD echo packets sent to each BFD.

frrk8s_bfd_session_down_events

Counts the number of times the BFD session with a peer entered the down state.

frrk8s_bfd_session_up

Indicates the connection state with a BFD peer. 1 indicates the session is up and 0 indicates the session is down.

frrk8s_bfd_session_up_events

Counts the number of times the BFD session with a peer entered the up state.

frrk8s_bfd_zebra_notifications

Counts the number of BFD Zebra notifications for each BFD peer.
Table 3. MetalLB BGP metrics
Name Description

frrk8s_bgp_announced_prefixes_total

Counts the number of load balancer IP address prefixes that are advertised to BGP peers. The terms prefix and aggregated route have the same meaning.

frrk8s_bgp_session_up

Indicates the connection state with a BGP peer. 1 indicates the session is up and 0 indicates the session is down.

frrk8s_bgp_updates_total

Counts the number of BGP update messages sent to each BGP peer.

frrk8s_bgp_opens_sent

Counts the number of BGP open messages sent to each BGP peer.

frrk8s_bgp_opens_received

Counts the number of BGP open messages received from each BGP peer.

frrk8s_bgp_notifications_sent

Counts the number of BGP notification messages sent to each BGP peer.

frrk8s_bgp_updates_total_received

Counts the number of BGP update messages received from each BGP peer.

frrk8s_bgp_keepalives_sent

Counts the number of BGP keepalive messages sent to each BGP peer.

frrk8s_bgp_keepalives_received

Counts the number of BGP keepalive messages received from each BGP peer.

frrk8s_bgp_route_refresh_sent

Counts the number of BGP route refresh messages sent to each BGP peer.

frrk8s_bgp_total_sent

Counts the number of total BGP messages sent to each BGP peer.

frrk8s_bgp_total_received

Counts the number of total BGP messages received from each BGP peer.

frrk8s_bgp_received_prefixes_total

Counts the number of load balancer IP address prefixes received from each BGP peer.

About collecting MetalLB data

You can use the oc adm must-gather CLI command to collect information about your cluster, your MetalLB configuration, and the MetalLB Operator. The following features and objects are associated with MetalLB and the MetalLB Operator:

  • The namespace and child objects that the MetalLB Operator is deployed in

  • All MetalLB Operator custom resource definitions (CRDs)

The oc adm must-gather CLI command collects the following information from FRRouting (FRR) that Red Hat uses to implement BGP and BFD:

  • /etc/frr/frr.conf

  • /etc/frr/frr.log

  • /etc/frr/daemons configuration file

  • /etc/frr/vtysh.conf

The log and configuration files in the preceding list are collected from the frr container in each frr-k8s pod in the metallb-system namespace.

In addition to the log and configuration files, the oc adm must-gather CLI command collects the output from the following vtysh commands:

  • show running-config

  • show bgp ipv4

  • show bgp ipv6

  • show bgp neighbor

  • show bfd peer

No additional configuration is required when you run the oc adm must-gather CLI command.

Additional resources