1. Documentation
×

Monitoring bare-metal events with the Bare Metal Event Relay

Bare Metal Event Relay is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

About bare-metal events

Use the Bare Metal Event Relay to subscribe applications that run in your OKD cluster to events that are generated on the underlying bare-metal host. The Redfish service publishes events on a node and transmits them on an advanced message queue to subscribed applications.

Bare-metal events are based on the open Redfish standard that is developed under the guidance of the Distributed Management Task Force (DMTF). Redfish provides a secure industry-standard protocol with a REST API. The protocol is used for the management of distributed, converged or software-defined resources and infrastructure.

Hardware-related events published through Redfish includes:

  • Breaches of temperature limits

  • Server status

  • Fan status

Begin using bare-metal events by deploying the Bare Metal Event Relay Operator and subscribing your application to the service. The Bare Metal Event Relay Operator installs and manages the lifecycle of the Redfish bare-metal event service.

The Bare Metal Event Relay works only with Redfish-capable devices on single-node clusters provisioned on bare-metal infrastructure.

How bare-metal events work

The Bare Metal Event Relay enables applications running on bare-metal clusters to respond quickly to Redfish hardware changes and failures such as breaches of temperature thresholds, fan failure, disk loss, power outages, and memory failure. These hardware events are delivered over a reliable low-latency transport channel based on Advanced Message Queuing Protocol (AMQP). The latency of the messaging service is between 10 to 20 milliseconds.

The Bare Metal Event Relay provides a publish-subscribe service for the hardware events, where multiple applications can use REST APIs to subscribe and consume the events. The Bare Metal Event Relay supports hardware that complies with Redfish OpenAPI v1.8 or higher.

Bare Metal Event Relay data flow

The following figure illustrates an example of bare-metal events data flow:

Bare-metal events data flow
Figure 1. Bare Metal Event Relay data flow

Operator-managed pod

The Operator uses custom resources to manage the pod containing the Bare Metal Event Relay and its components using the HardwareEvent CR.

Bare Metal Event Relay

At startup, the Bare Metal Event Relay queries the Redfish API and downloads all the message registries, including custom registries. The Bare Metal Event Relay then begins to receive subscribed events from the Redfish hardware.

The Bare Metal Event Relay enables applications running on bare-metal clusters to respond quickly to Redfish hardware changes and failures such as breaches of temperature thresholds, fan failure, disk loss, power outages, and memory failure. The events are reported using the HardwareEvent CR.

Cloud native event

Cloud native events (CNE) is a REST API specification for defining the format of event data.

CNCF CloudEvents

CloudEvents is a vendor-neutral specification developed by the Cloud Native Computing Foundation (CNCF) for defining the format of event data.

AMQP dispatch router

The dispatch router is responsible for the message delivery service between publisher and subscriber. AMQP 1.0 qpid is an open standard that supports reliable, high-performance, fully-symmetrical messaging over the internet.

Cloud event proxy sidecar

The cloud event proxy sidecar container image is based on the ORAN API specification and provides a publish-subscribe event framework for hardware events.

Redfish message parsing service

In addition to handling Redfish events, the Bare Metal Event Relay provides message parsing for events without a Message property. The proxy downloads all the Redfish message registries including vendor specific registries from the hardware when it starts. If an event does not contain a Message property, the proxy uses the Redfish message registries to construct the Message and Resolution properties and add them to the event before passing the event to the cloud events framework. This service allows Redfish events to have smaller message size and lower transmission latency.

Installing the Bare Metal Event Relay using the CLI

As a cluster administrator, you can install the Bare Metal Event Relay Operator by using the CLI.

Prerequisites

  • A cluster that is installed on bare-metal hardware with nodes that have a RedFish-enabled Baseboard Management Controller (BMC).

  • Install the OpenShift CLI (oc).

  • Log in as a user with cluster-admin privileges.

Procedure

  1. Create a namespace for the Bare Metal Event Relay.

    1. Save the following YAML in the bare-metal-events-namespace.yaml file:

      apiVersion: v1
kind: Namespace
metadata:
  name: openshift-bare-metal-events
  labels:
    name: openshift-bare-metal-events
    openshift.io/cluster-monitoring: "true"

    2. Create the Namespace CR:

      $ oc create -f bare-metal-events-namespace.yaml

  2. Create an Operator group for the Bare Metal Event Relay Operator.

    1. Save the following YAML in the bare-metal-events-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: bare-metal-event-relay-group
  namespace: openshift-bare-metal-events
spec:
  targetNamespaces:
  - openshift-bare-metal-events

    2. Create the OperatorGroup CR:

      $ oc create -f bare-metal-events-operatorgroup.yaml

  3. Subscribe to the Bare Metal Event Relay.

    1. Save the following YAML in the bare-metal-events-sub.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: bare-metal-event-relay-subscription
  namespace: openshift-bare-metal-events
spec:
  channel: "stable"
  name: bare-metal-event-relay
  source: redhat-operators
  sourceNamespace: openshift-marketplace

    2. Create the Subscription CR:

      $ oc create -f bare-metal-events-sub.yaml
Verification

To verify that the Bare Metal Event Relay Operator is installed, run the following command:

$ oc get csv -n openshift-bare-metal-events -o custom-columns=Name:.metadata.name,Phase:.status.phase
Example output
Name                                                          Phase
bare-metal-event-relay.4.12.0-xxxxxxxxxxxx            Succeeded

Installing the Bare Metal Event Relay using the web console

As a cluster administrator, you can install the Bare Metal Event Relay Operator using the web console.

Prerequisites

  • A cluster that is installed on bare-metal hardware with nodes that have a RedFish-enabled Baseboard Management Controller (BMC).

  • Log in as a user with cluster-admin privileges.

Procedure

  1. Install the Bare Metal Event Relay using the OKD web console:

    1. In the OKD web console, click Operators → OperatorHub.

    2. Choose Bare Metal Event Relay from the list of available Operators, and then click Install.

    3. On the Install Operator page, select or create a Namespace, select openshift-bare-metal-events, and then click Install.

Verification

Optional: You can verify that the Operator installed successfully by performing the following check:

  1. Switch to the Operators → Installed Operators page.

  2. Ensure that Bare Metal Event Relay is listed in the project with a Status of InstallSucceeded.

    During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

If the operator does not appear as installed, to troubleshoot further:

  • Go to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.

  • Go to the Workloads → Pods page and check the logs for pods in the project namespace.

Installing the AMQ messaging bus

To pass Redfish bare-metal event notifications between publisher and subscriber on a node, you must install and configure an AMQ messaging bus to run locally on the node. You do this by installing the AMQ Interconnect Operator for use in the cluster.

Prerequisites

  • Install the OKD CLI (oc).

  • Log in as a user with cluster-admin privileges.

Procedure
Verification

  1. Verify that the AMQ Interconnect Operator is available and the required pods are running:

    $ oc get pods -n amq-interconnect
    Example output
    NAME                                    READY   STATUS    RESTARTS   AGE
amq-interconnect-645db76c76-k8ghs       1/1     Running   0          23h
interconnect-operator-5cb5fc7cc-4v7qm   1/1     Running   0          23h

  2. Verify that the required bare-metal-event-relay bare-metal event producer pod is running in the openshift-bare-metal-events namespace:

    $ oc get pods -n openshift-bare-metal-events
    Example output
    NAME                                                            READY   STATUS    RESTARTS   AGE
hw-event-proxy-operator-controller-manager-74d5649b7c-dzgtl     2/2     Running   0          25s

Subscribing to Redfish BMC bare-metal events for a cluster node

As a cluster administrator, you can subscribe to Redfish BMC events generated on a node in your cluster by creating a BMCEventSubscription custom resource (CR) for the node, creating a HardwareEvent CR for the event, and a Secret CR for the BMC.

Subscribing to bare-metal events

You can configure the baseboard management controller (BMC) to send bare-metal events to subscribed applications running in an OKD cluster. Example Redfish bare-metal events include an increase in device temperature, or removal of a device. You subscribe applications to bare-metal events using a REST API.

You can only create a BMCEventSubscription custom resource (CR) for physical hardware that supports Redfish and has a vendor interface set to redfish or idrac-redfish.

Use the BMCEventSubscription CR to subscribe to predefined Redfish events. The Redfish standard does not provide an option to create specific alerts and thresholds. For example, to receive an alert event when an enclosure’s temperature exceeds 40° Celsius, you must manually configure the event according to the vendor’s recommendations.

Perform the following procedure to subscribe to bare-metal events for the node using a BMCEventSubscription CR.

Prerequisites

  • Install the OpenShift CLI (oc).

  • Log in as a user with cluster-admin privileges.

  • Get the user name and password for the BMC.

  • Deploy a bare-metal node with a Redfish-enabled Baseboard Management Controller (BMC) in your cluster, and enable Redfish events on the BMC.

    Enabling Redfish events on specific hardware is outside the scope of this information. For more information about enabling Redfish events for your specific hardware, consult the BMC manufacturer documentation.
Procedure

  1. Confirm that the node hardware has the Redfish EventService enabled by running the following curl command:

    curl https://<bmc_ip_address>/redfish/v1/EventService --insecure -H 'Content-Type: application/json' -u "<bmc_username>:<password>"

    where:

    bmc_ip_address

    is the IP address of the BMC where the Redfish events are generated.

    Example output
    {
   "@odata.context": "/redfish/v1/$metadata#EventService.EventService",
   "@odata.id": "/redfish/v1/EventService",
   "@odata.type": "#EventService.v1_0_2.EventService",
   "Actions": {
      "#EventService.SubmitTestEvent": {
         "EventType@Redfish.AllowableValues": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
         "target": "/redfish/v1/EventService/Actions/EventService.SubmitTestEvent"
      }
   },
   "DeliveryRetryAttempts": 3,
   "DeliveryRetryIntervalSeconds": 30,
   "Description": "Event Service represents the properties for the service",
   "EventTypesForSubscription": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
   "EventTypesForSubscription@odata.count": 5,
   "Id": "EventService",
   "Name": "Event Service",
   "ServiceEnabled": true,
   "Status": {
      "Health": "OK",
      "HealthRollup": "OK",
      "State": "Enabled"
   },
   "Subscriptions": {
      "@odata.id": "/redfish/v1/EventService/Subscriptions"
   }
}

  2. Get the Bare Metal Event Relay service route for the cluster by running the following command:

    $ oc get route -n openshift-bare-metal-events
    Example output