This topic contains steps to roll out Dynatrace OneAgent to all of the OKD cluster nodes to allow for monitoring all applications running within the OKD cluster as well as the cluster nodes itselfes.

The instructions provided in this guideline require access to the enterprise-ready dynatrace/oneagent image from the Red Hat Container Catalog.

Locate Dynatrace OneAgent installer URL

The first step is to obtain the location for ONEAGENT_INSTALLER_SCRIPT_URL. This information is presented to you during Dynatrace OneAgent installation.

Procedure

To get your ONEAGENT_INSTALLER_SCRIPT_URL

  1. Select Deploy Dynatrace from the navigation menu.

  2. Click Start installation and select Linux.

  3. Copy the complete URL from the wget command. This is your ONEAGENT_INSTALLER_SCRIPT_URL.

Installation

The following template uses a Dynatrace OneAgent image with a DaemonSet to install Dynatrace OneAgent for full-stack monitoring on each node of an OKD cluster.

Please note that enforcing the below dynatrace-oneagent.yml template requires a service account named dynatrace that can create privileged pods. See below for detailed instructions.

Procedure
  1. Log into your OKD cluster as system:admin

    $ oc login -u system:admin
  2. Select an OKD project to run the Dynatrace OneAgent image

    $ oc project openshift-infra
  3. In this project, create a service account named dynatrace

    $ oc create serviceaccount dynatrace
  4. Allow the dynatrace service account to pull images from the RHCC via registry.connect.redhat.com. Be sure to replace [username], [password] and [email] with your Red Hat Customer Portal’s account credentials

    $ oc secrets new-dockercfg rhcc \
        --docker-server=registry.connect.redhat.com \
        --docker-username=[username] \
        --docker-password=[password] \
        --docker-email=[email]
    $ oc secrets link dynatrace rhcc --for=pull
  5. Grant the dynatrace service account permissions to run Dynatrace OneAgent as a privileged container

    $ oc adm policy add-scc-to-user privileged -z dynatrace
  6. Deploy Dynatrace OneAgent using the dynatrace-oneagent.yml template. Replace [oneagent-installer-script-url] with your actual OneAgent installer script url. Be sure to enclose the URL with quotation marks. Otherwise the URL will break the commands

    The dynatrace-oneagent.yml template for OKD:

    kind: Template
    apiVersion: v1
    name: dynatrace-oneagent
    labels:
      template: dynatrace-oneagent
    metadata:
      name: dynatrace-oneagent
      annotations:
        openshift.io/display-name: Dynatrace OneAgent
        description: Installs Dynatrace OneAgent for all-in-one, full-stack monitoring of OpenShift with Dynatrace. Requires privileged access.
    objects:
      - apiVersion: extensions/v1beta1
        kind: DaemonSet
        metadata:
          name: dynatrace-oneagent
        spec:
          template:
            metadata:
              labels:
                name: dynatrace-oneagent
            spec:
              containers:
              - name: dynatrace-oneagent
                image: registry.connect.redhat.com/dynatrace/oneagent
                imagePullPolicy: Always
                env:
                - name: ONEAGENT_INSTALLER_SCRIPT_URL
                  value: "${ONEAGENT_INSTALLER_SCRIPT_URL}"
                - name: ONEAGENT_INSTALLER_SKIP_CERT_CHECK
                  value: "${ONEAGENT_INSTALLER_SKIP_CERT_CHECK}"
                args:
                - "APP_LOG_CONTENT_ACCESS=1"
                volumeMounts:
                - name: host-root
                  mountPath: /mnt/root
                securityContext:
                  privileged: true
              volumes:
              - name: host-root
                hostPath:
                  path: /
              hostIPC: true
              hostNetwork: true
              hostPID: true
              serviceAccountName: dynatrace
    parameters:
      - name: ONEAGENT_INSTALLER_SCRIPT_URL
        description: "A URL that points to your cluster's OneAgent download location (Select \"Deploy Dynatrace\" from the Dynatrace navigation menu to access your URL). Example: https://EnvironmentID.live.dynatrace.com/installer/oneagent/unix/latest/AbCdEfGhIjKlMnOp."
        required: true
      - name: ONEAGENT_INSTALLER_SKIP_CERT_CHECK
        description: "Must be true if the SSL certificate check upon OneAgent download will be omitted, otherwise false (default). If you're using a Dynatrace Managed cluster with a self-signed certificate, set this to true."
        value: "false"
        required: false
    $ oc process -f dynatrace-oneagent.yml ONEAGENT_INSTALLER_SCRIPT_URL="[oneagent-installer-script-url]" | oc create -f -
  7. Verify that the dynatrace-oneagent daemon set has been created successfully

    $ oc status
    In project openshift-infra on server https://127.0.0.1:8443
    
    pod/dynatrace-oneagent-abcde runs dynatrace/oneagent

    Check if dynatrace-oneagent pods are running

    $ oc get pods
    NAME                       READY     STATUS              RESTARTS   AGE
    dynatrace-oneagent-abcde   1/1       Running             0          1m

    Check logs from dynatrace-oneagent pod

    $ oc logs -f dynatrace-oneagent-abcde
    09:46:18 Deploying agent to /tmp/Dynatrace-OneAgent-Linux.sh via https://EnvironmentID.live.dynatrace.com/installer/oneagent/unix/latest/AbCdEfGhIjKlMnOp
    ...
    09:46:24 Validating agent installer in /tmp/Dynatrace-OneAgent-Linux.sh
    Verification successful
    09:46:24 Started agent deployment as container image, PID 1234.
    09:46:24 Container version: 1.x
    09:46:24 Checking root privileges...
    09:46:24 OK
    09:46:27 Installation started, version 1.x, build date: 01.01.2017, PID 1234.
    ...

For OKD versions 3.7 and higher, the Red Hat Container Catalog may not accept the auto-generated dockercfg secret type (because of BZ#1476330). Therefore, you must create a generic file-based secret using the generated file from a docker login command.

Uninstallation

Uninstalling Dynatrace OneAgent from each node of an OKD cluster can be achieved as follows.

Procedure
  1. Select the project that runs the dynatrace-oneagent daemon set

    $ oc project openshift-infra
  2. Delete the dynatrace-oneagent daemon set:

    $ oc delete ds/dynatrace-oneagent

Updating

Whenever a new version of Dynatrace OneAgent becomes available in Dynatrace, re-deploy Dynatrace OneAgent as explained in the steps below. The dynatrace/oneagent image will automatically fetch the latest version of Dynatrace OneAgent. If you’ve specified a default OneAgent install version for new hosts and applications in the OneAgent updates settings, the dynatrace/oneagent image will automatically fetch the defined default version of Dynatrace OneAgent.

Procedure
  1. Delete the dynatrace-oneagent daemon set:

    $ oc delete ds/dynatrace-oneagent
  2. Deploy Dynatrace OneAgent using the dynatrace-oneagent.yml template from the installation section. Be sure to replace [oneagent-installer-script-url] with an appropriate download location

    $ oc process -f dynatrace-oneagent.yml ONEAGENT_INSTALLER_SCRIPT_URL="[oneagent-installer-script-url]" | oc create -f -
    daemonset "dynatrace-oneagent" created

    Please note that quotes are needed to protect special shell characters within the Dynatrace OneAgent installer URL.

Troubleshooting

Find out how to solve problems that you may encounter when deploying OneAgent on an OKD. The first step is to obtain the location for ONEAGENT_INSTALLER_SCRIPT_URL. This information is presented to you during Dynatrace OneAgent installation.

Deployment seems successful, however the dynatrace/oneagent image can’t be pulled
$ oc get pods
NAME                       READY   STATUS         RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ErrImagePull   0          3s
$ oc logs -f dynatrace-oneagent-abcde
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: image can't be pulled

This is typically the case if the dynatrace service account hasn’t been allowed to pull images from the RHCC (please see the installation steps).

Deployment seems successful, but the dynatrace-oneagent container doesn’t produce meaningful logs
$ oc get pods
NAME                       READY   STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ContainerCreating   0          3s
$ oc logs -f dynatrace-oneagent-abcde
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: ContainerCreating

This is typically the case if the container hasn’t yet fully started. Simply wait a few more seconds.

Deployment seems successful, but the dynatrace-oneagent container isn’t running
$ oc process -f dynatrace-oneagent.yml ONEAGENT_INSTALLER_SCRIPT_URL="[oneagent-installer-script-url]" | oc create -f -
daemonset "dynatrace-oneagent" created

Please note that quotes are needed to protect the special shell characters in the Dynatrace OneAgent installer URL.

$ oc get pods
No resources found.

This is typically the case if the dynatrace service account hasn’t been configured to run privileged pods (please see the installation steps):

$ oc describe ds/dynatrace-oneagent
Name:   dynatrace-oneagent
Image(s): dynatrace/oneagent
Selector: name=dynatrace-oneagent
Node-Selector:  <none>
Labels:   template=dynatrace-oneagent
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Events:
  FirstSeen LastSeen  Count From    SubObjectPath Type    Reason    Message
  --------- --------  ----- ----    ------------- --------  ------    -------
  6m    3m    17  {daemon-set }     Warning   FailedCreate  Error creating: pods "dynatrace-oneagent-" is forbidden: unable to validate against any security context constraint: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[0].securityContext.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.containers[0].securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used]
Deployment was successful, but monitoring data isn’t available in Dynatrace
$ oc get pods
NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m

This is typically caused by a timing issue that occurs when application containers are started before Dynatrace OneAgent is fully installed on the system. As a consequence, some parts of your application may be uninstrumented. To be on the safe side, Dynatrace OneAgent should be fully installed and configured before you start your application containers. If your application is already running, restart its containers to achieve the same outcome.

If you plan to install Dynatrace OneAgent on more than 50 hosts, please consider serving the installer script via a dedicated server, such as Amazon S3. Otherwise, with more than 50 concurrent connections, Dynatrace Server may throttle requests.

Limitations

The same limitations apply as when deploying OneAgent as a container.