Soon, all Red Hat OpenShift documentation will only be available on docs.redhat.com, the official location of all Red Hat product documentation. Get familiar with the updated experience today!
  1. Documentation
×

Writing APBs: Getting Started

Overview

In this tutorial, you will walk through the creation of some sample Ansible Playbook Bundles (APBs). You will create actions for them to allow provision, deprovision, bind, and unbind. You can find more information about the design of APBs in the Design topic. More in-depth information about writing APBs is available in the Reference topic.

For the remainder of this tutorial, substitute your own information for items marked in brackets; for example, <host>:<port> might need to be replaced with 172.17.0.1.nip.io:8443.

Before You Begin

Before getting started creating your own APBs, you must set up your development environment:

  1. Ensure you have access to an OKD cluster. The cluster should be running both the service catalog and the OpenShift Ansible broker (OAB), which are installed by default.

  2. Install the APB tools as documented in the CLI Tooling topic. To verify, you can run the apb help command and check for a valid response.

  3. If you are developing against an OKD cluster that exists on a remote host or you do not have access to the docker daemon, see Working with Remote Clusters for alternative steps when using the apb push and apb run commands described in this guide.

Creating Your First APB

In this tutorial, you will create an APB for a containerized hello world application. You will work through a basic APB that will mirror the APB hello-world-apb.

  1. Your first task is to initialize the APB using the apb CLI tool. This creates the skeleton for your APB. The command for this is simple:

    $ apb init my-test-apb

    After initialization, you will see the following file structure:

    my-test-apb/
├── apb.yml
├── Dockerfile
├── playbooks
│   ├── deprovision.yml
│   └── provision.yml
└── roles
    ├── deprovision-my-test-apb
    │   └── tasks
    │       └── main.yml
    └── provision-my-test-apb
        └── tasks
            └── main.yml

    Two files were created at the root directory: an apb.yml (the APB spec file) and a Dockerfile. These are the minimum files required for any APB. For more information about the APB spec file, see the Reference topic. There is also an explanation of what you can do in the Dockerfile.

    apb.yml
    version: 1.0
name: my-test-apb
description: This is a sample application generated by apb init
bindable: False
async: optional
metadata:
  displayName: my-test
plans:
  - name: default
    description: This default plan deploys my-test-apb
    free: True
    metadata: {}
    parameters: []
    Dockerfile
    FROM ansibleplaybookbundle/apb-base

LABEL "com.redhat.apb.spec"=\

COPY playbooks /opt/apb/actions
COPY roles /opt/ansible/roles
RUN chmod -R g=u /opt/{ansible,apb}
USER apb

  2. In the Dockerfile, there are two updates to make:

    1. Change the FROM directive to use the image from the Red Hat Container Catalog. The first line should now read:

      FROM openshift3/apb-base

    2. Update com.redhat.apb.spec in the LABEL instruction with a base64 encoded version of apb.yml. To do this, run apb prepare:

      $ cd my-test-apb
$ apb prepare

      This updates the Dockerfile as follows:

      Dockerfile
      FROM openshift3/apb-base

LABEL "com.redhat.apb.spec"=\
"dmVyc2lvbjogMS4wCm5hbWU6IG15LXRlc3QtYXBiCmRlc2NyaXB0aW9uOiBUaGlzIGlzIGEgc2Ft\
cGxlIGFwcGxpY2F0aW9uIGdlbmVyYXRlZCBieSBhcGIgaW5pdApiaW5kYWJsZTogRmFsc2UKYXN5\
bmM6IG9wdGlvbmFsCm1ldGFkYXRhOgogIGRpc3BsYXlOYW1lOiBteS10ZXN0CnBsYW5zOgogIC0g\
bmFtZTogZGVmYXVsdAogICAgZGVzY3JpcHRpb246IFRoaXMgZGVmYXVsdCBwbGFuIGRlcGxveXMg\
bXktdGVzdC1hcGIKICAgIGZyZWU6IFRydWUKICAgIG1ldGFkYXRhOiB7fQogICAgcGFyYW1ldGVy\
czogW10="

COPY playbooks /opt/apb/actions
COPY roles /opt/ansible/roles
RUN chmod -R g=u /opt/{ansible,apb}
USER apb

  3. At this point, you have a fully formed APB that you can build. If you skipped using apb prepare, the apb build command will still prepare the APB before building the image:

    $ apb build

  4. You can now push the new APB image to the local OpenShift Container Registry:

    $ apb push

  5. Querying the OAB will now show your new APB listed:

    $ apb list
ID                                NAME            DESCRIPTION
< ------------ ID ------------->  dh-my-test-apb  This is a sample application generated by apb init

    Similarly, visiting the OKD web console will now display the new APB named my-test-apb in the service catalog under the All and Other tabs.

Adding Actions

The brand new APB created in the last section does not do much in its current state. For that, you must add some actions. The actions supported are:

  • provision

  • deprovision

  • bind

  • unbind

  • test

You will add each of these actions in the following sections. But before beginning:

  1. Ensure that you are logged in to your OKD cluster via the oc CLI. This will ensure the apb tool can interact with OKD and the OAB:

    # oc login <cluster_host>:<port> -u <user_name> -p <password>

  2. Log in to the OKD web console and verify your APB listed in the catalog:

    browse catalog my test
    Figure 1. OKD Web Console

  3. Create a project named getting-started where you will deploy OKD resources. You can create it using the web console or CLI:

    $ oc new-project getting-started

Provision

During the apb init process, two parts of the provision task were stubbed out. The playbook, playbooks/provision.yml, and the associated role in roles/provision-my-test-apb:

my-test-apb
├── apb.yml
├── Dockerfile
├── playbooks
│   └── provision.yml (1)
└── roles
    └── provision-my-test-apb
        └── tasks
            └── main.yml (2)
1 Inspect this playbook.
2 Edit this role.

The playbooks/provision.yml file is the Ansible playbook that will be run when the provision action is called from the OAB. You can change the playbook, but for now you can just leave the code as is.

playbooks/provision.yml
- name: my-test-apb playbook to provision the application
  hosts: localhost
  gather_facts: false
  connection: local
  roles:
  - role: ansible.kubernetes-modules
    install_python_requirements: no
  - role: ansibleplaybookbundle.asb-modules
  - role: provision-my-test-apb
    playbook_debug: false

The playbook will execute on localhost and execute the role provision-my-test-apb. This playbook works on its local container created by the service broker. The ansible.kubernetes-modules role allow you to use the kubernetes-modules to create your OKD resources. The asb-modules provide additional functionality for use with the OAB.

Currently, there are no tasks in the role. The contents of the roles/provision-my-test-apb/tasks/main.yml only contains comments showing common resource creation tasks. ou can currently execute the provision task, but since there are no tasks to perform, it would simply launch the APB container and exit without deploying anything.

You can try this now by clicking on the my-test APB and deploying it to the getting-started project using the web console:

provision my test
Figure 2. Provisioning my-test

When the provision is executing, a new namespace is created with the name dh-my-test-apb-prov-<random>. In development mode, it will persist, but usually this namespace would be deleted after successful completion. If the APB fails provisioning, the namespace will persist by default.

By looking at the pod resources, you can see the log for the execution of the APB. To view the pod’s logs:

  1. Find the namespaces by either using the web console to view all namespaces and sort by creation date, or using the following command:

    $ oc get ns
NAME                                STATUS    AGE
ansible-service-broker              Active    1h
default                             Active    1h
dh-my-test-apb-prov-<random>        Active    4m

  2. Switch to the project:

    $ oc project dh-my-test-apb-prov-<random>
Now using project "dh-my-test-apb-prov-<random>" on server "<cluster_host>:<port>".

  3. Get the pod name:

    $ oc get pods
NAME             READY     STATUS      RESTARTS   AGE
<apb_pod_name>   0/1       Completed   0          3m

  4. View the logs:

    $ oc logs -f <apb_pod_name>
...
+ ansible-playbook /opt/apb/actions/provision.yml --extra-vars '{"_apb_plan_id":"default","namespace":"getting-started"}'
PLAY [my-test-apb playbook to provision the application] ***********************
TASK [ansible.kubernetes-modules : Install latest openshift client] *************
skipping: [localhost]
TASK [ansibleplaybookbundle.asb-modules : debug] *******************************
skipping: [localhost]
PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=0

Creating a Deploying Configuration

At the minimum, your APB should deploy the application pods. You can do this by specifying a deployment configuration:

  1. One of the first tasks that is commented out in the provision-my-test-apb/tasks/main.yml file is the creation of the deployment configuration. You can uncomment it or paste the following:

    Normally, you would replace the image: value with your own application image.

    		 
    - name: create deployment config
  openshift_v1_deployment_config:
    name: my-test
    namespace: '{{ namespace }}' (1)
    labels: (2)
      app: my-test
      service: my-test
    replicas: 1 (3)
    selector: (4)
      app: my-test
      service: my-test
    spec_template_metadata_labels:
      app: my-test
      service: my-test
    containers: (5)
    - env:
      image: docker.io/ansibleplaybookbundle/hello-world:latest
      name: my-test
      ports:
      - container_port: 8080
        protocol: TCP
    1 Designates which namespace the deployment configuration should be in.
    2 Used to help organize, group, and select objects.
    3 Specifies that you only want one pod.
    4 The selector section is a labels query over pods.
    5 This containers section specifies a container with a hello-world application running on port 8080 on TCP. The image is stored at docker.io/ansibleplaybookbundle/hello-world.

    For more information, Writing APBs: Reference has more detail, and you can see the ansible-kubernetes-modules documentation for a full accounting of all fields.

  2. Build and push the APB:

    $ apb build
$ apb push

  3. Provision the APB using the web console.

  4. After provisioning, there will be a new running pod and a new deployment configuration. Verify by checking your OKD resources:

    $ oc project getting-started
$ oc get all
NAME         REVISION   DESIRED   CURRENT   TRIGGERED BY
dc/my-test   1          1         1         config

NAME           DESIRED   CURRENT   READY     AGE
rc/my-test-1   1         1         1         35s

NAME                 READY     STATUS    RESTARTS   AGE
po/my-test-1-2pw4t   1/1       Running   0          33s

    You will also be able to see the deployed application in the web console on the project’s Overview page.

The only way to use this pod in its current state is to use:

$ oc describe pods/<pod_name>

to find its IP address and access it directly. If there were multiple pods, they would be accessed separately. To treat them like a single host, you need to create a service, described in the next section.

To clean up before moving on and allow you to provision again, you can delete the getting-started project and recreate it or create a new one.

Creating a Service

You will want to use multiple pods, load balance them, and create a service so that a user can access them as a single host:

  1. Modify the provision-my-test-apb/tasks/main.yml file and add the following:

    - name: create my-test service
  k8s_v1_service:
    name: my-test
    namespace: '{{ namespace }}'
    labels:
      app: my-test
      service: my-test
    selector:
      app: my-test
      service: my-test
    ports:
      - name: web
        port: 80
        target_port: 8080

    The selector section will allow the my-test service to include the correct pods. The ports will take the target port from the pods (8080) and expose them as a single port for the service (80). Notice the application was running on 8080 but has now been made available on the default HTTP port of 80.

    The name field of the port allows you to specify this port in the future with other resources. More information is available in the k8s_v1_service module.

  2. Build and push the APB:

    $ apb build
$ apb push

  3. Provision the APB using the web console.

After provisioning, you will see a new service in the web console or CLI. In the web console, you can click on the new service under Networking in the application on the Overview page or under Applications → Services. The service’s IP address will be shown which you can use to access the load balanced application.

To view the service information from the command line, you can do the following:

$ oc project getting-started
$ oc get services
$ oc describe services/my-test

The describe command will show the IP address to access the service. However, using an IP address for users to access your application is not generally what you want. Instead, you should create a route, described in the next section.

To clean up before moving on and allow you to provision again, you can delete the getting-started project and recreate it or create a new one.

Creating a Route

You can expose external access to your application through a reliable named route:

  1. Modify the provision-my-test-apb/tasks/main.yml file and adding the following:

    - name: create my-test route
  openshift_v1_route:
    name: my-test
    namespace: '{{ namespace }}'
    labels:
      app: my-test
      service: my-test
    to_name: my-test
    spec_port_target_port: web

    The to_name is the name of the target service. The spec_port_target_port refers to the name of the target service’s port. More information is available in the openshift_v1_route module.

  2. Build and push the APB:

    $ apb build
$ apb push

  3. Provision the APB using the web console.

After provisioning, you will see the new route created. On the web console’s Overview page for the getting-started project, you will now see an active and clickable route link listed on the application. Clicking on the route or visiting the URL will bring up the hello-world application.

You can also view the route information from the CLI:

$ oc project getting-started

$ oc get routes
NAME      HOST/PORT                                   PATH      SERVICES   PORT      TERMINATION   WILDCARD
my-test   my-test-getting-started.172.17.0.1.nip.io             my-test    web                     None

$ oc describe routes/my-test
Name:			my-test
Namespace:		getting-started
...

At this point, your my-test application is fully functional, load balanced, scalable, and accessible. You can compare your finished APB to the hello-world APB in the hello-world-apb example repository.

Deprovision

For the deprovision task, you must destroy all provisioned resources, usually in reverse order from how they were created.

To add the deprovision action, you need a deprovision.yml file under playbooks/ directory and related tasks in the roles/deprovision-my-test-apb/tasks/main.yml. Both these files should already be created for you:

my-test-apb/
├── apb.yml
├── Dockerfile
├── playbooks
│   └── deprovision.yml (1)
└── roles
    └── deprovision-my-test-apb
        └── tasks
            └── main.yml (2)
1 Inspect this file.
2 Edit this file.

The content of the deprovision.yml file looks the same as the provision task, except it is calling a different role:

playbooks/deprovision.yml
- name: my-test-apb playbook to deprovision the application
  hosts: localhost
  gather_facts: false
  connection: local
  roles:
  - role: ansible.kubernetes-modules
    install_python_requirements: no
  - role: ansibleplaybookbundle.asb-modules
  - role: deprovision-my-test-apb
    playbook_debug: false

Edit that role in the file roles/deprovision-my-test-apb/tasks/main.yml. By uncommenting the tasks, the resulting file without comments should look like the following:

- openshift_v1_route:
    name: my-test
    namespace: '{{ namespace }}'
    state: absent

- k8s_v1_service:
    name: my-test
    namespace: '{{ namespace }}'
    state: absent

- openshift_v1_deployment_config:
    name: my-test
    namespace: '{{ namespace }}'
    state: absent

In the provision.yml file created earlier, you created a deployment configuration, service, then route. For the deprovision action, you should delete the resources in reverse order. You can do so by identifying the resource by namespace and name, and then marking it as state: absent.

To run the deprovision template, click on the menu on the list of Deployed Services and select Delete.

Bind

From the previous sections, you learned how to deploy a standalone application. However, in most cases applications will need to communicate with other applications, and often with a data source. In the following sections, you will create a PostgreSQL database that the hello-world application deployed from my-test-apb can use.

Preparation

For a good starting point, create the necessary files for provision and deprovisioning PostgreSQL.

A more in-depth example can be found at the PostgreSQL example APB.

  1. Initialize the APB using the --bindable option:

    $ apb init my-pg-apb --bindable

    This creates the normal APB file structure with a few differences:

    my-pg-apb/
├── apb.yml (1)
├── Dockerfile
├── playbooks
│   ├── bind.yml (2)
│   ├── deprovision.yml
│   ├── provision.yml
│   └── unbind.yml (3)
└── roles
    ├── bind-my-pg-apb
    │   └── tasks
    │       └── main.yml (4)
    ├── deprovision-my-pg-apb
    │   └── tasks
    │       └── main.yml
    ├── provision-my-pg-apb
    │   └── tasks
    │       └── main.yml (5)
    └── unbind-my-pg-apb
        └── tasks
            └── main.yml (6)
    1 bindable flag set to true
    2 New file
    3 New file
    4 New empty file
    5 Encoded binding credentials
    6 New empty file

    In addition to the normal files, new playbooks bind.yml, unbind.yml, and their associated roles have been stubbed out. The bind.yml and unbind.yml files are both empty and, because you are using the default binding behavior, will remain empty.

  2. Edit the apb.yml file. Notice the setting bindable: true. In addition to those changes, you must add some parameters to the apb.yml for configuring PostgreSQL. They will be available fields in the web console when provisioning your new APB:

    version: 1.0
name: my-pg-apb
description: This is a sample application generated by apb init
bindable: True
async: optional
metadata:
  displayName: my-pg
plans:
  - name: default
    description: This default plan deploys my-pg-apb
    free: True
    metadata: {}
    # edit the parameters and add the ones below.
    parameters:
      - name: postgresql_database
        title: PostgreSQL Database Name
        type: string
        default: admin
      - name: postgresql_user
        title: PostgreSQL User
        type: string
        default: admin
      - name: postgresql_password
        title: PostgreSQL Password
        type: string
        default: admin

    The playbooks/provision.yml will look like the following:

    - name: my-pg-apb playbook to provision the application
  hosts: localhost
  gather_facts: false
  connection: local
  roles:
  - role: ansible.kubernetes-modules
    install_python_requirements: no
  - role: ansibleplaybookbundle.asb-modules
  - role: provision-my-pg-apb
    playbook_debug: false

    The playbooks/deprovision.yml will look like the following:

    - name: my-pg-apb playbook to deprovision the application
  hosts: localhost
  gather_facts: false
  connection: local
  roles:
  - role: ansible.kubernetes-modules
    install_python_requirements: no
  - role: deprovision-my-pg-apb
    playbook_debug: false

  3. Edit the roles/provision-my-pg-apb/tasks/main.yml file. This file mirrors your hello-world application in many respects, but adds a persistent volume (PV) to save data between restarts and various configuration options for the deployment configuration.

    In addition, a new task has been added at the very bottom after the provision tasks. To save the credentials created during the provision process, you must encode them for retrieval by the OAB. The new task, using the module asb_encode_binding, will do so for you.

    You can safely delete everything in that file and replace it with the following:

    # New persistent volume claim
- name: create volumes
  k8s_v1_persistent_volume_claim:
    name: my-pg
    namespace: '{{ namespace }}'
    state: present
    access_modes:
      - ReadWriteOnce
    resources_requests:
      storage: 1Gi

- name: create deployment config
  openshift_v1_deployment_config:
    name: my-pg
    namespace: '{{ namespace }}'
    labels:
      app: my-pg
      service: my-pg
    replicas: 1
    selector:
      app: my-pg
      service: my-pg
    spec_template_metadata_labels:
      app: my-pg
      service: my-pg
    containers:
    - env:
      - name: POSTGRESQL_PASSWORD
        value: '{{ postgresql_password }}'
      - name: POSTGRESQL_USER
        value: '{{ postgresql_user }}'
      - name: POSTGRESQL_DATABASE
        value: '{{ postgresql_database }}'
      image: docker.io/centos/postgresql-94-centos7
      name: my-pg
      ports:
      - container_port: 5432
        protocol: TCP
      termination_message_path: /dev/termination-log
      volume_mounts:
      - mount_path: /var/lib/pgsql/data
        name: my-pg
      working_dir: /
    volumes:
    - name: my-pg
      persistent_volume_claim:
        claim_name: my-pg
      test: false
      triggers:
      - type: ConfigChange

- name: create service
  k8s_v1_service:
    name: my-pg
    namespace: '{{ namespace }}'
    state: present
    labels:
      app: my-pg
      service: my-pg
    selector:
      app: my-pg
      service: my-pg
    ports:
    - name: port-5432
      port: 5432
      protocol: TCP
      target_port: 5432

# New encoding task makes credentials available to future bind operations
- name: encode bind credentials
  asb_encode_binding:
    fields:
      DB_TYPE: postgres
      DB_HOST: my-pg
      DB_PORT: "5432"
      DB_USER: "{{ postgresql_user }}"
      DB_PASSWORD: "{{ postgresql_password }}"
      DB_NAME: "{{ postgresql_database }}"

    The encode bind credentials task will make available several fields as environment variables: DB_TYPE, DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, and DB_NAME. This is the default behavior when the bind.yml file is left empty. Any application (such as hello-world) can use these environment variables to connect to the configured database after performing a bind operation.

  4. Edit the roles/deprovision-my-pg-apb/tasks/main.yml and uncomment the following lines so that the created resources will be deleted during deprovisioning:

    - k8s_v1_service:
    name: my-pg
    namespace: '{{ namespace }}'
    state: absent

- openshift_v1_deployment_config:
    name: my-pg
    namespace: '{{ namespace }}'
    state: absent

- k8s_v1_persistent_volume_claim:
    name: my-pg
    namespace: '{{ namespace }}'
    state: absent

  5. Finally, build and push your APB:

    $ apb build
$ apb push

At this point, the APB can create a fully functional PostgreSQL database to your cluster. You can test it out in the next section.

Executing From the UI

To test your application, you can bind a hello-world application to the provisioned PostgreSQL database. You can use the application previously created in the Provision section of this tutorial, or you can use the hello-world-apb:

  1. First, provision my-test-apb.

  2. Then, provision my-pg-apb and select the option to Create a secret:

    provision my pg
    provision my pg params
    provision my pg binding
    provision my pg results

  3. Now, if you have not already done so, navigate to the project. You can see both your hello-world application and your PostgreSQL database. If you did not select to create a binding at provision time, you can also do so here with the Create binding link.

  4. After you the binding has been created, you must add the secret created by the binding into the application. First, navigate to the secrets on the Resources → Secrets page:

    my pg nav secrets
    my pg secrets

  5. Add the secret as environment variables:

    my pg add secret
    my pg add secret app

  6. After this addition, you can return to the Overview page. The my-test application may still be redeploying from the configuration change. If so, wait until you can click on the route to view the application:

    my pg overview

    After clicking the route, you will see the hello-world application has detected and connected to the my-pg database:

    my pg hello world

Test

Test actions are intended to check that an APB passes a basic sanity check before publishing to the service catalog. They are not meant to test a live service. OKD provides the ability to test a live service using liveness and readiness probes, which you can add when provisioning.

The actual implementation of your test is left to you as the APB author. The following sections provide guidance and best practices.

Writing a Test Action

To create a test action for your APB:

  • Include a playbooks/test.yml file.

  • Include defaults for the test in the playbooks/vars/ directory.

my-apb/
├── ...
├── playbooks/
    ├── test.yml
    └── vars/
        └── test_defaults.yml

To orchestrate the testing of an APB, you should use the include_vars and include_role modules in your test.yml file:

test.yml
- name: test media wiki abp
  hosts: localhost
  gather_facts: false
  connection: local

  roles:
  - role: ansible.kubernetes-modules (1)
    install_python_requirements: no

  post_tasks:
  - name: Load default variables for testing (2)
    include_vars: test_defaults.yaml
  - name: create project for namespace
    openshift_v1_project:
      name: '{{ namespace }}'
  - name: Run the provision role. (3)
    include_role:
      name: provision-mediawiki-apb
  - name: Run the verify role. (4)
    include_role:
      name: verify-mediawiki-apb
1 Load the Ansible Kubernetes modules.
2 Include the default values needed for provision from the test role.
3 Include the provision role to run.
4 Include the verify role to run. See Writing a Verify Role.
Writing a Verify Role

A verify role allows you to determine if the provision has failed or succeeded. The verify_<name> role should be in the roles/ directory. This should be a normal Ansible role.

my-apb/
├── ...
└── roles/
    ├── ...
    └── verify_<name>
        ├── defaults
             └── defaults.yml
        └── tasks
            └── main.yml

An example task in the main.yml file could look like:

 - name: url check for media wiki
   uri:
     url: "http://{{ route.route.spec.host }}"
     return_content: yes
   register: webpage
   failed_when: webpage.status != 200
Saving Test Results

The asb_save_test_result module can also be used in the verify role, allowing the APB to save test results so that the apb test command can return them. The APB pod will stay alive for the tool to retrieve the test results.

For example, adding asb_save_test_result usage to the previous main.yml example:

 - name: url check for media wiki
   uri:
     url: "http://{{ route.route.spec.host }}"
     return_content: yes
   register: webpage

  - name: Save failure for the web page
    asb_save_test_result:
      fail: true
      msg: "Could not reach route and retrieve a 200 status code. Recieved status - {{ webpage.status }}"
    when: webpage.status != 200

  - fail:
      msg: "Could not reach route and retrieve a 200 status code. Recieved status - {{ webpage.status }}"
    when: webpage.status != 200

  - name: Save test pass
    asb_save_test_result:
      fail: false
    when: webpage.status == 200
Running a Test Action

After you have defined your test action, you can use the CLI tooling to run the test:

$ apb test

The test action will:

  • build the image,

  • start up a pod as if it was being run by the service broker, and

  • retrieve the test results if any were saved.

The status of pod after execution has finished will determine the status of the test. If the pod is in an error state, then something failed and the command reports that the test was unsuccessful.