Rasa Testing Forms & Actions

In this post, we’ll review a method to automate testing of your Rasa custom actions and forms using Postman.

Rasa has many good testing capabilities described in the docs and there’s also a good post on incorporating tests into your CI/CD pipeline. But Rasa cannot currently run tests that require calls to custom actions including the calls that support forms.

Forms and custom actions are an integral part of most Rasa chatbots. As you add features to your bot, the logic contained in forms and actions can become more complex. Manual testing of complex forms is tedious and error prone. It would be nice to automate this.

Vincent Warmerdam did a recent post on the use of Postman to explore the Rasa API’s. Postman can also be used to automate API endpoint testing.

I’m going to write some Postman test scripts to test forms in the Rasa example helpdesk assistant. You can find a demo and introduction to the helpdesk assistant here. In the demo video, the user opens a ticket to report a problem accessing email. I’ll write a Postman test for this form.

Helpdesk Form Example

The email problem reporting example is handled by the helpdesk assistant using this rule and the open_incident_form.

The existing Rasa tests for the form include tests for the form including the email problem example. But the form validation action code and follow-up action_open_incident calls are not performed by the test.

Existing Form Test

Here’s the existing test for this example:

- story: email incident
  steps:
  - intent: greet
    user: |-
      hello
  - action: utter_greet
  - action: utter_help
  - intent: problem_email
    user: |-
      I have a problem with my email
  - action: open_incident_form
  - active_loop: open_incident_form
  - active_loop: null
  - action: action_open_incident

Because rasa test doesn’t call the action code, what’s not tested?

• Required Slots: The form definition for open_incident_form requires five slots (email, priority, problem_description, incident_title and confirm)
• Slot Validation: This form validates the email address submitted by the user
• Python Bugs: Any number of coding errors could exist and won’t be tested
• API Calls: Calls to external API’s are not tested

Testing with Postman

Let’s write some tests with Postman that will exercise the open_incident_form including a happy path test and tests that should cause us to exit the form without submitting a new incident.

To test with Postman we’ll use the Rasa REST channel. To enable the REST channel with the helpdesk assistant we need to create a credentials.yml file with a single line:

rest:

Then start the bot with rasa run --enable-api --debug and also start the action server.

Create Postman Tests

With the bot and action server running, start Postman and let’s create the tests for our form. Start by creating a new Collection which will contain all of the calls for our form testing.

The first call we’ll make to the Rasa REST endpoint is to send the message I'm having issues with my password to kick off the form. Here’s a screenshot of the API call.

Across the top I set this as a POST call to the endpoint /webhooks/rest/webhook. I’m using the `` variable to specify http://localhost:5005.

In the Body field I’ve specified the sender and message json values required by the Rasa REST endpoint. At this point, you should be able to click the Send button and your bot should respond and show the results in the body response.

Let’s add a test to check that we received the expected response from Rasa. Click on the Tests tab and you’ll see that I’ve added two tests. One that confirms that the POST call returned a status of 200 and the second that looks for the expected response text.

When the test succeeds, duplicate the test you just made and add tests for each of the bot prompts that we expect to follow the happy path for this form.

Run All Tests

Now that we’ve configured a set of calls to test our form, let’s run all of them as a collection and make sure that the test conditions succeed:

Notice that I’ve added a few more tests to the collection. I added calls to check the status and health of Rasa and the REST endpoint and I reset the bot before I start testing the form.

Import Postman Collection

You can import the Postman test collection and try it yourself. I’ve exported it to the Helpdesk Assistant in the tests/postman directory and added a Make command to run a test, make test-postman.

I also recently added use of the Postman htmlextra plugin to present the test results on an HTML page.

Continuous Integration

Now that we’ve created a complete end-to-end test of our form, we should automate this test in our CI pipeline. You can learn about how to do this in the CI with Postman API page.