In this post, we’ll review a method to automate testing of your Rasa custom actions and forms using Postman.
Rasa has many good testing capabilities described in the docs and there’s also a good post on incorporating tests into your CI/CD pipeline. But Rasa cannot currently run tests that require calls to custom actions including the calls that support forms.
Forms and custom actions are an integral part of most Rasa chatbots. As you add features to your bot, the logic contained in forms and actions can become more complex. Manual testing of complex forms is tedious and error prone. It would be nice to automate this.
I’m going to write some Postman test scripts to test forms in the Rasa example helpdesk assistant. You can find a demo and introduction to the helpdesk assistant here. In the demo video, the user opens a ticket to report a problem accessing email. I’ll write a Postman test for this form.
Helpdesk Form Example
The existing Rasa tests for the form include tests for the form including the email problem example. But the form validation action code and follow-up action_open_incident calls are not performed by the test.
Existing Form Test
Here’s the existing test for this example:
- story: email incident steps: - intent: greet user: |- hello - action: utter_greet - action: utter_help - intent: problem_email user: |- I have a problem with my email - action: open_incident_form - active_loop: open_incident_form - active_loop: null - action: action_open_incident
rasa test doesn’t call the action code, what’s not tested?
- Required Slots: The form definition for
open_incident_formrequires five slots (
- Slot Validation: This form validates the email address submitted by the user
- Python Bugs: Any number of coding errors could exist and won’t be tested
- API Calls: Calls to external API’s are not tested
Testing with Postman
Let’s write some tests with Postman that will exercise the
open_incident_form including a happy path test and tests that should cause us to exit the form without submitting a new incident.
To test with Postman we’ll use the Rasa REST channel. To enable the REST channel with the helpdesk assistant we need to create a
credentials.yml file with a single line:
Then start the bot with
rasa run --enable-api --debug and also start the action server.
Create Postman Tests
With the bot and action server running, start Postman and let’s create the tests for our form. Start by creating a new Collection which will contain all of the calls for our form testing.
The first call we’ll make to the Rasa REST endpoint is to send the message
I'm having issues with my password to kick off the form. Here’s a screenshot of the API call.
Across the top I set this as a
POST call to the endpoint
/webhooks/rest/webhook. I’m using the `` variable to specify
Body field I’ve specified the
message json values required by the Rasa REST endpoint. At this point, you should be able to click the
Send button and your bot should respond and show the results in the body response.
Let’s add a test to check that we received the expected response from Rasa. Click on the
Tests tab and you’ll see that I’ve added two tests. One that confirms that the
POST call returned a status of 200 and the second that looks for the expected response text.
When the test succeeds, duplicate the test you just made and add tests for each of the bot prompts that we expect to follow the happy path for this form.
Run All Tests
Now that we’ve configured a set of calls to test our form, let’s run all of them as a collection and make sure that the test conditions succeed:
Notice that I’ve added a few more tests to the collection. I added calls to check the status and health of Rasa and the REST endpoint and I reset the bot before I start testing the form.
Import Postman Collection
You can import the Postman test collection and try it yourself. I’ve exported it to the Helpdesk Assistant in the tests/postman directory and added a Make command to run a test,
I also recently added use of the Postman htmlextra plugin to present the test results on an HTML page.
Now that we’ve created a complete end-to-end test of our form, we should automate this test in our CI pipeline. You can learn about how to do this in the CI with Postman API page.