Unit Testing for Bot Applications

  • January 20, 2017
  • Views

    62,947

Image: Mr Robot has some RAM by Chris Isherwood, used by CC BY 2.0

First Things First

Writing code using ad-hoc technologies like Microsoft Bot Framework (MBF) is fun and exciting.
But before rushing to code bots that can make coffee and send spaceships to the moon, you need to think about unit testing your code.

Recently, we worked with Moed.ai on a bot that schedules tasks and manages resources’ time slots.
In order to ensure the quality of their bot Moed.ai wanted the ability to unit-test their bot logic.

This code story outlines the way we tackled the challenge of adding unit test support in Microsoft Bot Framework.

What Should We Test?

The first question we tried answering was “What are we trying to test?”

Bots built using MBF dialogs and intents usually consist of code intertwined with framework code.
Moreover, major parts of the framework require external connectivity for saving session states or querying LUIS models.

In most projects, when we think about unit testing, we think about segregated sections of logic that receive input and are expected to return specific output.
In case this logic has external dependencies, we need to search for ways to abstract those dependencies and provide stubs during testing.

This means we can take one of the following approaches:

  1. Move the project’s code to a separate module or directory and remove any dependencies on MBF
  2. Find a way to test the conversational logic of the bot without external dependencies

Code Separation Difficulty

Writing code for bots is a little different than writing code for other web-based applications.

Writing code for bot applications looks a lot like describing the way a conversational flow may evolve.
For example, just like in a customer service conversation, the bot application waits for a request that it can understand from the user to start a meaningful conversation. Then, during a series of questions and answers, it collects more data while trying to understand and provide the user with the best service.
Taking maintainability and readability into account, it is important that such conversations represented by your code are understandable.

Moreover, in bot scenarios, many actions and decisions require a high level of dependency on MBF.

Here are some examples of code that are dependent on MBF:

// Prompting the user for input
builder.Prompts.choice(session, 'What would you like to do?', [ "Play Games", "Do serious work", "Other" ]);

// Saving data in session variable
session.conversationData.userChoice = "Play games while appearing to work seriously";

// Beginning new dialogs or sending a "typing" status
session.send("Give a raise to smart worker");
session.sendTyping();

// Ending a conversation
session.endDialog("Good Bye");

A conversational flow consists of multiple REST calls and replies between the user and the web server. Trying to separate our logic from the conversational flow in many cases left us with almost no code to test. Moreover, that separation left the conversational flow logic untested.

When we thought about unit testing in the world of bots, we realized that conversational flow logic is intertwined with the core logic of our application. This realization led us to develop a unique approach to testing the bot.

How Do We Test Conversational Flow Logic?

For the purpose of this case study, we create a sample Alarm Clock Bot. A working copy of this bot application is available in this GitHub repo.

Let’s look at the following breakdown of a request to the bot application:

  • Receives request from API
  • Directs request to MBF
  • Performs dialog step with bot input
  • Communicates back to MBF

The code we want to test is wedged in the middle of this flow, but is entirely orchestrated by MBF.

Microsoft Bot Framework offers a class called ConsoleConnector. This class enables simulation of communications with the bot object without requiring an external connection:

var builder = require('botbuilder');

var connector = new builder.ConsoleConnector();

// BotToTest is the bot class exposing a collection of intents.
// To understand how to build such a class, follow the code in the Alarm Clock Bot repository
var bot = new BotToTest(connector);
bot.on('send', function (message) {
  /* Check returned message */
});

connector.processMessage('Hello World');

Next, let’s see how to test multiple steps in a dialog.

For this purpose we use a step indicator to tell us the index of the message:

var step = 1;
bot.on('send', function (message) {
  
  switch(step++) {
    case 1:
      assert(message.text == 'What would you like to do?');
      connector.processMessage('Play Games');
      break;

    case 2:
      assert(message.text == 'When?');
      connector.processMessage('Always');
      break;

    case 3:
      assert(message.text == 'Why?');
      connector.processMessage('I'm too cool for school');
      break;

    case 4:
      assert(message.text == 'No problem');
      break;

    default:
      assert(false); // The conversation should have ended
  }
  
});
connector.processMessage('Hello World');

To enable easy addition of more tests to the suite, we moved the conversational flow to an external module that exports JSON and consumes that via a generic tester:

/test/dialog-flows/context-switching.js:

module.exports = [
  {
    out: "set alarm in 10 seconds"
  },
  {
    in: "What would you like to call your alarm?",
    out: "test"
  },
  {
    in: /^(Creating alarm named "test" for)/i,
    out: "delete alarm named test"
  },
  {
    in: "Deleted the 'test' alarm." // the message sent by the bot after a few seconds
  }
];

/test/common.js

function testBot(bot, messages, done) {
  var step = 1;
  var connector = bot.connector();
  bot.on('send', function (message) {
      
    var check = messages[step - 1];
    
    // Check input message
    if (check.in) {
      assert(message.text === check.in);
    }

    // Send an output reply
    if (check.out) {
      connector.processMessage(check.out);
    }

    // End conversation in the last message
    step++;
    if (step - 1 == messages.length) {
      done();
    }
  });
}

module.exports = {
  testBot
}

Testing LUIS

LUIS is an external service integrated into the MBF SDK and used by many bot applications for intent recognition and entity extraction. To enable testing of that service we need to mimic URL calls to LUIS. We used Nock to simulate LUIS’s call, but Nock can be used to mimic any URL calls.

For the following mocked-up call, we made a GET call to LUIS with the appropriate query and copied the responses:

  nock('https://luis.url')
    .get('/?id=appId&subscription-key=subId&q=' + encodeURIComponent('set alarm test in 10 seconds'))
    .reply(200, {
      "query": "set alarm test in 10 seconds",
      "intents": [
        {
          "intent": "builtin.intent.alarm.set_alarm"
        }
        /* ... */
      ],
      "entities": [
        {
          "entity": "in 10 seconds",
          "type": "builtin.alarm.start_time",
          "resolution": {
            "resolution_type": "builtin.datetime.time",
            "time": "2016-12-14T15:31:59" // some time in the past
          }
        }
      ]
    })
    .get('/?id=appId&subscription-key=subId&q=' + encodeURIComponent('delete alarm named test'))
    .reply(200, {
      "query": "delete alarm named test",
      "intents": [
        {
          "intent": "builtin.intent.alarm.delete_alarm"
        }
      ],
      "entities": [
        {
          "entity": "test",
          "type": "builtin.alarm.title"
        }
      ]
    });

Creating The Suite

Finally, to wrap it all up in a test suite, we used a describe call:

/* requires */

var historyMessages = require('./dialog-flows/history-intents');
var switchingMessages = require('./dialog-flows/context-switch');
common.setup();

//Our parent block
describe('Bot Tests', () => {

  it('should recognize history intents', function (done) { 
      var connector = new builder.ConsoleConnector();
      var bot = TestedBot.create(connector);

      common.testBot(bot, historyMessages, done);
      
      connector.processMessage('hi');
  });

  it('context switching', function (done) { 
      var connector = new builder.ConsoleConnector();

      var bot = TestedBot.create(connector);       
      common.testBot(bot, switchingMessages, done);
      
      connector.processMessage('hi');
  });
});

To be able to do that, we needed the bot to expose a create call that receives a connector object:

var builder = require('botbuilder');

function create(connector) {

  var bot = new builder.UniversalBot(connector);
  var intents = new builder.IntentDialog();

  bot.dialog('/', intents);
  /* ... */

  return bot;
}

module.exports = { create };

This way, we can supply a builder.ChatConnector in runtime and a builder.ConsoleConnector in testing.

Opportunities for Reuse

You should reuse this code in any Microsoft Bot Framework project that requires unit testing for conversational flow.

Repositories

This is a sample of building a Set Alarm project with integrated unit testing:
https://github.com/CatalystCode/alarm-bot-unit-testing
It can also be used as a quickstart template for developing bots.

To see the full solution integrated into a bot see:
https://github.com/CatalystCode/multilingual-uber-bot

Related Articles

Leave a reply

Your email address will not be published. Required fields are marked *

2018-05-04 11:35:43

Joel says:

Still trying it out

2017-08-22 22:03:18

Pedro Dias says:

The title of this blogpost suggests Unit Testing, however, all I see here are functional tests. Unit Tests are done in isolation, so the very fact that you're actually using connectors dismisses these as unit tests. They are for all intents and purposes functional tests.So even though this article may be useful for many people wanting to do a functional test of their bots, I would suggest you change it's title. This has nothing to do with unit testing.

2017-08-20 07:39:31

Mor Shemesh says:

Hi Sandy, What exactly do you mean? This solution enables functional testing of the dialog without using the "UniversalBot" class which means, you are testing only the logic of your application without the bot framework.You can potentially run the same tests with UniversalBot and MBF in the process (although I haven't tried it yet).Or you mean something new altogether?

2017-07-12 17:06:31

Sandy says:

Thanks for the article! It is a time consuming task to write all the test cases incl. expected results on a script. Do you know how I can perform a functional test on the MBF?

2017-07-07 18:45:49

Matt says:

There are several limitations to this approach, including but not limited to not being able to send attachments, not being able to check session state, and not being able to check behavior dependent on address.If you check out https://github.com/matts310/BotTester, you'll find a more advanced, yet still fairly simple, method of building tests. You can easily build simple or complex tests that can verify anything from a random collection of responses to session state. Its much easier to work with than the unwieldily switch with step counter.

2017-05-25 08:59:17

Mor Shemesh says:

Hi Kum, Not sure I understand your question but I'll try to answer: 1. Bots, in most scenarios are useful as a new type of application. 2. Bots are not designed for testing other applications, but to understand user intents and translate them into actionable logic. I am not aware of any bots thus far that have been used for any sort of testing on other applications (functional or otherwise). 3. This article explains how, once you've build a bot, you can test the code you've built and run unit testing on your code before publishing to production.Hope this helps.

2017-05-23 22:01:00

kum says:

Hi Shemesh,Is bots are useful to test functional testing of any application, like similar to Selenium ?..or is it designed only for to test messaging applications(facebook,skype ...e.t.c)..Please clarify my doubt.Regards, Kum

2017-04-20 14:38:34

Mor Shemesh says:

Hi ajay, not sure what you mean, what is elastic query?

2017-04-20 13:28:31

ajay says:

how to call an elastic query in bot