Bot to Human Handoff in Node.js

  • June 30, 2017
  • Views

    35,462

One common request from companies and organizations considering bots is the ability to “hand off” a customer from a bot to a human agent as seamlessly as possible. To solve this problem, we implemented an unopinionated e-2-e solution called Handoff, which enables bot authors to implement a wide variety of scenarios with minimal changes to the actual bot. We recently worked with a partner who wanted to use the Ibex dashboard to display all the Handoff interactions. A scenario we see often is a is help desk application; although little changes are needed to implement Handoff in your bot, creating your own help centre application that works with Handoff would require a bit of work.

This framework has been developed in TypeScript and Node.js, and a similar handoff scenario has also been developed in C# by Tomi Paananen. You can see his write-up of how the C# version works and his Intermediator-Bot sample on GitHub.

This project, Bot-Handoff, is open-sourced and available on GitHub. Contributions are always welcome!

How It Works

This framework connects Customers and Agents through the Microsoft Bot Framework.

The Handoff depends on a database of conversations, including a transcription of every message sent between the Customer and the bot, or between the Customer and the Agent.

A Handoff conversation consists of:

  • address information for this conversation with the Customer
  • the current state of this conversation (Bot, Waiting, or Agent)
  • the conversational transcript
  • address information for the Agent, if the Agent is currently connected to this Customer (Handoff does not record conversational metadata for the Agent, except when they are connected to a Customer)

Essentially, we have some middleware intercepting every message sent to and from the bot. If the message is from the bot, Customer, or an Agent who is speaking to a Customer, it transcribes the message before sending the message through.

If the message is from the Customer and contains the text “help me” (a command that can be configured by the author of the bot) we update the state of the Customer from ‘Bot’ to ‘Waiting’ meaning the user now waits for an Agent.

Once an Agent picks up the conversation, the state is changed from ‘Waiting’ to ‘Agent’. In this state, any message sent from the Customer is intercepted by the middleware and routed to the Agent using the stored address information for the Agent that picked up the conversation, and vice versa.

This code story will go into further detail about key aspects of this framework, including:

Middleware Routing

The heart of Handoff is the message router. Using the conversational metadata above, each message from a Customer, Bot, or Agent is routed appropriately. It is implemented as Bot middleware, and can be combined with any other middleware your bot is already using.

middleware process to transcribe message and connect Customer to Agent

The code snippet below shows the implementation of  bot.use()  which is how middleware is implemented in the Node.js Microsoft Bot Framework SDK. The middleware performs two key actions: first, it checks to see if what the user enters matches any of our commands. For example, if the user is an Agent and types “list”, it will list all of the current conversations with the bot. Next, the middleware routes the message:

If the activity is from the Customer or Agent, it will go through the botbuilder  method. We only want to transcribe activities of type “message” so only in that scenario do we call routeMessage() ; otherwise, we just pass the activity through as usual. Activities from the bot go through the send method where we do something similar.

Here we also check to make sure there are no entities in the message activity. If there is an entity,  it means that the message is from the Agent acting as the bot. We don’t want the message to send twice as it will already be routed through the botbuilder  method. (When the Handoff is active, the Agent is speaking to the bot to speak to the Customer, and that same message is then routed as the bot.)

After getting a message activity from a Customer or Agent, we use routeMessage()  to check who the user is.

Agent Recognition

Customers and Agents are both just users connected to bots, so Handoff needs a way to identify an Agent. We use a function of the form  isAgent(session: Session) => boolean  to determine if the user is an Agent, which can be customized by the author of the bot in the initial setup of the Handoff module. For example, in the snippet below, we determine that a user is an Agent if their username starts with “Agent”.

const isAgent = (session) => session.message.user.name.startsWith("Agent");

This information is passed by the bot author in the initial setup  handoff.setup(bot, app, isAgent, { }) .

There are multiple ways this could be set up:

  • Create a hardcoded directory of channel-specific user IDs for Agents (e.g., “Fred Doe on Facebook Messenger is one of our Agents”)
  • Create a WebChat-based call center app that specially encodes Agent user IDs (e.g., “Agent001”, “Agent002”), which is easy with WebChat
  • Create a WebChat-based call center app that authenticates users and then passes auth tokens to the bot via the WebChat backchannel
  • Use authbot to identify the user as an Agent via OAuth2 (e.g., “This authenticated user is marked as an Agent in our employee database”)

Transcripts and Logging

Once we know who the user is, several checks are made before sending the message through the bot. In the Agent routing method, we check to see if the Agent is in a conversation; this means checking if they are currently talking to a Customer. If not, we don’t need further routing and we just pass the message through. Otherwise, if they are talking to a Customer, we route the message through the bot to the Customer they are in the conversation with.

On the Customer side, we check their state. If they are taking to the bot, we just pass the message through. If they are waiting, we inform them they are being connected to an Agent. And finally, if they are in conversation with an Agent (and the Agent is in the conversation) we route the message through the bot to the Agent.

For both scenarios, we update the conversation’s transcript array, using the conversation ID, so we have all the messages that have gone between the Customer and the bot, and the Customer and the Agent.

The datetime variable used is of type  new Date().toISOString() so that it matches the format returned from localTimestamp . Something worth noting is that localTimestamp  (a method returned with a message activity) is not provided when the message is coming from WebChat. The SDK recommends not using the timestamp  value,  so we put a check in place to use localTimestamp  where possible.

Commands

As mentioned earlier, the middleware of the bot looks for commands in all the messages sent to the bot. There are several commands made available through the commands.ts  middleware feature. After checking to make sure the activity is of the message type, we check if this command is coming from the Agent or from the Customer in a similar fashion as routeMessage() using isAgent .

On the Agent side, there are four commands available:

CommandDescription
optionsDisplays all of the commands available to the Agent.
listLists all of the conversations with the bot, using the conversation data in MongoDB
connectThis is the main command that activates the Handoff, taking the Customer from Waiting (1) state to Agent (2) state. After this state change, all messages the Customer sends to the bot, and all messages the Agent sends to the bot, are routed using the address information sorted for that conversation.
disconnectThis command disconnects the Agent from the Customer, resetting the Customer’s state to Bot (0). If the bot author chooses, they can keep all of the conversation data by providing a retainData  option/environment variable. Otherwise, if the user has spoken to an Agent and the Agent has disconnected, the conversation is deleted from the database.

On the Customer side, there is only one command. The bot author can provide a custom keyword or phrase that a Customer can use to trigger the Handoff (the default is set to “help“) through an environment variable or the Handoff module initial setup options.

Regex is used to make sure the text is an exact match:

If the text matches exactly, the user’s message is transcribed, and they are queued to speak to an agent, changing their state from 0 (Bot) to 1 (Waiting).

Storing State on MongoDB

The conversation data, which consists of the items listed below, are all stored in a Mongo database:

  • address information for this conversation with the Customer
  • the current state of this conversation (Bot, Waiting, or Agent)
  • the conversational transcript
  • address information for the Agent, if the Agent is currently connected to this Customer (Handoff does not record conversational metadata for the Agent, except when they are connected to a Customer)

The Mongoose npm module was used to enable this feature; the bot author simply has to provide the MongoDB connection string as an option or environment variable. Another option is to use Azure Cosmos DB which automatically indexes all data and allows you to use the MongoDB API. You can learn more from this blog post on how to connect a MongoDB application to CosmosDB.

mongoose-provider.ts  handles all of the database interactions, and is where all of the schemas are described and, through a series of promises, the conversation is created, deleted, and updated.  For example:

Application Insights

The need to record all the conversation details in Azure Application Insights came from a hackfest we did with a partner. They wanted to use the Ibex dashboard to display all the Handoff interactions.

We implemented Application Insights as an optional feature by only logging the data to Application Insights if they provide the Application Insights instrumentation key in the initial setup of the handoff module, or through an environment variable.

We used the Application Insights npm module to set up and start an Application Insights client. This client is then used to track an event under the title “Transcript”.

The Application Insights logging happens in the addToTranscript()  method. Two things worth noting here:

  1. You can’t log embedded JSON objects in Application Insights, so we flatten the object to one item.
  2. We had to first stringify the object we got from MongoDB before parsing it so that functions from MongoDB don’t get logged  JSON.parse(JSON.stringify(OBJECT)).

Sentiment Analysis

We wanted to give bot authors the option to also log the sentiment score of messages from the Customer using Microsoft Cognitive Services Text Analytics; this API returns a value between 0 and 1 where 1 is a very positive sentiment and 0 is a very negative sentiment. If the user is not the Customer, or they do not provided an API key, the default sentiment score is set to -1.

We get the sentiment score by sending the user’s text to the Text Analytics API.

Force Handoff via API

To provide additional means of controlling and customizing the Handoff experience, an endpoint ( POST) is available to queue a Customer to speak to an Agent using the Customer’s conversation ID.  This API useful for a partner who wanted to use their own system to view all current conversations happening with a bot, and trigger the handoff manually with a button click, for any given conversation.

You can use the GET endpoint to retrieve a list of all current conversations:

The images below show a third party application took advantage of the Application Insights logs and this API:

Using the npm package

To make this framework easy to use, we packaged it up and made it available through npm under the name botbuilder-handoff.  Below is an example of how you can use the framework in a normal bot:

The README for this project explains every option in detail.

Conclusion

This framework enables anyone developing a bot using the Node.js Bot Framework SDK to integrate bot to human handoff feature. The added functionality of Application Insights and the sentiment score provide bot authors with more data about interactions with their bot. This data can then be used to customize the bot-to-human handoff experience and analyze the overall customer experience with their bot and agents.

This project is open sourced and available on Github:Bot-Handoff. We continue to add major new features, but it is in a usable state now. Contributions to this project are always welcome!

Resources

Related Articles

Leave a reply

Your email address will not be published. Required fields are marked *