Chatbot Testing – Democratizing Artificial Intelligence (AI)

Chatbots, the most advanced form of AI (Artificial Intelligence) are essentially the Computer programs / Software that imitates human behavior and conduct human – like conversations. Chatbots in today’s day and age are driving more than 60% of the web traffic, thus garnering stupendous amount of popularity and human engagement. They are being used for a variety of applications, Business Use Cases, Dating apps, Social Media etc. And generating an ROI of more than around 10 times than the traditional systems like Email marketing makes the Chatbots success all the more elusive.

A significant number of bots have been released in the market till date. To name a few –

1. Facebook’s M
2. Duolingo’s Renee the Driver, Chef Roberto, Officer Ada
3. Microsoft’s XiaoIce
4. Domino Pizza’s Dom
5. Fandengo
6. Wall Street Journal
7. National Geographic

Now, to build this state of the art product which quite literally performs the art of conversation, interpretation, validation and speech fluidity that is in a sense replacing humans with bots to engage with the target audience, it is very important to ensure that the chatbots do not come in market and become public failures considering that not just a lot of funding is involved in these ambitious projects to realize their full potential, but there can also be threats like compromised data sharing, data security etc. Effective testing can help us reduce the chatbot failures.

A Chatbot in order to be considered as successfully implemented needs to have several features which require rigorous testing taking into account the categorization in terms of features such as –

1.  Personality of a chatbot
2.  The way of interaction with the users.
3. How the chatbot interprets things like – abbreviations, emoji’s, slangs, requests etc.
4. How context relevant the answers are to what has been asked by the user.
5. How easy it is to communicate with the bot.
6. Recovery from errors and their management.
7. How the bot remembers things.
8. How quick the response time is.
9. How engaging the bot is in terms of conversations with the users.
10. How quickly it interprets and responds to the changed topic during a conversation.

So, we can see that there are many areas which require a very deep level of testing in order to ensure that there are absolutely no chances of deviation from the responses that are being expected from the bot making it’s testing a prime task.

Chatbot Testing Frameworks being used –

1. Botium.at                                     Open Source       Used for Chatbot Test Automation
2. ChatbotTest.com                        Open Source       Used to standardize Chatbot Testing
3. Dimon.co                                      Proprietary          Used for Chatbot Test Automation
4. Qbox.ai                                          Proprietary         Used for NLP training data optimization
5. Zypnos.com                                  Proprietary         Used for Regression Testing for Chatbots

Types of Testing done for Chatbot Implementation –

1. General Testing – This is done to test if the bot is atleast able to answer the basic, general questions like – greetings, how are you etc. The General Testing plays a very important part as it plays the deciding factor to if a bot will be further tested or not. Because if the bot doesn’t pass this first level of testing, then it won’t make any sense to test it further.
2. Domain Specific Testing – This one targets the specific product or service . The testing at this stage is done particularly using keywords which are aimed for a particular consumer group. Like for example – For a bot developed for a food delivery app should understand that Pizza and Pasta both come into the Italian cuisine category. Etc. and that the bot should be able to respond to the customer queries on the related topics.
3. Limit Testing – This is done to test the limits of the Chatbot as the name itself suggests. We need to see how the bot handles the irrelevant topics, questions and information. This shows how robust is the bot designed and see if it fails under such conditions or not.
4. Manual Testing – This is basically done for further testing and it taps on human intelligence, and how humans emote and interact with bots, to kind of create real-time scenarios and become more confident about the whole testing process and about the product.
5. A/B Testing – This type of testing is basically done is two parts(or we can say two versions of the product under test) , one focusing on the visual aspect of the bot and the other focusing on the conversational part of it.
6. Ad-Hoc Testing – No matter how much testing we do for Chatbots, even after they go live , there is always some scope for testing several more categories which might come up in the lifecycle of the bot at a later stage.

Challenges for Chatbot Testing –

1. Usage of different platforms – A user can decide to interact with a bot through any platform like – a desktop, mobile, tablet, social media. Now, the expectation here from the bot is that it should be able to communicate and respond seamlessly with the user irrespective of the platform used, and without losing the context.
2. There could be scenarios wherein the Chatbot user requires help on different topics. How seamlessly the bot switches from one domain to another is another limitation as there could be so many permutations and combinations that need to be tested in such scenarios.
3. Different users have different ways of talking, usage of words could be entirely different even if two people are using the same language. So, here the task of the bot is to understand the intent of the conversation and thus engage accordingly, which in itself could be a nagging limitation.

Chatbot Success Stories –

Most successful Chatbots implemented till date are –

1. Microsoft’s XiaoIce
2. Mitsuku
3. Replika.ai
4. GPT-3
5. Insomnobot-3000

Chatbot Failures-

1. Scatter Lab’s Luda Lee
2. Tencent Baby Q
3. Microsoft bot Zo
4. Microsoft bot Tay
5. Yandex’s Alice

CONCLUSION

Chatbots are very quickly taking over the traditional human conversations in terms of marketing and many other areas and gaining success as well as popularity at that but there is a big price to pay, as we have almost same percentage of chatbot failure stories as the success ones. Having said that, this is the future.