Bot essentials 15 : NLP implementations in chatbots – 1

Conversations between a human and a chatbot have a distinct style. Typically humans use short phrases, short sentences, sometimes just a single word, question like sentence, with or without proper context and lowercase in most cases when seeking answers from a chatbot. This is where we talk about NLP implementations in chatbots.

NLP implementations in chatbots

The number of variations could be as much as the number of users interacting with the chatbot. This is true because everyone has a distinct style of conversing.

For a Retrieval chatbot, users interact with a chatbot in a question answer mode to get information on various topics e.g. a customer service chatbot, a sales enquiry chatbot.

Reference documents having questions with answers cover various topics.

Users may have minimal to no information on the topics and may use an enquiry style to ask a high level question and drill down with follow up questions based on responses.

Sometimes they may start with a sentence..

.. that carries the context and then ask follow up questions without the context. It is not necessary to have a questioning style. While a short phrase as a message could mean a proxy question.

Users can jump back and forth among various topics or may continue a threaded conversation. Based on the language comfort level of the user the words and phrases will widely differ.

There is also the case of out of vocabulary (OOV). This implies that users may choose to use a word that is not available in the vocabulary of the chatbot.

A large variety of human interactions

Given the widespread variety of how humans interact with a chatbot, it is extremely challenging to discover and nail down the exact answer that match the expectation of the user.

After we have enough data representing the overall user base of a chatbot, we have a good chance of using supervised machine learning algorithms to train the chatbot to handle all representative conversations.

However chances are that any new chatbot that is built will not have this data upfront. This is a cold start problem and is widely prevalent. Or the data is not enough to get good accuracy from supervised techniques.

Thus, using a hybrid approach of using a mix of supervised and unsupervised techniques maybe a good solution to start with.

Read about Bot Essentials 16, which is the continuation of this blog in part 2