Chatbots have become a very popular medium
To ask questions, access data and share information using Natural Language we now have chatbots. Most of us have had experiences with chatbots in our daily lives to perform simple tasks quickly and in our own convenient time. However, adoption of chatbots to access data within an enterprise, ask questions, quickly get to the locked away information in the enterprise is still very slow.
A lot of enterprise data, e.g., customer details, employee details, order details etc., are stored in structured format in RDBMS, Excel files, ERPs or other such proprietary structured format. Most chatbots operate on text data and do not have a good way to process structured data. Imagine a future where we can use chatbots to access this data using natural language.
Let’s say a HR personnel wants to find out how many employees joined in the previous month. Or the names and contact numbers of employees who stay in a specific location. Further, the names of the managers of those employees who have not recognized their team members and so on. How would the HR personnel get access to this data?
S/he would have to login to a HRIS system and then look through many reports or formulate search queries to get to the data. This takes a lot of time and in every case the exact data and relationships of data may not be determined from predefined reports.
The biggest challenge in enterprise
One of the biggest challenges of human computer interface is that humans think and talk in natural language, which is unstructured, whereas most softwares organize data in a structured format and we expect the human to interpret the structured data and map it to the information s/he is looking for. Having a chatbot perform this task will be a huge efficiency driver and lead to less friction in human-computer interaction.
SQL is a high level language which was designed to provide quick access to data in RDBMS without learning a full fledged programming language. Along with the the enhancements and improvements over many years to perform this task very well. Wouldn’t it be nice if we develop the capability to ‘translate’ natural language to SQL? What we are looking for is the ability to map natural language elements in a question/query to columns and rows in a table with filters.
Let’s take the employee details example again.
The HR personnel may ask: Give me the names and contact details of employees staying in ‘London’ or Can I get information of employees located in ‘London’ or I am looking for details of our ‘London’ employees. A human would ask the same question in many variations and chatbots have to understand all these variants and map them to the same question.
One idea is to start by identifying the ‘domain entities’ in the query
E.g., in this case ‘Name’, ‘Contact Details’, ‘Employees’, ‘London’. These are named entities that are relevant to the specific domain (in some cases they are generalized too). In NLP world you use a NER engine (named entity recognition engine) to predict the named entities in the query. This can obviously happen after you train the NER engine with a good enough dataset of labeled named entities which are part of query variations.
A popular algorithm to train an NER engine is CRF. Many open source NER engines based on CRF are available but these are general. They may not work for specific domains. You would rather take such open source code and train with your own domain entities till they can predict them. Since Deep Learning is the flavor of the day, researchers have also come up with Deep Learning networks for NER and these outperform the traditional CRF based NERs.
E.g., Spacy NER engine.
Adding simple rules to map identified entities
Once you have identified the entities, you could add simple rules to map the identified entities to columns/rows in a table and use a SQL generator algorithm to translate this to an equivalent SQL query. For simple querying use cases which do not need complex analytical output this mechanism will work well.
For solving complex use cases which can translate a query to a complex SQL query with multi-table joins and filtering the above mechanism may not be an elegant solution.
Deep Learning for further advancements
This is an active area of research. Many researchers are using a Deep Learning approach to translate a user query directly to an equivalent complex SQL. They are using a Deep Learning network. Effectively, they are building the intelligence in the network to generate complex SQL queries. They are doing it after training the network to understand the natural language query. And also, are using predictive models to predict the different parts of the equivalent SQL query. Salesforce’s Seq2SQL is one such example of a Deep Learning approach to solve this problem.
Imagine a future where you can tap into enterprise data locked in proprietary databases with a single touch. This will not only boost the productivity and efficiency but also democratize the data in enterprise data systems. The velocity at which it curates the data and stores in an enterprise is really high. It will be impossible to catch up with it by designing reports and search applications to give access to that data. Employing a chatbot would give access to that data as and when they are made available for consumption.