Custom Natural Language Understanding for Healthcare Chatbots and A Case Study

Text message-based conversational agents or chatbots are computer programs that run on a variety of messaging platforms such as Facebook Messenger, short messaging service, and Slack, among others. In particular, chatbots as conversational agents have been establishing means of communication in healthcare to help provide automated services and improve communication between patients and healthcare professionals. Recently, there is a surge in publicly available Natural Language Understanding (NLU) and Processing (NLP) services, which facilitate creation of chatbots. However, for domain specific cases the technical customizations can be exploited for chatbot optimizations, considering constrained chatbot vocabularies. This paper investigates NLU performance for applications with a limited number of intents and utterances per intent to improve classification performance in closed domain dialog systems. To facilitate this investigation, firstly a customized NLU service is presented with two state-of-the-art deep learning models for a healthcare intervention-based chatbot: 1) a Continuous Bag of words with shallow neural network classifier (CBOW-NN), and 2) a Bidirectional Encoder Representation from Transformers (BERT)-based SciBERT-NLU. The SciBERT is fine-tuned on a domain-specific dataset. Secondly, based on this dataset and other chatbot style datasets, a validation methodology is adapted for the classification performance of the custom and the most popular existing cloud-based off- the-shelf NLU services. Experimental results confirm that the custom BERT-based service with limited vocabulary outperforms, and the CBOW-NN approach achieves comparable performance with cloud-based reference NLU services.