Deep open-domain chatbots: A study on the ParlAI Blended Skill Talk chatbot

Abstract

Natural language processing has seen many advancements in recent years due to the availability of large amounts of data and breakthroughs in deep learning. One evolving field of NLP is dialogue systems which are used for conversation with others. These systems can be roughly divided into two categories: task-oriented and open-domain chit-chat. Task-oriented systems focus on a single task while open-domain systems are capable of discussing a wide range of topics. This work focuses on the latter.

Certain open-domain conversation models are reaching levels comparable to humans in metrics such as engagingness and interestingness. This work aims to explain the architecture and methods used to implement the ParlAI BST model. The methods include the training data and the evaluation methods for the model. Then experiments with the model are done to study its performance. Finally, a system that can be used to integrate this model onto other systems is made publicly available.

The model examined was able to produce very human-like utterances, but had major flaws in the following areas of conversation: contradiction, memory and factual correctness. These flaws occur less often in short conversations compared to longer ones. Model size also affects how prominent these flaws are and large models show them less compared to small models. More research has to be done to mitigate the current flaws and to allow for longer conversations. Another area that also requires more research, is the automatic evaluation of open-domain models. Many of the available methods do not correlate with human judgments well, which makes human evaluations almost a requirement for accurate evaluation.