Why p not q? An investigation into the affective and behavioral effects of contrastive explanation by a product recommendation chatbot
by Zhimiao Li
Thesis Supervisor: Dr. Theo Araujo
The need for chatbot transparency and output interpretability arises as a consequence of the increasing complexity of current systems of product recommendation chatbots. This may, in turn, result in users questioning the chatbot and subsequently a low recommendation adherence, especially in the contexts (e.g., healthcare and financial services) where the outputs from the chatbots are significant. Recently, accompanying recommendations with a contrastive explanation that provides justifications for why a recommendation rather than the alternative(s) was made – a novel approach to explainable AI has been proposed to address the need. Incorporating contrastiveness in explanations refers to a comparison in the form of “In contrast to Q, P is more……” where P is the fact, and Q is the contrast case.
While many existing studies in social science emphasize the potential benefits of contrastive explanation, most of them have a theoretical or conceptual nature, often not reporting empirical test results. Few have explored the user-centered evaluation of contrastive explanation from a perspective of communication science. Moreover, investigating how a contrastive explanation by a communicator is perceived by a receiver may help communicators to make recommendations more persuasive.
As a result, my research aims to address the research question: To what extent do contrastive explanations and anthropomorphic cues influence users’ affective and behavioral responses to a recommendation from a chatbot?
To answer the research question, I conducted a 2 (contrastive versus non-contrastive explanation) x 2 (with versus without andromorphic cues) between-subjects experiment (N = 200). For the experiment, I built a chatbot using Conversational Agent Research Toolkit (CART) designed by Dr. Theo Araujo. I also designed chatbot dialogues specifically for the research with the use of Google DialogFlow. An example of stimulus material is shown in the figure above. In addition, I created a questionnaire using Qualtircs. Using the grant from digicomlab, I recruited 200 participants from an online survey platform – Prolific. Participants were aged between 18 and 35 and living in the United Kingdom (UK). After receiving greetings from participants, the chatbot would be activated, introduce itself to the participants, ask the participants about their personal information (gender and age), cigarette use, and personal preferences for a new individual health insurance policy.
| Hypotheses | Findings |
| H1a: Contrastive explanation Explanation Quality | Not supported. |
| H1b: Contrastive explanation Source Credibility | Not supported. |
| H2a: Explanation Quality Attitude | Supported. |
| Explanation Quality Attitude Intention | Supported. |
| H2b: Source Credibility Attitude | Supported. |
| Source Credibility Attitude Intention | Supported.
|
| H3: Contrastive explanation x Anthropomorphic cues Source Credibility | Not supported. |
To test the hypotheses shown above, I analyzed a series of regression models using PROCESS macro for R. As you can see in the table, the study found no evidence of the effects of contrastive explanation, indicating that a contrastive explanation by a chatbot in recommending personal health insurance did not have a positive impact on perceived explanation quality and source credibility. The results also indicated that, in the context, anthropomorphic cues might not play a role in the effect of contrastive explanation on the user’s perception of source credibility. In addition, an investigation on the indirect impact on behavioral intentions discovered that explanation quality and source credibility had influences on user’s behavioral intentions to adopt the recommendation from the chatbot by way of a positive attitudinal outcome.
Although no evidence of the effects of contrastive explanation was found in the context of a chatbot in recommendation service, the results shed light on the importance of user’s perceptions of explanation quality and source credibility. The study operationalized explanation quality as the representational quality of an explanation by a chatbot and source credibility as perceived trustworthiness and expertness of the chatbot. Therefore, for the companies that intend to employ recommender chatbots for marketing purposes, the findings could help them to consider and identify determinants (e.g., explanation representation and chatbot expertness) of user’s acceptability of recommendation from a chatbot. Future studies should explore this further, by for example addressing which explanation representation styles in a chatbot context are positively associated with explanation quality and source credibility.