Blog Back ArrowThird I Implements Chatbot Analytics for Worldwide Teahouse Chain Resulting in 60% Reduction in Customer Service Calls

Our client is an Australian tea brewery chain with over 2500 stores across 38 countries. With the  expansion in stores and customers, it was challenging for our client to provide consistent and continuous customer service and support. The number of customers who came to the website surged along with their queries. Third I has built a chatbot that enables uninterrupted customer service by responding to these queries to provide immediate solutions around the clock. This chatbot solution features an analytics and AI system to automate and continuously improve their customer service experience.

Background + Business Need

As our client expanded into new countries, there was an influx of customer queries coming from different time zones. This made it difficult to provide a consistent customer service experience to those customers who were asking questions. In order to offer uninterrupted customer service to their customers, they needed a solution that could help to provide a better customer service experience to customers who came to the website, emailed, or called to ask a question.

Third I has worked on this project in two phases: first, we have built and deployed a chatbot for our client that would use automation to provide answers to common customer questions. These questions typically pertain to membership, loyalty points, franchise opportunities, and employment opportunities.

Second, we have built a chatbot analytics solution to monitor the performance of the chatbot. The client needed an analytics solution that could provide accurate statistics related to customer interaction, chatbot performance, chatbot accuracy, and customer satisfaction. Our analytics solution has helped our client to make timely, necessary changes in customer support questionnaires and chatbot responses to gradually improve the chatbot’s accuracy from 40% to 80% within the first 2 months of implementation. Following the continuous improvement of the chatbot, the client has seen a 60% reduction in customer service calls and emails related to common queries within the first month of implementation, saving them time to focus on inquiries that truly needed their attention.

Technical Challenges

  • Creating the data model on the chatbot logs that consists of semi-structured JSON data and is stored in Azure Cosmos DB was necessary to integrate with Power BI.
  • Capturing customer session-based data on chatbot interactions is required to enable users to distinguish between new customer sessions and new queries.
  • The analytics dashboard requires the calculation of chatbot accuracy based on the chatbot’s responses to customer queries. This is calculated on the basis of the intent score values that were between 0 and 1 generated by Azure Cognitive Services (LUIS).

Our Approach

Third I experts have developed a data model in Power BI for analysis. Chatbot logs are pulled from Azure Cosmos DB (NoSQL database) using the Power BI connector to import data into the data model. We then have developed a dashboard for the chatbot logs to visually monitor the accuracy of the chatbot.

This was done through the following steps:

  • A user-defined function was created to distribute the intent score into buckets of percentage. Chatbot accuracy was then calculated based on these intent score values. In scenarios with low chatbot accuracy, the least accurate responses were triggered by Azure Data Factory in a new table. We then automated calibration of these triggered responses in Azure Cognitive Services (LUIS) via REST APIs to improve accuracy.
  • We integrated Google Analytics and Google Tag Manager into the client’s website. This helped us capture the “customer sessions” by defining custom dimensions in Google Analytics as this is not a predefined dimension in Google Analytics. This helps us to identify new customers as well as new queries from the sessions of chatbot interactions.
  • Datasets of support queries were created which contained the standard support questionnaires, conversational AI library, and data from Google Analytics.
  • These datasets are used in Power BI. DAX expressions are used to calculate some of the defined metrics (e.g. user interactions, engaged users) for the chatbot analytics dashboard.

Chatbot Analytics Architecture

The chatbot analytics solution required highly specific KPIs as the capabilities of the chatbot were to be measured through these metrics. After a thorough deep-dive into understanding the metrics that were to be measured, the following KPIs have been implemented:

Chatbot Performance Metrics

  • Self-Service Rate: This metric identifies the number of customers who received the information they were looking for from the chatbot without any human input/intervention.
  • User Interactions: This metric captures the total number of conversations, highlighting the usage of the chatbot over a certain period of time.
  • Retention Rate: This refers to the percentage of users that return to using the chatbot in a given time frame.
  • User Satisfaction: This captures the users’ responses to survey questions asking about satisfaction with the chatbot to measure its overall success.
  • Fall-Back Rate: This was the percentage of the number of times the chatbot has failed or has reached a near-failure situation.


User Metrics

  • Total Users: This metric captures the number of people using the chatbot
  • Active Users: Active users are the people who read a message in the chatbot in a defined time frame.
  • Engaged Users: This is the number of users who communicate with the chatbot.
  • New Users: People who interact with the chatbot for the first time within a defined time frame.

Conversation Metrics

  • Total Conversations: Number of conversations that start and complete successfully on a given day.
  • New Conversations: Number of new conversations that start on a given day.
  • Missed Messages: The messages from users that the chatbot cannot process/respond to.

Chatbot Performance Dashboard

Chatbot Analytics Dashboard

Third I has built a chatbot analytics dashboard that visualizes the data from the chatbot’s performance in Power BI.

The report provides an overview of the chatbot’s interaction data, chatbot accuracy, the performance of the chatbot, and insights about customer satisfaction. This report is also used to create exception reporting on chatbot performance. Instant email alerts are sent out to the chatbot deployment team for certain issues, such as consistently low accuracy of the chatbot.

Tech Stack: Azure Cosmos DB, Power BI, Azure Cognitive Services (LUIS), Azure Data Factory


  • Linkedin
  • Twitter
  • Facebook