Original Source Here
Harvey increases responsiveness for customer support agents
Customer support agents respond to hundreds of emails daily. Amongst those emails, there are bound to be repetitive queries. This causes the support agent to draft the same reply and risks missing replying to certain queries. Besides, relying on manually drafting the replies also denies them the opportunity to learn an experienced agent’s response.
At Hiver, we recognized the need for customer support agents to be more responsive when responding to emails. In this article, we’ll share how we use Email Templates to empower customer support agents in their job.
What are Email Templates in Hiver?
Hiver’s Email Templates are professionally structured email replies for addressing customer queries that are repetitive. They help agents to respond customer queries quickly so they can dedicate their attention to other complex issues. Hiver’s Email Templates let agents focus on people and not waste time finding the right message structure.
The problems with conventional email templates
Although email templates are a great way to breeze through common customer queries for support agents, these come with their own set of challenges.
Template discovery: Support agents find it difficult to search through a long list of email templates when addressing customer queries.
Template usage: Sometimes, a few templates are needed to sufficiently address a query, but the agent has difficulty choosing the right parts. Furthermore, the current setup for email templates does not support this user flow.
More number of clicks: It takes more clicks in order to search email templates and insert that email templates into the inbox in order to reply to the customer query.
Hiver’s new approach to Email Templates
In spite of all these above problems, Hiver observed that support agents are already using email templates in their existing setup. Hence, we believe it’s viable to make email templates more intuitive to improve the agent’s productivity further.
After thorough discussions, we concluded that it would be more helpful for agents If Harvey could analyze the customer query and recommend appropriate email templates to the agent as a prompt. It will reduce the manual effort of finding, searching, and inserting email templates.
As a privacy-first company, Hiver continues to uphold our customer’s email privacy throughout this development. We ensure that users have full control of their data and that we’re fully transparent on how we use such data.
So let’s see how Harvey helps customer agents by recommending email templates in real-time.
How does Harvey analyze customer queries?
Customer emails contain information that can be helpful in choosing an email template. With that said, emails also consist of unnecessary information, which is unhelpful to agents when making such decisions.
Harvey uses a Question extractor module which takes care of all non-informative text of the email body and extracts questions out of it. If the question extractor modules fail to identify questions, Harvey takes the whole email body, analyzes, and makes suggestions. The implementation of the Question extractor code is given below
How does Harvey make recommendations?
Harvey compares and uses the similarity between customer queries and email templates when making recommendations. We assume that email templates tend to be similar to the queries most of the time. Therefore, we use the following techniques to help us make accurate recommendations.
Syntactic Similarity– It’s the similarity based on syntax, i.e how many words are common in between the sentences.
Semantic Similarity — It is a type of similarity based on meaning and intent.
Word Embedding- It maps words from the text (word)to N-dimensional space so that words with the same meaning will lie nearer to each other and vice versa.
Sentence Embedding:- It will aggregate all the embeddings of constituting words by some mathematical function and then generate an N-dimensional array. Feel free to visit this blog link to learn more about sentence embeddings.
In our project, we are using all-mpnet-base-v2 sentence transformers model from Hugging face . It maps sentence and paragraph to 768 dimension-dense vector space and can be used for tasks like clustering and semantic search. Below code snippet demonstrates how this model encodes sentences.
Now that we have converted our sentences into vectors, How can we measure the similarity between vectors? There are a couple of metrics to measure vector similarities like Euclidean distance, Hamming distance, cosine similarity, and dot product. Cosine similarity works pretty well in the case of semantic similarity hence we go ahead with cosine similarity in the project.
Cosine similarity at scale:- Building an in-house solution in order to store the embeddings of email templates and getting top k similar email templates to the customer query on the fly doesn’t scale that well in production. Hence we use AWS-managed OpenSearch (specifically KNN plugin) in order to store embeddings of email templates and query email templates in real time. To read more KNN plugins from OpenSearch, visit this link.
Performance measurement pipeline
Performance measurement pipeline is an essential part of every Machine Learning project. It helps to evaluate the performance of your machine learning systems and take necessary business decisions. Just like every machine learning project, Harvey’s email template suggestions have their own recall and precision. Let’s define recall and precision for our project
Precision:- Precision can be calculated by dividing the total number of correct suggestions (templates on which the user either copies or inserts the entire templates) by the total number of times we displayed suggestions.
Recall:- Recall can be calculated by dividing the total number of correct suggestions by the total number of emails that can be resolved by templates (which includes the total number of correct suggestions and no of times emails that could have been resolved by email templates)
Note- As we don’t know how many emails could have been resolved by email templates, It is not feasible to calculate recall in real-time.
Weighted Precision– It’s the precision that is calculated by taking template rank (as weight) into account. It tends to give a higher weightage to the template that appears at the top than the bottom.
We carried out a small PoC on the Shared Mailbox that our support team uses. First, we identified all the emails that had been replied to by email templates and then passed those emails through our systems to generate predictions. We calculated weighted and Unweighted precision at different threshold values (threshold means it is the minimum cosine similarity below which we don’t consider the suggestions).
Note:- In OpenSearch the cosine similarity values ranges from 0 to 2, instead of -1 to 1.
How does Harvey surface template recommendations?
First of all, in order to view suggestions for customer emails, the shared mailbox admin has to enable Harvey email template suggestions. After that, whenever a customer email arrives in the agent inbox, our backend service will process this customer email and try to generate the top few relevant recommendations which are greater than our desired threshold. Then, the backend services store these suggestions in a database.
Once an agent clicks on the reply button, A backend API will be called from the frontend to fetch suggestions from the database. If the number of suggestions is greater than 0, then the agent will see the Harvey icon with the tooltip.
Once the agent clicks on the Harvey icon, the suggestions prompt will appear
In case of no suggestions, Harvey icon will greyed out, No suggestions will be displayed
Every machine-learning system is incomplete without a feedback loop. It helps researchers to evaluate and compare model performance with the benchmark performance and to decide on retraining the system.
In the Harvey template suggestions project, we will constantly be monitoring the system’s precision (as discussed in performance measurement systems). If we observe any degradation in system performance, we will try to identify the root cause within the system.
Apart from this precision, there are a couple of Gainsight events, such as (TemplateSuggestionsEnabled, TemplateSuggestionsDisabled, InsertWholeTemplate, CopiedPartialSuggestedTemplate … etc), that we keep on tracking as feedback.
Hiver is continuously developing innovative features powered by Harvey (Hiver’s AI bot) We have deployed the thank you detection feature for thousands of our users. Learn more about thank you detection in this article.
By the way, if you have a passion for solving real-world problems with the latest technologies or are thrilled by building scalable and distributed systems, feel free to check out exciting opportunities at Hiver (Hiver Jobs).
Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot