November 27, 2020

Using customer reviews to improve product — Sentiment Analysis

Using Sentiment on Analysis on customer’s data to understand their opinions, complaints etc, and then using it to make improvements in product or marketing

Written by

Harsh Gupta

Think of it, you have tonnes and tonnes of customers interaction with many touchpoints in the company. Be it email correspondence, chat, phone calls etc. And what if we told you just from those data points, we can predict -

  1. if the customer is going to churn
  2. when is it going to happen
  3. the next purchase they will do
  4. how much they will spend next and in their lifetime
  5. If they are satisfied
  6. and so on

Sentiment analysis (or opinion mining) is a mechanized method used by businesses to detect positive sentiments or negative sentiments in a text to understand the customers better and keep an eye on its reputation. Sentiment analysis helps us to identify the motive and emotions behind a purchase, gain many more insights and eventually predict the future response.

Let us take the example of a client, a television manufacturing company, we worked with a few months ago, to understand how opinion mining can help. We had the business problem to analyze the sentiment from reviews. They wanted to understand the contentment of user in product pricing and features. We conducted a very thorough analysis of the reviews and found out the features most liked, features most disliked, views on pricing and segments of users with which priorities (for example, certain users were more concerned about the pricing than the quality, some just need bare minimal features in tv). After this study, the manufacturer was able to understand how they need to pitch what type of television to which type of users. Also, the most unanimous features liked were upgrade permanently and the most unanimous disliked features were removed.

Types of Sentiment Analysis

Knowing the best type of sentiment analysis required for the business is essential. These are the following types -

Intent Analysis gives a deeper meaning to the customer’s review. It helps find out the actual intention of whether the client is interested in buying the product or service. There are different methods to find the intent using machine learning- the first one, rule-based and the second one is automatic. These methods can be used to interpret every customer’s reaction and find out the true meaning using patterns and then be used for marketing and change in approach.

Emotion detection analysis uses lexicons (a set of words that are either positive or negative) and algorithms (like Spacy) to find the natural feeling behind the feedback.

Customers give feedback in many ways. The simplest is when they are absolutely positive or negative. But it gets tricky when they give a review like this -

a) Sarcastic in nature

b) Positive opinion but in negative words, for example — “I will not buy the product again, but love most of the things about it.”

c) Negative opinion but in positive words — “This product may be the most hated one, but I still love it and recommend it to everyone”

Complexities behind Sentiment Analysis

Sentiment analysis is done using natural language processing (NLP), a branch of artificial intelligence that deals with the interaction between machines and humans. Understanding and analyzing emotions and feelings are difficult for machines. Some of the complexities are -

  • How close the subjects are to each other in data. How similar the sentences are. For instance, in television company data, some reviews were all about expensive pricing. These type of reviews can be classified as one subject or text cluster or text topics. Finding out subjectivity, which is, the similarity of opinions is called subjectivity analysis.
  • Factual or opinion — Most text is an opinion but some are factual as well. We need to be making a distinction here.
  • Identifying neutral — “I think that’s polished furniture.”, “I love that furniture.”, “I don’t think that will work for me.” The first sentence is clearly a neutral sentence. It is very important to not classify such text under positive or negative sentiments. They should be treated like another class altogether.

The process behind Sentiment Analysis (some statistics coming !)

  • Stemming — Grouping of word from the same stem. ‘Worked’ and ‘working’ are grouped together as one under ‘work’
  • Remove punctuations
  • Lowercase everything in text
  • Remove stop words such as ‘a’. ‘an’, ‘the’
  • Misspelt words — Correct the spellings of misspelt words and bring the same words together.
  • Works on parts of speech tagging — Each word is assigned the type of part of speech.
  • Removing the parts of speech — Depending on every use case, we may decide to drop certain parts of speech, like pronoun, adverb etc.
  • Using various libraries pre-built in R and Python, find out subjectivity, polarity, topics of text

Now the data is ready to do machine learning. If the dataset has other types of variables as well, be it categorical or numerical, that can be fed into modeling as well. Try out various models and in the end, we have predictions!

Later for visualizations, you can have word clouds and visualize topics to give the user an idea of what topics are they.

Later for visualizations, you can have word clouds and visualize topics to give the user an idea of what topics are they.

Word cloud

Words in Text Topic

All this sounds too complex?

No worries, we have made it possible using our one-click automation platform. Just upload the results and see all of this without writing a single line of code.

Arrow left icon.
Return to Blog