Now, we even have a symposium dedicated to it which was recently held at New York city with a good representation from vendors and researchers in this space. There are many companies who are in this space like Lexalytics, Brandwatch, Attensity, Scoutlabs, Istrategy labs, Sentiment 360, Saplo, Serendio and many others from text analytics and natural language processing space who are positioning themselves to take advantage of this promising market. Despite various claims, it is not an easy task to determine whose algorithm is better and how good is their machine learning and language processing modules. Right now, there are hardly any standards out there to compare them. I doubt that we will see any standards in near future also. Till this date, I have seen sentiment analysis being applied more in context of reputation management or understanding voice of customer like:
- Which conversations are relevant and active
- Which products are most talked about
- Sentiments behind each product
- Intentions to buy product
- Top issues, cries for help
- product suggestions
- Early warning of issues
Even the most optimistic of researchers and vendors agree that it is not an exact science and according to them even seventy five percent of accuracy is hard to get - basically, law of diminishing returns sets in if you want to stretch beyond it. Linguistic nuances or ambiguity of the language, culture and getting the right context is often cited as the reason which leads to lack of accuracy. Overall, it is still worth it to do the sentiment analysis when you have to do deal with piles of content in this web 2.0 and beyond world. Other than issue of accuracy there are many more challenges like:
- How do you do segmentation of the audience? I have a hard time understanding many sentiment analysis graph which doesn't tell me anything about the profile of people whose sentiment is being tracked.
- You do get a broader idea about the polarity of a sentiment but it is very difficult to understand the degree of emotion
- The industry is realizing that the sentiment analysis has to be used in conjunction with other research techniques though it is still not clear or defined about how it will be done in a repeatable manner?
- How do you track and interpret sentiments about a product on global basis? It is just not about supporting international languages. There are various instances where a product has done well in US but has failed in a different country or viceversa.
- Do you apply the same criteria of sentiment analysis whether it is a product, person, service, social or international issue?
- Technical Integration with CRM or business intelligence will not be an issue. But what should be the criteria to assign weightage to a sentiment?
- Sentiments can vary considerably if you measure them in different durations. In this article from BBC, it is much easier to measure the positive and negative sentiments associated with Gordon Brown in the UK elections debate during two hours of duration. It seems there were negative comments when he expressed his views about immigration. It just proves that there is a component of time dimension also in an opinion about a famous personality. Though, a sentiment about a product can't vary within hours. That too from the same person.
Overall, the risk is that this discipline shouldn't be oversold and needs to go beyond technologists. It is still at an early stage but it is going to stay because the business case is solid. It needs to be positioned as an aid to human analysis rather than a stand alone discipline. Someone rightly said about sentiment analysis - "I think of these tools like a metal detector, sure it beeps when it finds something, but it’s still up to you to expend the energy to dig in up to your elbows and find the nugget." For it to become mainstream, marketers, product management and analysts needs to work in tandem with technologists and develop methods to measure its effectiveness in the long run. For mainstream adaption, more customized use cases have to be thought of and this technology has to be applied in a very pragmatic and realistic way to succeed.
In my opinion, voice of a customer and reputation management is just too crowded and new areas like financial services seems promising. Though, there are already products like Reuters Newscope who is playing in the financial services market. Pharma is another area but I have heard that there are still open regulatory issues out there which needs to be sorted out. I often think that at some point in future, sentiment analysis can also be integrated with early warning systems or some flavor of predictive analytics.
The topic of sentiment analysis always reminds me of saying from Bertrand Russel, the founder of analytic philosphy, that "The fact that an opinion has been widely held is no evidence whatever that is not utterly absurd."