1) The document discusses a presentation given by Dr. John Feland, CEO of Argus Insights, at a sentiment analysis symposium on checking the accuracy of sentiment analysis provided by third parties. 2) Argus Insights analyzed over 1900 user reviews of tablets from major online retailers and found that a major sentiment analysis provider's scores aligned with the actual sentiment distribution of reviews only 7% of the time and the correct sentiment (positive, negative, neutral) 51% of the time. 3) After adjusting the third party's scores to match the actual distribution of sentiments, its accuracy improved significantly with scores in the correct bin 33% of the time originally but nearly 5 times better after adjusting, and

1. Looking at the world through
NLP-colored glasses
Dr. John Feland, CEO, Argus Insights
Sentiment Analysis Symposium
Research Workshop 8 Nov 2011
Tweet it! @argusinsights

2. If you can not measure it, you can
not improve it.
-Lord Kelvin
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 2

3. Always Check your math
-my First Grade Teacher

4. Trust, but Verify
- Russian Proverb
(Доверяй, но проверяй)

5. Most Corporate Clients Treat Sentiment as a Black Box
Raw musings of Numerical Market
Magic
consumers Sentiment
Sentiment
Analysis
• Corporate Decision Makers blinded by precision
• Using these numbers to drive performance
• What is the impact of getting it wrong?
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 5

6. How we checked the math
Unnamed
sentiment
provider
• Over 1900 user reviews taken from the Tablet Market
• Sources include most of the major online retailers
• Leveraged best-in-class sentiment provider
– Scored entire verbatim using tuned sentiment model
– Usage is outside of vendor’s recommended use
• Surprised at the results
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 6

7. Initial compared sentiment scores directly
0.6
Actual Distribution SV Distribution
Percentage Distribution
0.5
0.4
0.3
0.2
0.1
0
-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1
Consumer Sentiment
• All scales normalized (linearly) to -1 to +1
• SV Sentiment Score within Review Range 7% of the time
• SV Sentiment within correct Positive, Neutral or Negative
territory 51% of the time
• SV results are biased towards the negative for Tablets
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 7

8. Scaled so that SV distro matches User distro
0.6
Actual Distribution SV Distribution
P{ercentage of Distribution
0.5
0.4
0.3
0.2
0.1
0
-1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1
Consumer Sentiment
• Trimmed and scaled SV results to match actual
distribution
• Signficant performance improvements
– Correct Bin: 33% -> almost 5X improvement
– Correct Sign: 74% -> almost 50% improvement
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 8

9. Comparing actual to scored by product
Reviews Raw SV Score Corrected SV Score
1 ASUS Eee Slate RIM PlayBook RIM PlayBook
2 Iconia Andr Tablet iPad 2 iPad 2
3 Motorola Xoom Iconia Andr Tablet Iconia Andr Tablet
4 iPad 2 Samsung Galaxy Tab 7 Motorola Xoom
5 RIM PlayBook Motorola Xoom Samsung Galaxy Tab 7
6 Samsung Galaxy Tab 7 ASUS Eee Slate ASUS Eee Slate
7 Iconia Win Tablet Iconia Win Tablet Iconia Win Tablet
8 HP TouchPad HP TouchPad HP TouchPad
• Calculated sentiment could lead to the wrong
competitive understanding
• Selecting the wrong target can be disastrous for the
enterprise
• Responsibility to our clients demand transparency
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 9

10. New sentiment models - where the math checks out
• Argus Insights uses customer reported sentiment
• Proven predictor of market performance
• Full transparency to clients, enabling them to trust and verify
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 10

11. With great power comes
great responsibility
-Voltaire
© 2011 Argus Insights, Inc. Confidential: Do Not Distribute Page 11

12. Questions?