The Web and the Collective intelligence - How to use Collective Intelligence techniques to ensure that your web application can extract valuable data from its usage and deliver that value right back to the users.

1. How to use Collective Intelligence techniques to ensure that your
web application can extract valuable data from its usage and deliver
that value right back to the users.
MODULE 1
Hélio Teixeira
New York, May 2010

2. Web users are undergoing a
transformation…
 Users are expressing themselves. This expression may be in the
form of:
 sharing their opinions on a product or a service through reviews or
comments; through sharing and tagging content; through
participation in an online community; or by contributing new
content.
 This increased user interaction and participation gives rise to data
that can be converted into intelligence in your application. The
use of collective intelligence to personalize a site for a user, to aid
him in searching and making decisions, and to make the application
more sticky are cherished goals that web applications try to fulfill.

3. What do these two companies have in
common?
Netflix
 They both drew new conclusions and created new business
opportunities by using sophisticated algorithms to combine data
collected from many different people.
 The ability to collect information and the computational power to
interpret it has enabled great collaboration opportunities and a better
understanding of users and customers.

4. Wisdom of the Crowds…
 “Under the right circumstances, groups are remarkably intelligent,
and are often smarter than the smartest people in them.”
 “If the process is sound, the more people you involve in solving a
problem, the better the result will be.”
 A crowd’s collective intelligence will produce better results than
those of a small group of experts if four basic conditions are met.

5. That “wise crowds” are effective when they’re
composed of individuals who…
 Have diverse opinions;
 When the individuals aren’t afraid to express their opinions;
 When there’s diversity in the crowd; and
 When there’s a way to aggregate all the information and use it in the
decision-making process.

6. What is collective intelligence?
 Collective intelligence is an active field of research that predates
the web. Scientists from the fields of sociology, mass behavior, and
computer science have made important contributions to this field.
 When a group of individuals collaborate or compete with each other,
intelligence or behavior that otherwise didn’t exist suddenly emerges;
this is commonly known as collective intelligence.
 The actions or influence of a few individuals slowly spread across
the community until the actions become the norm for the community.
 This forms a self-reinforcing feedback loop, commonly known as a
network effect, which enables wider adoption of the service.

7. What is collective intelligence?
 Although the expression may bring to mind ideas of group
consciousness or supernatural phenomena, when technologists use
this phrase they usually mean the combining of behavior,
preferences, or ideas of a group of people to create novel insights.
 Collective intelligence was, of course, possible before the Internet.
You don’t need the Web to collect data from disparate groups of
people, combine it, and analyze it. One of the most basic forms of
this is a survey or census. Collecting answers from a large group of
people lets you draw statistical conclusions about the group that no
individual member would have known by themselves. Building new
conclusions from independent contributors is really what collective
intelligence is all about.

8. Collective intelligence on the Web
 Although methods for collective intelligence existed before the
Internet, the ability to collect information from thousands or even
millions of people on the Web has opened up many new possibilities.
 At all times, people are using the Internet for making purchases,
doing research, seeking out entertainment, and building their own
web sites.
 All of this behavior can be monitored and used to derive information
without ever having to interrupt the user’s intentions by asking him
questions.
 There are a huge number of ways this information can be processed
and interpreted.

9. Collective intelligence of users in essence is
 The intelligence that’s extracted out from the collective set of
interactions and contributions made by your users.
 The use of this intelligence to act as a filter for what’s valuable in
your application for a user—This filter takes into account a user’s
preferences and interactions to provide relevant information to the
user.

10. To apply collective intelligence in your
application. You need to...
 1 - Allow users to interact with your site and with each other,
learning about each user through their interactions and
contributions.
 2 - Aggregate what you learn about your users and their
contributions using some useful models.
 3 - Leverage those models to recommend relevant content to a
user.

13. What Is Machine Learning?
 Machine learning is a subfield of artificial intelligence (AI) concerned
with algorithms that allow computers to learn.
 What this means, in most cases, is that an algorithm is given a set of
data and infers information about the properties of the data—and
that information allows it to make predictions about other data that it
might see in the future.
 This is possible because almost all nonrandom data contains
patterns, and these patterns allow the machine to generalize. In
order to generalize, it trains a model with what it determines are the
important aspects of the data.

14. Limits of Machine Learning
 Machine learning is not without its weaknesses. The algorithms vary
in their ability to generalize over large sets of patterns, and a pattern
that is unlike any seen by the algorithm before is quite likely to be
misinterpreted.
 While humans have a vast amount of cultural knowledge and
experience to draw upon, as well as a remarkable ability to
recognize similar situations when making decisions about new
information, machine-learning methods can only generalize based
on the data that has already been seen, and even then in a very
limited manner.

15. Real-Life Examples
 Google, Amazon, eBay, Last.fm, Netflix, Pandora, e muitos outros
 Prediction markets are also a form of collective intelligence. One of
the most well known of these is the Hollywood Stock Exchange
(http://hsx.com), where people trade stocks on movies and movie
stars.
 Other Uses for Learning Algorithms:
 Biotechnology - Advances in sequencing and screening technology
have created massive datasets of many different kinds, such as DNA
sequences, protein structures, compound screens, and RNA expression.
Machine-learning techniques are applied extensively to all of these kinds
of data in an effort to find patterns that can increase understanding of
biological processes.

16. Real-Life Examples
 Financial fraud detection - Credit card companies are constantly
searching for new ways to detect if transactions are fraudulent. To this
end, they have employed such techniques as neural networks and
inductive logic to verify transactions and catch improper usage.
 Machine vision - Interpreting images from a video camera for military or
surveillance purposes is an active area of research. Many machine-
learning techniques are used to try to automatically detect intruders,
identify vehicles, or recognize faces. Particularly interesting is the use of
unsupervised techniques like independent component analysis, which
finds interesting features in large datasets.
 Product marketing - For a very long time, understanding demographics
and trends was more of an art form than a science. Recently, the
increased ability to collect data from consumers has opened up
opportunities for machine-learning techniques such as clustering to better
understand the natural divisions that exist in markets and to make better
predictions about future trends.

17. Real-Life Examples
 Supply chain optimization - Large organizations can save millions of
dollars by having their supply chains run effectively and accurately
predict demand for products in different areas. The number of ways in
which a supply chain can be constructed is massive, as is the number of
factors that can potentially affect demand. Optimization and learning
techniques are frequently used to analyze these datasets.
 Stock market analysis - Ever since there has been a stock market,
people have tried to use mathematics to make more money. As
participants have become ever more sophisticated, it has become
necessary to analyze larger sets of data and use advanced techniques to
detect patterns.
 National security - A huge amount of information is collected by
government agencies around the world, and the analysis of this data
requires computers to detect patterns and associate them with potential
threats.

18. Collaborative Filtering
 A collaborative filtering algorithm usually works by searching a large
group of people and finding a smaller set with tastes similar to yours.
It looks at other things they like and combines them to create a
ranked list of suggestions. There are several different ways of
deciding which people are similar and combining their choices to
make a list.
 Collecting Preferences
 Recommending Items
 Matching Products
 Item-Based Filtering

20. Benefits of collective intelligence
 Applying collective intelligence to your application impacts it in the
following manner:
 Higher retention rates —The more users interact with the application, the
stickier it gets for them, and the higher the probability that they’ll become
repeat visitors.
 Greater opportunities to market to the user —The greater the number of
interactions, the greater the number of pages visited by the user, which
increases the opportunities to market to or communicate with the user.
 Higher probability of a user completing a transaction and finding
information of interest —The more contextually relevant information that a
user finds, the better the chances that he’ll have the information he needs to
complete the transaction or find content of interest. This leads to higher click-
through and conversion rates for your advertisements.
 Boosting search engine rankings —The more users participate and
contribute content, the more content is available in your application and
indexed by search engines. This could boost your search engine ranking and
make it easier for others to find your application.

21. Harnessing Collective Intelligence to transform
from content-centric to user-centric applications
 Prior to the user-centric revolution, many applications put little
emphasis on the user. These applications, known as content-centric
applications, focused on the best way to present the content and
were generally static from user to user and from day to day.
 User-centric applications leverage Collective Intelligence to
fundamentally change how the user interacts with the web
application.
 User-centric applications make the user the center of the web
experience and dynamically reshuffle the content based on what’s
known about the user and what the user explicitly asks for.

22. User-centric applications are composed of the
following four components:
 Core competency —The main reason why a user comes to the
application.
 Community — Connecting users with other users of interest, social
networking, finding other users who may provide answers to a user’s
questions.
 Leveraging user-generated content — Incorporating generated
content and interactions of users to provide additional content to
users.
 Building a marketplace — Monetizing the application by product
and/or service placements and showing relevant advertisements.

25. LinkedIn application leverages the four
components of user-centric applications:
 Core competency — Users come to the site to connect with others and
build their professional profiles.
 Community — Users create connections with other users; connections
are used while looking up people, responding to jobs, and answering
questions asked by other users. Other users are automatically
recommended as possible connections by the application.
 User-generated content — Most of the content at the site is user-
generated. This includes the actual professional profiles, the questions
asked, the feed of actions—such as a user updating his profile,
uploading his photograph, or connecting to someone new.
 Marketplace —The application is monetized by means of
advertisements, job postings, and a monthly subscription for the power-
users of the system, who often are recruiters. The monetization model
used is also commonly known asfreemium9— basic services are free
and are used by most users, while there’s a charge for premium services
that a small minority of users pay for.

27. Classifying intelligence
 The three types of intelligence
 Explicit information that the user provides in the application.
 Implicit information that a user provides either inside or outside the
application and is typically in an unstructured format.
 Intelligence that’s derived by analyzing the aggregate data collected.
This piece of derived intelligence is shown on the upper half of the
triangle, as it is based on the information gathered by the other two parts.

28. Data comes in two forms:
 Structured data has a well defined form, something that makes it
easily stored and queried on. User ratings, content articles viewed,
and items purchased are all examples of structured data.
 Unstructured data is typically in the form of raw text. Reviews,
discussion forum posts, blog entries, and chat sessions are all
examples of unstructured data.

29. Explicit intelligence
 REVIEWS AND RECOMMENDATIONS
 VOTING
 TAGGING

30. Implicit intelligence
 Information relevant to your application may appear in an
unstructured free-form text format through reviews, messages,
blogs, and so forth.
 A user may express his opinion online, either within your application
or outside the application, by writing in his blog or replying to a
question in an online community. Thanks to the power of search
engines and blog-tracking engines, this information becomes easily
available to others and helps to shape their opinions.
 You may want to augment your current application by aggregating
and mining external data. For example, if your area is real estate
applications, you may want to augment your application with
additional data harvested from freely available external sites, for
example, public records on housing sales, reviews of schools and
neighborhoods, and so on.

32. Derived intelligence
 DATA AND TEXT MINING
 CLUSTERING AND PREDICTIVE ANALYSIS
 INTELLIGENT SEARCH
 RECOMMENDATION ENGINE
END MODULE 1

33. Thanks!
Helio Teixeira is founder of novoDialogo{.
He is an analyst, digital communication expert,
public speaker and editor of the Chapa Branca
Blog.
@helioteixeira
+55 82 9901 5090
heliolteixeira@gmail.com
helioteixeira@novodialogo.com.br
http://comunicacaochapabranca.com.br
http://novodialogo.com.br