An overview on how we use Hadoop and other open source technologies to provide reporting and basic clustering services to AOL's websites.

1. The Data Layer
‘because data has needs’
Ian Holsman
Hadoop World, October 2010

2. Who Am I?
• Ian Holsman
• CTO of Relegence
• Started in open source in 2000 on the Apache Web Server
• Joined AOL in 2007
• I work in the ‘content’ side of AOL
2

3. AOL has
• 3 large (>100 boxes) hadoop clusters
• 1 in adverQsing
• 1 in search
• 1 in content
• I am talking today about the ‘content’ side of the house
3

4. Agenda
• The Opportunity
• How we addressed it
• Unexpected benefits and issues we had
• What we are doing today
4

5. It started with a quesQon
? can we do better than a ‘top stories’ link?

6. The Opportunity ‐ circa 2008
• Get more informaQon about our customers
• Increase recirculaQon
• Increase RPM of our pages
6

7. Which we translated into
• Build a be[er ‘related’ page module
• iniQally site‐specific
• but the plan was to make it site‐wide
7

8. How we addressed it

9. How we addressed it
• Custom Javascript injected onto the page so we can start
measuring things
• Custom web server modules to handle cookies over mulQple
domains
• Custom log processing infrastructure to push data onto HDFS
every 15m
• Map‐Reduce jobs to provide reports & create MySQL databases
• built a co‐visitaQon algorithm to produce related pages
9

10. Privacy
• We have tried our best to keep things anonymous from the start
• We don’t track IP‐level data, we translate to WOEIDs
• So we can’t tell you (or governments) what a parQcular IP did
• It’s not perfect
• We avoided pucng it on ‘sensiQve’ sites
10

11. IniQal architecture (July 2008)
11

12. Did it on the cheap
• 2‐3 person project
• Grabbed 50 ‘spare’ machines that were lying around
• Installed hadoop
• Put our ‘beacon’ on a site (AOL real‐estate)
• and away we went
• a ‘skunkworks’ with the blessing of the CIO
• minimized red‐tape.
12

13. in 2‐3 months
• we had infrastructure up
• we were processing page views & uniques
• we installed the beacon on other ‘small’ sites
• we had ‘data’ and a proof of concept that was meaningful for
business owners
13

14. We got people’s a[enQon
• Start doing basic reports for bebo
• 300m PV’s a day
• needed to move from skunkworks to a ‘real’ project.
14

15. Major issues
• Hadoop
• Map‐reduce was slow to write and inflexible
• Hadoop kept on hanging, both the name server and our custom push jobs would stall
• OperaQons
• how to move from 0.18 to 0.19 ?
• Jobs failing meant we were gecng paged, and restart‐ability was never really designed
• Felt like we were building our house on quicksand.
• we were running off factory‐defaults
• network wasn’t opQmized at ALL
• People
• zero experience going in
• people were learning by doing.
• lots of new things made fault detecQon ‘interesQng’
• our group started becoming a bo[leneck
• Map reduce hard to learn
15

16. OperaQonal Issues

17. OperaQonal issues
• Got ‘real’ machines
• put onto same switches/racks
• built the filesystem to be[er match how we used hadoop
• upgraded to 0.19 at same Qme
• took 48 hours to migrate
• Spent some Qme listening to experts
• tuned our cluster a bit be[er
• removed developer access to the ‘hadoop’ user
• SQll not a 100% “producQon” system
• but close enough for my liking
17

18. then Yahoo open sourced ‘PIG’

19. PIG fixed a lot of ‘people’ issues
• easy to use
• didn’t require much training
• enabled the system to be used by ‘regular’ channel developers
19

20. We felt like Alice going through the rabbit hole
http://www.flickr.com/photos/spam/3355824586/

21. The data unlocked innovaQons
• The channel developers knew their data
• They used it in ways we never expected
21

22. Some cool tools
Built off the data
22

23. The heatmap tool
23

24. The heatmap tool
24

25. Aol’s Traffic Exchange
25

26. The URL viewer
• Get stats about any URL
• Page views
• Google Searches
• Referrers
• Exits
• Custom parameters
• Geographic regions
• Have similar tool for
anonymous userIDs
26

27. Some useful applicaQons
using simple aggregation techniques and mahout
27

28. Shopping recommendaQons
• UQlizes Mahout
• UQlizes custom parameters
• Be[er click through rate
than external vendors
• Can apply technique to any
product‐based channel
28

29. User recommendaQons
{
"algo": "recoByPVPartDay",
"UnauthId": "007e3dc60bbe11dfadba39f9fdfe11b5#2",
"url-pv-Info": [
{
"pv": 54.0,
"url": "joystiq.com/2010/07/14/sengoku-basara-samurai-heroes-sticks-six-swords-into-north-amer"
},
{
"pv": 49.0,
"url": "joystiq.com/2010/07/14/maxis-hiring-development-director-for-online-simulation-game"
},
{
"pv": 35.0,
"url": "joystiq.com/2010/07/14/how-to-play-sin-and-punishment-star-successor"
},
{
"pv": 10.0,
"url": "news.bigdownload.com/2010/07/14/bioware-co-founder-were-working-on-smaller-mmo-type-games"
},
{
"pv": 3.0,
"url": "news.bigdownload.com/2010/07/14/natural-selection-2-alpha-test-coming-july-26-for-special-editio"
}
]
}
29

30. User Interests
{
"tags": [
{
"tag": "Video games",
"score": 435.0
},
{
"tag": "Internet search engines",
"score": 96.0
},
{
"tag": "Internet",
"score": 84.0
}
],
"unauthId": "007e3dc60bbe11dfadba39f9fdfe11b5"
}
30

31. Where we are today

32. The current deliverables
• Get more informaQon about our customers
• Increase recirculaQon
• Increase RPM of our pages
• Build metrics into our plaxorm
• What works on pages
• How are we performing
• Build intelligence on the page
• CollaboraQve filtering
• Product recommendaQons
• Top‐K type lists
• Make it closer to real Qme
• not the focus of this talk
32

33. What data are we processing?
• Beacon Web servers
• Tracking beacon injected into the HTML page via custom javascript
• Tracks
• Page views
• Page clicks
• Custom event that the content developer wants
• Tracks standard things like referrers, and user agents, and LocaQon
• Developer can add custom parameters to tell us about the page
• needed to write a custom module to generate anonymous user ids + 3rd party domain tracking
• custom module to map IP#’s to geographic WOEID‐based locaQons
• Ad impressions
• User viewed a campaign
• Integrate it with campaign manage to determine actual revenue
• URL context (through relegence)
• We can determine who & what a arQcle is about
• through relegence, similar to what OpenCalais does
33

34. The Data Layer Infrastructure today
Web Page Publishing
Platforms
Advertising MySQL
Webservers Relegence
Reporting
Beacon Analytics
Hadoop
Webservers Tools
Cassandra Cassandra Redis MySQL
34