2. @DACORT
By Day By Night
Founder/Director of Twitter Addict, including
Alchemy Security
TweetStats
Web app security Twitter Timeline Viz
Security Product Dev TweepSearch
Twitter Profile Search
Firewall Visualization
3. TWEETSTATS?
GRAPHING TWITTER SINCE 2007
Visualize your Twitter timeline - tweetstats.com
Tweets per month/hour
Top replies and apps
TweetCloud
Trend monitoring since September ’08
Global Twitter usage since December ’08
Over 250M tweets (no content) logged
Thanks, Gnip (http://gnip.com)
4. CHALLENGES
1. Data, data, data
Twitter allows retrieval of up to 3,200 tweets/user
Whitelisting allows 20,000 API queries/hour
Not uncommon now for 20k + tweets
Disparate - timeline vs. friends/followers
2. User experience
It takes time to gather data, graph
13. USER EXPERIENCE
BACKGROUND QUEUES
Generate XML data
asynchronously
Activate charts via JavaScript
14. USER EXPERIENCE
K.I.S.S.
Tons of data, easy to get lost
Lots of features TweetStats _could_ have...
Important to keep the experience simple
And this doesn’t pay the bills ... yet. ;)
TweetStats v3 - More features? More analytics?
16. SCALING TWEETSTATS
WHY IT BROKE?
One server (affectionately named after @darkgracie)
Web server + Rails stack
Database w/150,000 tables
= #FAIL
Various other applications
Synchronous DB queries
Thousands of NKOTB Fans
17. SCALING TWEETSTATS
TYPICAL TWITTER TRAFFIC
Peaks and valleys
Influential tweet = immediate need for resources
18. SCALING TWEETSTATS
“FIXING” IT
Nods to:
Amazon EC2
Scalr.net
Architecture: Scaled!
Wallet: Empty.
20. DATA MINING
Other services - TweepSearch, TweetSum
Background Queues are King
TweetStats - BackgroundRB
TweepSearch - Workling
TweetSum - BackgroundJob
Why so many? Can’t judge without experience.
21. DATA MINING
VOLUME
TweepSearch - ~4.5M Twitter Profiles Indexed
Just a small fraction of actual users
TweetStats - Global twitter usage via Gnip
Previously - 2M updates/day
Now - ~10M updates/day
~9,000 updates/minute
aka ... my servers hate me.
22. DATA MINING
OPTIONS
Brute Force (historical)
Crawl profiles, timelines
Twitter Search - 1,500 results or 20-30(??) days
Official Twitter Streaming API
“spritzer”, “gardenhose”, “firehose” streams
Gnip - Not just Twitter!