Brisk: Truly peer-to-peer hadoop Talk at SFJava http://bit.ly/jqClhK

1. Brisk: Truly peer­to­peer Hadoop
srisatish.ambati AT gmail.com
Apache Cassandra/OpenJDK
@srisatish

2. Brisk: Hive + Hadoop + Cassandra
@srisatish

3. Map Reduce
@srisatish

4. Have large sets of data & you can
work on small pieces in parallel.
@srisatish

5.
Map Reduce
@srisatish

6. Multi­core map reduce framework,
Kunle, et al
@srisatish

7.
Parallel Execution View @srisatish

8.
@srisatish

9.
@srisatish

10. JobTracker
NameNode
HDFS
@srisatish

11. Write­once­read­many!
File once created, written & closed need change
@srisatish

12. Move computation, not data
@srisatish

13.
@srisatish

14. DataNodes: Read, Write Blocks
@srisatish

15. NameNode:
Single Master node
Single Machine Address space
Single Point of failure

16. When “it” does not fit in a single node!
… Enter the distributed dragon!
Enter the Cassandra:
High Scale
Peer­to­peer
@srisatish

17. NameNode
DataNodes

18. One­kind­of­node!

19. Cassandra:
High Scale
Peer­to­peer
@srisatish

20. Portfolio Demo
Low latency
Live tick prices for stocks.
Batch Analytics
Historical EOD prices.
Value at Risk.
http://www.datastax.com/docs/0.8/brisk/brisk_demo

21. Demo URLs (good for this demo only)
http://ec2­50­19­4­143.compute­1.amazonaws.com:8888/opscenter/index.html
http://ec2­67­202­12­176.compute­1.amazonaws.com:50030/jobdetails.jsp?job
http://ec2­50­19­4­143.compute­1.amazonaws.com:8983/portfolio/

22. Dynamo, 2007
Bigtable, 2006
OSS, 2008
Incubator, 2009 TLP, 2010

23. Y
Key “C”
A
W
Cassandra:
High Scale
U
Peer­to­peer F
No SPOF
T
L
P
@srisatish

24. “dynamic” columnfamilies
Following
zznate driftx: thobbs:
driftx
thobbs zznate:
jbellis driftx: mdennis: pcmanus: thobbs: xedin: zznate:

28. Brisk
@srisatish

29. Brisk
HowStuffWorks version
@srisatish

30. YDH security edition (soon to be Apache)
Apache Hive – Access via SQL like
Cassandra 0.8
CQL Interface
Apache Thrift

31. Use ColumnFamilies
inode
sblock
@srisatish

32.
String keyspace = “cfs”;
CfDef cf = new CfDef();
cf.setName(inodeDefaultCf);
cf.setComparator_type("BytesType");
…
cf.setName(sblockDefaultCf);
cf.setKey_cache_size(1M);
cf.setComment(
"Stores blocks of information associated with a inode");
cf.setKeyspace(keyspace);
@srisatish

34. Consistency: R + W > N
"brisk.consistencylevel.read", "QUORUM";
"brisk.consistencylevel.write", "QUORUM";
@srisatish

35. Hadoop:
job tracker, task tracker
@srisatish

36. BriskSnitch:
brisk nodes, cassandra nodes
@srisatish

37. BriskSimpleSnitch.java
if(TrackerInitializer.isTrackerNode)
{
myDC = BRISK_DC;
logger.info("Detected Hadoop trackers
are enabled, setting my DC to " + myDC);
}
else
{
myDC = CASSANDRA_DC;
logger.info("Looks like Vanilla Cassandra
nodes, setting my DC to " + myDC);
}
@srisatish

38. Hive: SQL­like access
cli, hwi, jdbc, metastore
Pushdown predicates (v beta2)
@srisatish

39. hive> CREATE TABLE invites (foo INT, bar
STRING)PARTITIONED BY (ds STRING);
hive> LOAD DATA LOCAL INPATH
'$BRISK_HOME/resources/hive/examples/files
/kv2.txt' OVERWRITE INTO TABLE invites
PARTITION (ds='2008­08­15');
hive> SELECT count(*), ds FROM invites
GROUP BY ds;
http://www.datastax.com/docs/0.8/brisk/about_hive @srisatish

40. ETL
Real­time
Cassandra CFs
DataCenters
Scale
@srisatish

41.
@srisatish

42. No me in team!
● Ben Coverston ● Jonathan Ellis
● Ben Werther ● Michael Allen
● Brandon Williams ● Mike Bulman
● Cathy Daw ● Nate McCall
● Daria Hutchinson ● Nick M Bailey
● Eric Gilmore ● Patricio Echague
● Jackson Chung ● Tyler Hobbs
● Jake Luciani ● SriSatish Ambati
● Joaquin Casares ● Yewei Zhang
@srisatish

43.
100­node Brisk Cluster on Opscenter
@srisatish

44. Dynamo, 2007
Bigtable, 2006 +
OSS, 2008
Incubator 2009
TLP, 2010
Cassandra
+ +
Brisk

45. Git started:
git clone git@github.com:riptano/brisk.git
http://www.datastax.com/product/brisk
Getting Started via Brisk AMI.
Thank You.
@srisatish

46. References
● MapReduce: Simplified Data Processing on Large Clusters, 2004, Jeffrey Dean and
Sanjay Ghemawat, http://bit.ly/googmr_pdf
● Multi­core MapReduce, Kunle, et al. http://bit.ly/iRJd1n
@srisatish