Ironfan is the foundation for your Big Data stack, making provisioning and configuring your Big Data infrastructure simple. Spin up clusters when you need them, kill them when you don't, so you can spend your time, money, and engineering focus on finding insights, not getting your machines ready. Learn more at http://infochimps.com

1. Ironfan:
Principle-Driven
Scalable Ops Tools
http://github.com/infochimps-labs/ironfan

2. Ironfan
• Open-source core (Apache) with enterprise features
• http://github.com/infochimps-labs/ironfan
• Based on Chef (http://www.opscode.com/chef)

3. Principles
• Progressive
- match what you’d whiteboard
• Declaratory
- show all (and only) what’s interesting
• Coherent
- full state is versioned as a whole
• Decoupled
- minimize shared knowledge
• Helpful
- common things are simple
• Inevitable
- declarations imply actions

4. Ironfan:
your system diagram
made real

8. Declaratory:
simple stories, simply told

9. HBase Cluster
• HBase Master
• HBase RegionServers
Simple, right?

10. HBase Cluster
• one-ish:
• HBase master
• HBase secondary master
• Hadoop Namenode • many:
• Hadoop 2NN • HBase RegionServer
• Hadoop Jobtracker • HBase Stargate
• Zookeeper Quorum x3 • HBase Thrift
• Hadoop Datanode
• Hadoop Tasktracker

14. Decouple
Powerful Black Boxes
Beautiful Glue

15. Data Stores in Production
• HBase • MySQL
• ElasticSearch • Redis
• Cassandra • sqlite
• TokyoTyrant • whisper (graphite)
• SimpleDB • file system
• MongoDB • S3

16. Programs Used for This Talk
• Emacs • Skitch
• Keynote • finder
• Preview • flickr.com
• Chrome • google image search
• ruby (pry) • ssh

17. A Database is just a Bucket

18. Service Discovery:
game recognize game

20. gibbon-hadoop-namenode
gibbon-hadoop-jobtracker
ack ack ack
gibbon-nnjt-0
¿dónde está mi namenode?
gibbon-nnjt-0
gibbon-
worker-0

21. gibbon-hadoop-namenode
gibbon-hadoop-jobtracker
ack ack ack
gibbon-nnjt-0
¿dónde está mi namenode?
gibbon-nnjt-0
¿dónde está mi jobtracker?
gibbon-nnjt-0
gibbon-
worker-0

22. gibbon-hadoop-namenode
gibbon-hadoop-jobtracker
ack ack ack
gibbon-nn-0 gibbon-jt-0
¿dónde está mi namenode?
gibbon-nn-0
¿dónde está mi jobtracker?
gibbon-jt-0
gibbon-
worker-0

24. declaratory + decoupled =
Recapitulation
ex: DB Migrations
the Ironfan Way

26. alice
alice
bob
alice
bob
bob

27. Aspects:
Cross-cutting Concerns

28. Amenities
• Logs
• Ports
• Daemons
• Dashboards
• Libs / Executables

29. Amenities:
It’s good to be king

30. Amenities
• Logs: rotation, archiving
• Ports: remote monitoring (on and off)
• Daemons: process monitoring
• Dashboards: Dashboard dashboard
• Libs / Executables: Decoupled imports
• Versions: Shims, Discovery

31. Inevitability:
born this way

32. Chef:
Ironfan:

33. 10 lines

34. 10 lines, 5 flaws
can you spot all 5?

35. 10 lines, 5 flaws
hard coded
SECURITY HOLE
daemon-writeable
conf dir
fails on bsd
not DRY ‘0755’ not 755

36. Ironfan

37. Inevitability
must do this:
to get a log dir
so when you announce,
gibbon-hadoop-namenode
AWESOMENESS IS
INEVITABLE

38. Contract:
word is bond yo

40. Resource Sharing:
can’t we all just get along?

41. HBase
• Region Server (dominate)
• TaskTracker + Children (wussy)
• DataNode (moderate)
Hadoop
• DataNode (moderate)
• TaskTracker + Children (dominate)

42. Rails Apps > 5k LOC
considered Harmful
one big app many apps tax

43. Resource Sharing:
can’t we all just get along?

45. Monolithic Software
means Meetings

46. Meetings
are Death

47. Decentralize. Decouple.

48. It’s not about Performance
Fast
Resilient
Simple

49. jobs@infochimps.com
We’re Hiring
github.com/
infochimps-labs

50. ...

52. • System Diagram made real
• Decoupled Components connect
• Components Announce their capabilities
• Announcements enable Service Discovery
• Components announce cross-cutting Aspects
• Aspects enable zero-conf Amenities
• Announcement as Contract
• Contracts enable zero-conf specification testing
• Specs + monitoring enable zero-conf integration
testing
• Systems Bind to provisioned resources
• Resource Sharing