This presentation given by Flip Kromer and Huston Hoburg on March 24, 2014 at the MongoDB Meetup in Austin.
Vayacondios is a system we're building at Infochimps to gather metrics on highly complex systems and help humans make sense of their operation. You can think of it as a "data goes in, the right thing happens" machine: send in facts from anywhere about anything, and Vayacondios will promptly process and syndicate them to all consumers. Producers don't have to (or get to) worry about the needs of those who will use the data, or the details of transport, storage, filtering or anything else: the data will go where it needs to go. Each consumer, meanwhile, finds that everything they need to know is available to them, on the fly or on demand, without crufty adapters or extraneous dependencies. They don't have to (or get to) worry about the distribution of their sources, the tempo of update, or how the data came to be.
Vayacondios was built for our technical ops team to monitor all the databases and systems they superintend, but it suggests a better way to build database driven applications of any kind. The quiet tyranny of developing against a traditional database has left us with many bad habits: not duplicating data, using models that serve the query engine not the user, assembling application objects from raw parts on every page refresh. Combining streaming data processing systems with distributed datastores like MongoDB let you do your query on the way _in_ to the database -- any number of queries, decoupled, of any complexity or tempo. The resulting approach is simpler, fault-tolerant, and scales in terms of machines and developers. Most importantly, your data models are purely faithful to the needs of your application, uncontaminated by differing opinions of other consumers or by incidentals of the robots that gather and process and store the data.
2. Infochimps
• Big Data Platform for Large Companies
• Cloud::Queries (ElasticSearch,MongoDB,HBase)
• Cloud::Hadoop (Dynamic Hadoop)
• Cloud::Streams (Storm+Trident)
• Managed Service, Enterprise Features
• Recently sold to CSC, and it’s quite awesome
• We’re Hiring (natch)
3. Vayacondios
• Built for ourVisibility Stack…
• … but we think it has wider use
!
• “Data Goes In, the Right Thing Happens”
• Prompt, Comprehensive and Faithful
12. Systems
• Anatomical Systems: Circulatory, Immune, etc
• Interventions: Drugs, Surgeries, …
• Course of Treatment: topline progress indicators
• Diagnosis
• Practitioner
• Medical Devices
13. ICU
• Model the patient, not the data source
• Highlight Interactions among systems
• Highlight Interactions among numbers
• Broaden your view of “systems”
16. System != Machine
• Whole-System MongoDB:
• Machines it runs on,Volumes it uses
• Systems writing to it
• Applications and Collections
• Data Files, Logs, Repl Sets, Oplog, Arbiters
• Codebase repo, Cookbooks, Configuration
• Issue Tracker Tickets, Change Events
17. Operations
• Cognitive model for Humans, not from Robots
• Go beyond the Time-series Graph
• Highlight Interactions
• Link to Systems that write to this DB
• Link to Github for Repos & Cookbooks
• Drill into System
• Issues in Issue Tracker
• Broaden your view of “systems”
20. Vayacondios
• Visibility Stack for our operations team
• Open-sourcing this summer
• Internals in Ruby
• Access anywhere (HTTP or log file)
• MongoDB (but now please forget that fact)
21.
22. Cognitive Model
• MongoDB:
• is_a Data store
• has_many Network Services
• has_many Daemons
• has_many Machines
• has_manyVolumes
• has_many Collections
• …etc
25. Faithful
• Whiteboard rule: how do folks talk about system?
• If you need it,it’s in the system
Prompt
• As fast as joint laws of Economics & Physics allow
Comprehensive
30. Faithful to Source
• crap data => well-formed data
• uniform JSON-ready hash
• syntax cleaned up
• semantically unchanged
• encouraged to model it, but let Wookiee win
31.
32. Write Contract
• Vaya Con Dios,“Go with God”. As the kingdom
of heaven is unknowable, so is further fate of data:
• How used
• By Whom
• How Processed
• Where Stored
40. Model/Presenter/View
• More targets that just dashboard!
• Splunk+PagerDuty Alerts
• Cucumber tests
• Auditing reports (Security, Good Manners)
41. System Checks
• Correctness, Consistency
• Attached Directly to the Model
• No worthwhile distinction between
QA (integration tests) and live Alerts
• Drive Splunk+Pager Duty for Alerts
• Author Cucumber specs(!) for QA tests
44. Inevitability
• If configured and reported, consistency checks
• If reported, dashboard exists
• If is_a generic system (eg filesystem), gets
correctness tests (eg “capacity < 75%”)
• If system A discovers system B:
• dashboard has link from A to B
• connectivity & security checks from A to B
45. Interaction
• Monitoring systems do a terrible job here
• Hard sources of failure:
• Drift
conceived != realized
• Interaction
unexpected consequences
• Change
oops
47. Application Design
• Visibility into complex systems:
• Biography of raw parts (raw Model) =>
Reporter (Presenter) =>
Summary of Systems (View-ready Model)
• Database-driven Application
• Model =>
Presenter =>
View
50. Blog: Views Author Page
Post Page
Index Page
PostSynopsisReport
PostReport
UserReport
CommentReport
51. “Query on the way In”
!
• New/Updated Post: Update Post triggers…
• Update PostReport
• Update SynopsisReport
• Update UserReport
52. “Query on the way In”
!
• User fullname changes: Update User triggers…
• Update UserReport
• Update their SynopsisReports
• Update their PostReports
• Update their CommentReports
54. Faithful
• Whiteboard rule: how do folks talk about system?
• If you need it,it’s in the system
Prompt
• As fast as joint laws of Economics & Physics allow
Comprehensive
55. Faithful
• Single concern: subject of the biography
• look at what’s offered,look at what reports need
Prompt
• Run as often as needed (not your concern)
Comprehensive
56. Faithful
• One Reporter per Application (*) & Topic
• USCE Method:Utiliz’n,Saturat’n,Connections,Errors
Prompt
• Run as often as needed (not your concern)
Comprehensive