Data Modeling for NoSQL
•
4 likes•1,830 views
Video and slides synchronized, mp3 and slide download available at http://bit.ly/16I9vsI. Tony Tam shares tips for modeling data with MongoDB for a fast and scalable system based on his experience migrating billions of records from MySQL to MongoDB. Filmed at qconsf.com. Tony Tam is the CEO and technical co-founder of Wordnik, where he leads development efforts of the site's innovative word navigation system. He is a member of the MongoDB Masters group and has lead the Swagger API Framework open-source initiative. He was a founder and SVP of Engineering at Think Passenger, Lead Engineer at Composite Software and a software development lead at Galileo Labs.
Recommended
Recommended
More Related Content
Viewers also liked
More from C4Media
More from C4Media (20)
Recently uploaded
Recently uploaded (20)
Data Modeling for NoSQL
- 1. Data Modeling for NoSQL Tony Tam @fehguy
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /data-modeling-mongodb
- 3. Presented at QCon San Francisco www.qconsf.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 5. Why Modeling Matters! • NoSQL => no joins! • What replaces joins?! • Hierarchy! • Duplication of data! • Different models for querying, indexing! • Your optimal data model is (probably) very different than with relational! • Simpler! • More like you develop!
- 6. Stop Thinking Like This!! endless layers of abstraction (and misery)
- 7. Hierarchy before NoSQL! • Simple User Model!
- 8. Hierarchy before NoSQL! • Tuned Queries! • Write some brittle SQL:! • “select user.id, … inner join settings on …! • Pick out the fields and construct object hierarchy (this gets nasty, fast)! • (outer joins for optional values?)! • Object fetching! • Queries follow object graph, PK/FK! • 5 queries to fetch object in this example!
- 10. Hierarchy with NoSQL! • JSON structure mapped to objects! • Fetch json from MongoDB**! • Unmarshall into objects/tuples! • Use it! Using JSON4S
- 11. Hierarchy with NoSQL! Focus on your Software, not DB layer!
- 12. Hierarchy with NoSQL! • Write operations! • Atomic upsert (create, update or fail)! ! • Saves all levels of object atomically! • Reduces need for transactions!
- 13. Hierarchy with NoSQL! • Write operations! • Atomic upsert (create, update or fail)! ! • Saves all levels of object atomically! • Reduces need for transactions! All or Convenienc not magic
- 14. Unique Identifiers in your Data! • Relational design => PK/FK! • Often not “meaningful” identifiers for data! • User Data Model!
- 15. Unique Identifiers in your Data! • Relational design => PK/FK! • Often not “meaningful” identifiers for data! • User Data Model! Unique by username
- 16. Unique Identifiers in your Data! • Words! Ensured to be constant
- 17. Data Duplication ! • Without Joins, what about SQL lookup tables?! • Duplication of data in NoSQL is required! • Trade storage for speed!
- 18. Data Duplication ! • Without Joins, what about SQL lookup tables?! • Duplication of data in NoSQL is required! • Trade storage for speed! …Can move logic to app
- 19. Data Duplication! • Many fields don’t change, ever! • But… many do! • New decisions for the developer!! • Often background updates!
- 20. Data Duplication! • Many fields don’t change, ever! • But… many do! • New decisions for the developer!! • Often background updates! How often does this change?
- 22. Reaching into Objects! • Incredible feature of MongoDB! • Dot syntax safely** traverses the object graph!
- 23. Inner Indexes! • Convenience at a cost! • No index => table scan! • No value? => table scan! • No child value? => table scan! • Table scan with big collection?! • Can’t index everything!! 96GB of Indexes?
- 24. Inner Indexes! • This will should drive your Data Model! • Sparse Data test! Even with only 2000 non-empty values!
- 25. Adding & Modifying! • Append in mongo is blazing fast! • “tail” of data is always in memory! • Pre-allocated data files! • Main expense is “index maintenance”! • Some marshalling/unmarshalling cost**! • Modifying? Object growth! • Pre-allocation of space built in collection design!
- 26. Adding & Modifying! • Each object has allocated space! • Exceed that space, need to relocate object! • Leaves “hole” in collection! • Large increases to documents hurts your overall performance! • Your data model should strive for equally- sized objects as much as possible!
- 27. Retrieval! • Many same rules apply as relational! • Indexes ! • complex/inner or not! • Indexes in RAM? Yes! • Cardinality matters! • New(ish) considerations! • Complex hierarchy not free! • Marshalling ó unmarshalling!
- 29. Marshalling & Unmarshalling! • All you can eat from your Data Model?! • Techniques have tremendous impact! • Development ease until it matters! • 50% speed bump with manual mapping! Only demand what you can consume!
- 30. Making the most of _id! • Indexes matter! • Tailor your _id to be meaningful by access pattern! • It’s your first defense when auto-sharding! • Date-driven data?! • Monotonically _id value! • Ensures recent data is “hot”!
- 31. Making the most of _id! • Other time-based data techniques! • Flexibility in querying!
- 32. Making the most of _id! • Other time-based data techniques! • Flexibility in querying! Case- sensitive REGEX is
- 33. Making the most of _id! • Hot indexes are happy indexes! • Access should strive for right bias! • Random access with large indexes hit disk 17 15
- 34. Your Data Model! • NoSQL gets you started faster! • Many relational pain points are gone! • New considerations (easier?)! • Migration should be real effort! • Designed by access patterns over object structure! • Don’t prematurely optimize, but know where the knobs are!