SlideShare uma empresa Scribd logo
1 de 54
Who, what, how & in the cloud :)

                                       Mark	
  Hillick
                                     mark@10gen.com
                      http://www.10gen.com	
  &	
  http://www.mongodb.org
Tuesday 3 July 2012
Agenda



Tuesday 3 July 2012
Agenda
                      • Who
                      • What
                      • Example Deployments
                      • EC2 Notes
                      • EC2 Best Practices
                      • Further Tuning
Tuesday 3 July 2012
Who, what & where!



Tuesday 3 July 2012
Who?
       • 10gen -> company behind MongoDB
       • Created by Dwight & Eliot
       • MongoDB is open-source & community is key
       • Offices in California, NY, Dublin, London & Sydney
       • $73.4 million in VC funding

Tuesday 3 July 2012
What? (1)
       • Powerful, flexible, scalable, fast data store
       • Document-oriented
           • Embedded docs & arrays
       • Scale out
       • Easy to start & develop with

Tuesday 3 July 2012
What? (2)
       • Indexing
       • Stored Javascript
       • Javascript Shell
       • Fixed-size collections
       • File Storage

Tuesday 3 July 2012
On What?
       • Linux
         • 10gen repos for Debian & Red Hat like distros
       • Windows
       • MAC OS (primarily dev)
       • Raspberry PI :)


Tuesday 3 July 2012
Today’s Challenges




Tuesday 3 July 2012
Where?




Tuesday 3 July 2012
The Past & Now




Tuesday 3 July 2012
The Past & Now




Tuesday 3 July 2012
A bit deeper



Tuesday 3 July 2012
Getting Started
       • Document -> basic unit of data ~ a row in RBMS
       • Collection -> schema equivalent of a table
       • Single instance can have multiple dbs
       • JS Shell -> administration
       • Event document has special, unique key -> _id

Tuesday 3 July 2012
Collection

                      > show collections
                          file_tweets
                         mini_tweets
                        system.indexes




Tuesday 3 July 2012
Document

         { "_id" : 122, "user_id" : 123456, "total" : 100, "items" :
            [ { "item_name" : "my_item_one", "price" : 20 },
             { "item_name" : "my_item_one", "price" : 20 },
          { "item_name" : "my_item_three", "price" : 30 } ] }




Tuesday 3 July 2012
The Shell




Tuesday 3 July 2012
Drivers
       • Python
       • Ruby
       • JavaScript [node.js]
       • PHP
       • Perl
       • Scala
       • And more......
       • http://docs.mongodb.org/manual/applications/drivers
       (Wiki’s & Docs)

Tuesday 3 July 2012
Indexes
       • Data structure collecting info
       • Enhance query performance
       • Optimiser sorts through & orders docs
       • db.file_tweets.ensureIndex({“user_id” : 1})
       • http://www.mongodb.org/display/DOCS/Indexes

Tuesday 3 July 2012
Wire Protocol

       • Simple socket-based, request-response style
       protocol

       • MongoDB stores “BSON” documents, which are
       binary encoded serialisation of JSON documents

       • TCP /IP socket on port 27017 - fire & forget

       •http://www.mongodb.org/pages/viewpage.action?pageId=131351

Tuesday 3 July 2012
Deployments



Tuesday 3 July 2012
Deployment Types

       • Master-Slave (not recommended)
       • Replica Sets
       • Shards
       • Some notes on EC2 deployments


Tuesday 3 July 2012
Replica Set Configurations

                       Primary     Primary      Primary

                      Secondary   Secondary    Secondary

                       Arbiter    Secondary    Secondary

                      (Minimum)    (Typical)
                                               Secondary

                                               Secondary


Tuesday 3 July 2012
Replica Set Info
       • Asynchronous replication (single primary)
       • Automatic failover
       • App-level definition of “write replication”
       • Secondary nodes can replicate with a slaveDelay
       • Secondary nodes can be hidden
       • Max of 12 nodes, with 7 voting
Tuesday 3 July 2012
Sharding
                                                                                 config DB


                      mongos     mongos       mongos        mongos               config DB


                                                                                 config DB




           Primary             Primary          Primary              Primary

           Secondary           Secondary        Secondary            Secondary


           Secondary           Secondary        Secondary            Secondary




Tuesday 3 July 2012
Sharding Notes
       • Each “shard” usually a Replica Set (same options)
       • Copy of meta data stored in-memory by mongos
       • Data split into chunks, using range based shard key
       • Chunks may be migrated between shards
       • New chunks created by “splitting” old chunks

Tuesday 3 July 2012
Shard Server in EC2 (1)
                      Category/Impact   Low   Medium   High

                        Disk Speed                      x

                       Disk Capacity                    x

                           RAM                          x

                           CPU                  x



Tuesday 3 July 2012
Shard Server in EC2 (2)
       • MongoDB designed for OS defaults on 64 bit
       instance

       • Use standard virtual memory page size
       • Raise “nofiles” ulimit (20,000)
       • Use RAID10 & modern f/s -> ext4, xfs etc
       • Use “noatime” mount option
Tuesday 3 July 2012
Server in EC2 (1)
       • kernel >= 2.6.23/2.6.25 respectively
       • Readahead: how much more to read than what you
       asked for

       • If too high => possible performance impact
       • Set to 0 on EBS devices
       • Set to desired value on RAID device
Tuesday 3 July 2012
Server in EC2 (2)
       • Turn off atime on filesystem (pre-2.6.30 especially)
       • RAID 10 is recommended everywhere
         • mitigates slow EBS volumes (fail the bad volume)
       • Do not use large VM pages
       • Do configure swap to prevent OOM Killer

Tuesday 3 July 2012
Config Server
       • Meta Data for shard stored in ConfigDB
       • Copy of meta data stored in-memory by mongos
       • Config DB cluster is *not* a replica set -> run 3!!
       • If config server goes down then:
         • no splits and migrates
         • new mongos cannot be started
         • running mongos can still use cache to route r/w
Tuesday 3 July 2012
Config Server in EC2 (1)
                       Category/Impact   Low   Medium   High

                         Disk Speed       x

                        Disk Capacity     x

                            RAM                  x

                            CPU           x



Tuesday 3 July 2012
Config Server in EC2 (2)

       • Use Raid10


       • Use 64 bit instance
         • Can run on shard servers


Tuesday 3 July 2012
Mongos in EC2 (1)
                      Category/Impact   Low   Medium   High

                        Disk Speed

                       Disk Capacity

                           RAM                  x

                           CPU           x



Tuesday 3 July 2012
Mongos in EC2 (2)

       • Often run on application servers


       • 32 bit mongos ok with 64-bit mongod



Tuesday 3 July 2012
Arbiter in EC2 (1)
                      Category/Impact   Low   Medium   High

                        Disk Speed       x

                       Disk Capacity     x

                           RAM           x

                           CPU           x



Tuesday 3 July 2012
Arbiter in EC2 (2)
       • Can use micro instance
         • Elections may be slower


       • Can use instance store
         • Still want backups :)


Tuesday 3 July 2012
HA in EC2

       • Replica Sets



       • EC2 Availability Zones


Tuesday 3 July 2012
DR in EC2

       • Replica Sets



       • EC2 Regions


Tuesday 3 July 2012
Security
       • Turn on authentication
       • Create a key between shards
       • EC2 Security Groups
         • Can reference other Security Groups
       • EC2 Regions
       • Follow SDLC in coding your app

Tuesday 3 July 2012
Monitoring
           • Links in with Cacti, Nagios, Munin-Node etc.
           • MMS - > it’s free




Tuesday 3 July 2012
Mongo Ports
       • mongos, unsharded mongod : 27017
       • sharded mongod : 27018
       • config mongod : 27019
       • http port : 28017


Tuesday 3 July 2012
Some More EC2 Notes



Tuesday 3 July 2012
Instances Guidelines (1)
     • Use 64-bit only, 32-bit is not recommended
     • Primary/Secondary should be equal*
     • High CPU is not necessary
     • High Memory for large mongod instances
     • Network capacity is also IO capacity (EBS)

Tuesday 3 July 2012
Instances Guidelines (2)
       • Note the trade-offs - memory/network
         • m1.large to m2.xlarge = 2x Mem, 0.5x Network
       • Do not use micro except for testing & config
       • m1.medium is usually sufficient for config DB
       • m1.small can be used for Arbiters


Tuesday 3 July 2012
Backups
         • EBS Snapshots - RAID complicates things
         • Single EBS volume, with journaling means:
           • No fsync & lock required
         • Similar applies to LVM snapshots
         • EC2
         • General
Tuesday 3 July 2012
EC2/MongoDB Best Practices


       • https://wiki.10gen.com/display/DOCS/Amazon+EC2

       • https://wiki.10gen.com/display/DOCS/Production+Notes




Tuesday 3 July 2012
node.js



Tuesday 3 July 2012
node.js

       • server-side written in javascript


       • orginally written for push web apps

       • created by Ryan Dahl


Tuesday 3 July 2012
Sample node.js code
                       var express = require('express'),
                         Db = require('mongodb').Db,
                         Server = require('mongodb').Server,
                         Connection = require('mongodb').Connection;

                       var host = 'localhost';
                       var port = Connection.DEFAULT_PORT;
                       var db = new Db('node-mongo-examples',
                                new Server(host, port, {}), {native_parser:false});
                       var app = express.createServer();

                       app.get('/', function(req, res){
                           res.send('Hello World');
                       });

                       db.open(function(err, db) {
                         if(err) throw err
                         app.listen(8124);
                       });




Tuesday 3 July 2012
node.js with mongo

       • https://github.com/christkv/node-mongodb-native



       • http://www.nodebeginner.org



Tuesday 3 July 2012
Need Help?



Tuesday 3 July 2012
• MongoDB Google User Group
       • New MongoDB Docs & Old MongoDB Docs
       • Presentations
       • If you’re curious :)
       Image Source: http://www.cannotstartoutlook.com/wp-content/uploads/2012/06/outlook-problems-help.jpg



Tuesday 3 July 2012
Credits


       • Credit must go to @comerford, @jonnyeight &
       @mikefiedler as I borrowed some of the knowledgeable
       slides :)




Tuesday 3 July 2012

Mais conteúdo relacionado

Mais procurados

So you want to liberate your data?
So you want to liberate your data?So you want to liberate your data?
So you want to liberate your data?Mogens Heller Grabe
 
Introduction to MongoDB (from Austin Code Camp)
Introduction to MongoDB (from Austin Code Camp)Introduction to MongoDB (from Austin Code Camp)
Introduction to MongoDB (from Austin Code Camp)Chris Edwards
 
Mysql to mongo
Mysql to mongoMysql to mongo
Mysql to mongoAlex Sharp
 
Intro to GemStone/S
Intro to GemStone/SIntro to GemStone/S
Intro to GemStone/SESUG
 
Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...
Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...
Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...CUBRID
 
Rich internet application development using the dojo toolkit
Rich internet application development using the dojo toolkitRich internet application development using the dojo toolkit
Rich internet application development using the dojo toolkitalexklaeser
 
Congratsyourthedbatoo
CongratsyourthedbatooCongratsyourthedbatoo
CongratsyourthedbatooDave Stokes
 
High-Performance Storage Services with HailDB and Java
High-Performance Storage Services with HailDB and JavaHigh-Performance Storage Services with HailDB and Java
High-Performance Storage Services with HailDB and Javasunnygleason
 
An introduction to CouchDB
An introduction to CouchDBAn introduction to CouchDB
An introduction to CouchDBDavid Coallier
 
This is redis - feature and usecase
This is redis - feature and usecaseThis is redis - feature and usecase
This is redis - feature and usecaseKris Jeong
 
Introduction To MongoDB
Introduction To MongoDBIntroduction To MongoDB
Introduction To MongoDBYnon Perek
 
ModeShape 3 overview
ModeShape 3 overviewModeShape 3 overview
ModeShape 3 overviewRandall Hauch
 
An Overview of ModeShape
An Overview of ModeShapeAn Overview of ModeShape
An Overview of ModeShapeRandall Hauch
 

Mais procurados (15)

So you want to liberate your data?
So you want to liberate your data?So you want to liberate your data?
So you want to liberate your data?
 
Introduction to MongoDB (from Austin Code Camp)
Introduction to MongoDB (from Austin Code Camp)Introduction to MongoDB (from Austin Code Camp)
Introduction to MongoDB (from Austin Code Camp)
 
Mysql to mongo
Mysql to mongoMysql to mongo
Mysql to mongo
 
Mysql
MysqlMysql
Mysql
 
Intro to GemStone/S
Intro to GemStone/SIntro to GemStone/S
Intro to GemStone/S
 
Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...
Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...
Database Sharding the Right Way: Easy, Reliable, and Open source - HighLoad++...
 
Rich internet application development using the dojo toolkit
Rich internet application development using the dojo toolkitRich internet application development using the dojo toolkit
Rich internet application development using the dojo toolkit
 
Congratsyourthedbatoo
CongratsyourthedbatooCongratsyourthedbatoo
Congratsyourthedbatoo
 
High-Performance Storage Services with HailDB and Java
High-Performance Storage Services with HailDB and JavaHigh-Performance Storage Services with HailDB and Java
High-Performance Storage Services with HailDB and Java
 
An introduction to CouchDB
An introduction to CouchDBAn introduction to CouchDB
An introduction to CouchDB
 
This is redis - feature and usecase
This is redis - feature and usecaseThis is redis - feature and usecase
This is redis - feature and usecase
 
NoSQL Technology
NoSQL TechnologyNoSQL Technology
NoSQL Technology
 
Introduction To MongoDB
Introduction To MongoDBIntroduction To MongoDB
Introduction To MongoDB
 
ModeShape 3 overview
ModeShape 3 overviewModeShape 3 overview
ModeShape 3 overview
 
An Overview of ModeShape
An Overview of ModeShapeAn Overview of ModeShape
An Overview of ModeShape
 

Destaque

PHP Loves MongoDB - Dublin MUG (by Hannes)
PHP Loves MongoDB - Dublin MUG (by Hannes)PHP Loves MongoDB - Dublin MUG (by Hannes)
PHP Loves MongoDB - Dublin MUG (by Hannes)Mark Hillick
 
Peeling back your Network Layers with Security Onion
Peeling back your Network Layers with Security OnionPeeling back your Network Layers with Security Onion
Peeling back your Network Layers with Security OnionMark Hillick
 
CTF: Bringing back more than sexy!
CTF: Bringing back more than sexy!CTF: Bringing back more than sexy!
CTF: Bringing back more than sexy!Mark Hillick
 
Scareware Traversing the World via Ireland
Scareware Traversing the World via IrelandScareware Traversing the World via Ireland
Scareware Traversing the World via IrelandMark Hillick
 
Chandra pratapdwivedi [31.05.16]
Chandra pratapdwivedi [31.05.16]Chandra pratapdwivedi [31.05.16]
Chandra pratapdwivedi [31.05.16]C.P. DWIVEDI
 
Roger Carrillo Resume 2015 - PROOF (2)
Roger Carrillo Resume 2015 - PROOF (2)Roger Carrillo Resume 2015 - PROOF (2)
Roger Carrillo Resume 2015 - PROOF (2)Roger T. Carrillo, AIA
 
website status and seo assessment altec 2015
website status and seo assessment altec 2015website status and seo assessment altec 2015
website status and seo assessment altec 2015Manzur Ashraf
 
Ava ova morales rojas_1101
Ava ova morales rojas_1101Ava ova morales rojas_1101
Ava ova morales rojas_1101mariamar99
 
Characteristics of a Muslim (Bangla)
Characteristics of a Muslim (Bangla)Characteristics of a Muslim (Bangla)
Characteristics of a Muslim (Bangla)Manzur Ashraf
 
fully automated nursing system.PDF
fully automated nursing system.PDFfully automated nursing system.PDF
fully automated nursing system.PDFMahmoud Wanis
 
Siber Güvenlik Eğitiminde Uluslararası CTF Yarışmaları
Siber Güvenlik Eğitiminde Uluslararası CTF YarışmalarıSiber Güvenlik Eğitiminde Uluslararası CTF Yarışmaları
Siber Güvenlik Eğitiminde Uluslararası CTF YarışmalarıDr. Emin İslam Tatlı
 
BGA CTF Ethical Hacking Yarışması Çözümleri
BGA CTF Ethical Hacking Yarışması ÇözümleriBGA CTF Ethical Hacking Yarışması Çözümleri
BGA CTF Ethical Hacking Yarışması ÇözümleriBGA Cyber Security
 

Destaque (13)

PHP Loves MongoDB - Dublin MUG (by Hannes)
PHP Loves MongoDB - Dublin MUG (by Hannes)PHP Loves MongoDB - Dublin MUG (by Hannes)
PHP Loves MongoDB - Dublin MUG (by Hannes)
 
Peeling back your Network Layers with Security Onion
Peeling back your Network Layers with Security OnionPeeling back your Network Layers with Security Onion
Peeling back your Network Layers with Security Onion
 
CTF: Bringing back more than sexy!
CTF: Bringing back more than sexy!CTF: Bringing back more than sexy!
CTF: Bringing back more than sexy!
 
Scareware Traversing the World via Ireland
Scareware Traversing the World via IrelandScareware Traversing the World via Ireland
Scareware Traversing the World via Ireland
 
Chandra pratapdwivedi [31.05.16]
Chandra pratapdwivedi [31.05.16]Chandra pratapdwivedi [31.05.16]
Chandra pratapdwivedi [31.05.16]
 
Roger Carrillo Resume 2015 - PROOF (2)
Roger Carrillo Resume 2015 - PROOF (2)Roger Carrillo Resume 2015 - PROOF (2)
Roger Carrillo Resume 2015 - PROOF (2)
 
website status and seo assessment altec 2015
website status and seo assessment altec 2015website status and seo assessment altec 2015
website status and seo assessment altec 2015
 
Ava ova morales rojas_1101
Ava ova morales rojas_1101Ava ova morales rojas_1101
Ava ova morales rojas_1101
 
Muhannad cv
Muhannad cvMuhannad cv
Muhannad cv
 
Characteristics of a Muslim (Bangla)
Characteristics of a Muslim (Bangla)Characteristics of a Muslim (Bangla)
Characteristics of a Muslim (Bangla)
 
fully automated nursing system.PDF
fully automated nursing system.PDFfully automated nursing system.PDF
fully automated nursing system.PDF
 
Siber Güvenlik Eğitiminde Uluslararası CTF Yarışmaları
Siber Güvenlik Eğitiminde Uluslararası CTF YarışmalarıSiber Güvenlik Eğitiminde Uluslararası CTF Yarışmaları
Siber Güvenlik Eğitiminde Uluslararası CTF Yarışmaları
 
BGA CTF Ethical Hacking Yarışması Çözümleri
BGA CTF Ethical Hacking Yarışması ÇözümleriBGA CTF Ethical Hacking Yarışması Çözümleri
BGA CTF Ethical Hacking Yarışması Çözümleri
 

Semelhante a MongoDB - Who, What & Where!

Scaling with mongo db (with notes)
Scaling with mongo db (with notes)Scaling with mongo db (with notes)
Scaling with mongo db (with notes)emiltamas
 
MongoDB Pros and Cons
MongoDB Pros and ConsMongoDB Pros and Cons
MongoDB Pros and Consjohnrjenson
 
MongoDB 2.4 and spring data
MongoDB 2.4 and spring dataMongoDB 2.4 and spring data
MongoDB 2.4 and spring dataJimmy Ray
 
Lessons Learned Migrating 2+ Billion Documents at Craigslist
Lessons Learned Migrating 2+ Billion Documents at CraigslistLessons Learned Migrating 2+ Billion Documents at Craigslist
Lessons Learned Migrating 2+ Billion Documents at CraigslistJeremy Zawodny
 
Codeworks'12 Rock Solid Deployment of PHP Apps
Codeworks'12 Rock Solid Deployment of PHP AppsCodeworks'12 Rock Solid Deployment of PHP Apps
Codeworks'12 Rock Solid Deployment of PHP AppsPablo Godel
 
Deployment Strategy
Deployment StrategyDeployment Strategy
Deployment StrategyMongoDB
 
The Care + Feeding of a Mongodb Cluster
The Care + Feeding of a Mongodb ClusterThe Care + Feeding of a Mongodb Cluster
The Care + Feeding of a Mongodb ClusterChris Henry
 
Why databases cry at night
Why databases cry at nightWhy databases cry at night
Why databases cry at nightMichael Yarichuk
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDBRavi Teja
 
Deployment Strategies
Deployment StrategiesDeployment Strategies
Deployment StrategiesMongoDB
 
Leveraging MongoDB: An Introductory Case Study
Leveraging MongoDB: An Introductory Case StudyLeveraging MongoDB: An Introductory Case Study
Leveraging MongoDB: An Introductory Case StudySean Laurent
 
Cassandra Community Webinar: From Mongo to Cassandra, Architectural Lessons
Cassandra Community Webinar: From Mongo to Cassandra, Architectural LessonsCassandra Community Webinar: From Mongo to Cassandra, Architectural Lessons
Cassandra Community Webinar: From Mongo to Cassandra, Architectural LessonsDataStax
 
Meetup#2: Building responsive Symbology & Suggest WebService
Meetup#2: Building responsive Symbology & Suggest WebServiceMeetup#2: Building responsive Symbology & Suggest WebService
Meetup#2: Building responsive Symbology & Suggest WebServiceMinsk MongoDB User Group
 
Scaling with MongoDB
Scaling with MongoDBScaling with MongoDB
Scaling with MongoDBMongoDB
 
Challenges with MongoDB
Challenges with MongoDBChallenges with MongoDB
Challenges with MongoDBStone Gao
 

Semelhante a MongoDB - Who, What & Where! (20)

Scaling with mongo db (with notes)
Scaling with mongo db (with notes)Scaling with mongo db (with notes)
Scaling with mongo db (with notes)
 
MongoDB
MongoDBMongoDB
MongoDB
 
MongoDB Pros and Cons
MongoDB Pros and ConsMongoDB Pros and Cons
MongoDB Pros and Cons
 
MongoDB 2.4 and spring data
MongoDB 2.4 and spring dataMongoDB 2.4 and spring data
MongoDB 2.4 and spring data
 
Lessons Learned Migrating 2+ Billion Documents at Craigslist
Lessons Learned Migrating 2+ Billion Documents at CraigslistLessons Learned Migrating 2+ Billion Documents at Craigslist
Lessons Learned Migrating 2+ Billion Documents at Craigslist
 
Deployment
DeploymentDeployment
Deployment
 
Codeworks'12 Rock Solid Deployment of PHP Apps
Codeworks'12 Rock Solid Deployment of PHP AppsCodeworks'12 Rock Solid Deployment of PHP Apps
Codeworks'12 Rock Solid Deployment of PHP Apps
 
Deployment Strategy
Deployment StrategyDeployment Strategy
Deployment Strategy
 
The Care + Feeding of a Mongodb Cluster
The Care + Feeding of a Mongodb ClusterThe Care + Feeding of a Mongodb Cluster
The Care + Feeding of a Mongodb Cluster
 
Why databases cry at night
Why databases cry at nightWhy databases cry at night
Why databases cry at night
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
 
Deployment Strategies
Deployment StrategiesDeployment Strategies
Deployment Strategies
 
Mongodb
MongodbMongodb
Mongodb
 
Leveraging MongoDB: An Introductory Case Study
Leveraging MongoDB: An Introductory Case StudyLeveraging MongoDB: An Introductory Case Study
Leveraging MongoDB: An Introductory Case Study
 
MongoDB tuning on AWS
MongoDB tuning on AWSMongoDB tuning on AWS
MongoDB tuning on AWS
 
Cassandra Community Webinar: From Mongo to Cassandra, Architectural Lessons
Cassandra Community Webinar: From Mongo to Cassandra, Architectural LessonsCassandra Community Webinar: From Mongo to Cassandra, Architectural Lessons
Cassandra Community Webinar: From Mongo to Cassandra, Architectural Lessons
 
Drop acid
Drop acidDrop acid
Drop acid
 
Meetup#2: Building responsive Symbology & Suggest WebService
Meetup#2: Building responsive Symbology & Suggest WebServiceMeetup#2: Building responsive Symbology & Suggest WebService
Meetup#2: Building responsive Symbology & Suggest WebService
 
Scaling with MongoDB
Scaling with MongoDBScaling with MongoDB
Scaling with MongoDB
 
Challenges with MongoDB
Challenges with MongoDBChallenges with MongoDB
Challenges with MongoDB
 

Último

UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 

Último (20)

UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 

MongoDB - Who, What & Where!

  • 1. Who, what, how & in the cloud :) Mark  Hillick mark@10gen.com http://www.10gen.com  &  http://www.mongodb.org Tuesday 3 July 2012
  • 3. Agenda • Who • What • Example Deployments • EC2 Notes • EC2 Best Practices • Further Tuning Tuesday 3 July 2012
  • 4. Who, what & where! Tuesday 3 July 2012
  • 5. Who? • 10gen -> company behind MongoDB • Created by Dwight & Eliot • MongoDB is open-source & community is key • Offices in California, NY, Dublin, London & Sydney • $73.4 million in VC funding Tuesday 3 July 2012
  • 6. What? (1) • Powerful, flexible, scalable, fast data store • Document-oriented • Embedded docs & arrays • Scale out • Easy to start & develop with Tuesday 3 July 2012
  • 7. What? (2) • Indexing • Stored Javascript • Javascript Shell • Fixed-size collections • File Storage Tuesday 3 July 2012
  • 8. On What? • Linux • 10gen repos for Debian & Red Hat like distros • Windows • MAC OS (primarily dev) • Raspberry PI :) Tuesday 3 July 2012
  • 11. The Past & Now Tuesday 3 July 2012
  • 12. The Past & Now Tuesday 3 July 2012
  • 13. A bit deeper Tuesday 3 July 2012
  • 14. Getting Started • Document -> basic unit of data ~ a row in RBMS • Collection -> schema equivalent of a table • Single instance can have multiple dbs • JS Shell -> administration • Event document has special, unique key -> _id Tuesday 3 July 2012
  • 15. Collection > show collections file_tweets mini_tweets system.indexes Tuesday 3 July 2012
  • 16. Document { "_id" : 122, "user_id" : 123456, "total" : 100, "items" : [ { "item_name" : "my_item_one", "price" : 20 }, { "item_name" : "my_item_one", "price" : 20 }, { "item_name" : "my_item_three", "price" : 30 } ] } Tuesday 3 July 2012
  • 17. The Shell Tuesday 3 July 2012
  • 18. Drivers • Python • Ruby • JavaScript [node.js] • PHP • Perl • Scala • And more...... • http://docs.mongodb.org/manual/applications/drivers (Wiki’s & Docs) Tuesday 3 July 2012
  • 19. Indexes • Data structure collecting info • Enhance query performance • Optimiser sorts through & orders docs • db.file_tweets.ensureIndex({“user_id” : 1}) • http://www.mongodb.org/display/DOCS/Indexes Tuesday 3 July 2012
  • 20. Wire Protocol • Simple socket-based, request-response style protocol • MongoDB stores “BSON” documents, which are binary encoded serialisation of JSON documents • TCP /IP socket on port 27017 - fire & forget •http://www.mongodb.org/pages/viewpage.action?pageId=131351 Tuesday 3 July 2012
  • 22. Deployment Types • Master-Slave (not recommended) • Replica Sets • Shards • Some notes on EC2 deployments Tuesday 3 July 2012
  • 23. Replica Set Configurations Primary Primary Primary Secondary Secondary Secondary Arbiter Secondary Secondary (Minimum) (Typical) Secondary Secondary Tuesday 3 July 2012
  • 24. Replica Set Info • Asynchronous replication (single primary) • Automatic failover • App-level definition of “write replication” • Secondary nodes can replicate with a slaveDelay • Secondary nodes can be hidden • Max of 12 nodes, with 7 voting Tuesday 3 July 2012
  • 25. Sharding config DB mongos mongos mongos mongos config DB config DB Primary Primary Primary Primary Secondary Secondary Secondary Secondary Secondary Secondary Secondary Secondary Tuesday 3 July 2012
  • 26. Sharding Notes • Each “shard” usually a Replica Set (same options) • Copy of meta data stored in-memory by mongos • Data split into chunks, using range based shard key • Chunks may be migrated between shards • New chunks created by “splitting” old chunks Tuesday 3 July 2012
  • 27. Shard Server in EC2 (1) Category/Impact Low Medium High Disk Speed x Disk Capacity x RAM x CPU x Tuesday 3 July 2012
  • 28. Shard Server in EC2 (2) • MongoDB designed for OS defaults on 64 bit instance • Use standard virtual memory page size • Raise “nofiles” ulimit (20,000) • Use RAID10 & modern f/s -> ext4, xfs etc • Use “noatime” mount option Tuesday 3 July 2012
  • 29. Server in EC2 (1) • kernel >= 2.6.23/2.6.25 respectively • Readahead: how much more to read than what you asked for • If too high => possible performance impact • Set to 0 on EBS devices • Set to desired value on RAID device Tuesday 3 July 2012
  • 30. Server in EC2 (2) • Turn off atime on filesystem (pre-2.6.30 especially) • RAID 10 is recommended everywhere • mitigates slow EBS volumes (fail the bad volume) • Do not use large VM pages • Do configure swap to prevent OOM Killer Tuesday 3 July 2012
  • 31. Config Server • Meta Data for shard stored in ConfigDB • Copy of meta data stored in-memory by mongos • Config DB cluster is *not* a replica set -> run 3!! • If config server goes down then: • no splits and migrates • new mongos cannot be started • running mongos can still use cache to route r/w Tuesday 3 July 2012
  • 32. Config Server in EC2 (1) Category/Impact Low Medium High Disk Speed x Disk Capacity x RAM x CPU x Tuesday 3 July 2012
  • 33. Config Server in EC2 (2) • Use Raid10 • Use 64 bit instance • Can run on shard servers Tuesday 3 July 2012
  • 34. Mongos in EC2 (1) Category/Impact Low Medium High Disk Speed Disk Capacity RAM x CPU x Tuesday 3 July 2012
  • 35. Mongos in EC2 (2) • Often run on application servers • 32 bit mongos ok with 64-bit mongod Tuesday 3 July 2012
  • 36. Arbiter in EC2 (1) Category/Impact Low Medium High Disk Speed x Disk Capacity x RAM x CPU x Tuesday 3 July 2012
  • 37. Arbiter in EC2 (2) • Can use micro instance • Elections may be slower • Can use instance store • Still want backups :) Tuesday 3 July 2012
  • 38. HA in EC2 • Replica Sets • EC2 Availability Zones Tuesday 3 July 2012
  • 39. DR in EC2 • Replica Sets • EC2 Regions Tuesday 3 July 2012
  • 40. Security • Turn on authentication • Create a key between shards • EC2 Security Groups • Can reference other Security Groups • EC2 Regions • Follow SDLC in coding your app Tuesday 3 July 2012
  • 41. Monitoring • Links in with Cacti, Nagios, Munin-Node etc. • MMS - > it’s free Tuesday 3 July 2012
  • 42. Mongo Ports • mongos, unsharded mongod : 27017 • sharded mongod : 27018 • config mongod : 27019 • http port : 28017 Tuesday 3 July 2012
  • 43. Some More EC2 Notes Tuesday 3 July 2012
  • 44. Instances Guidelines (1) • Use 64-bit only, 32-bit is not recommended • Primary/Secondary should be equal* • High CPU is not necessary • High Memory for large mongod instances • Network capacity is also IO capacity (EBS) Tuesday 3 July 2012
  • 45. Instances Guidelines (2) • Note the trade-offs - memory/network • m1.large to m2.xlarge = 2x Mem, 0.5x Network • Do not use micro except for testing & config • m1.medium is usually sufficient for config DB • m1.small can be used for Arbiters Tuesday 3 July 2012
  • 46. Backups • EBS Snapshots - RAID complicates things • Single EBS volume, with journaling means: • No fsync & lock required • Similar applies to LVM snapshots • EC2 • General Tuesday 3 July 2012
  • 47. EC2/MongoDB Best Practices • https://wiki.10gen.com/display/DOCS/Amazon+EC2 • https://wiki.10gen.com/display/DOCS/Production+Notes Tuesday 3 July 2012
  • 49. node.js • server-side written in javascript • orginally written for push web apps • created by Ryan Dahl Tuesday 3 July 2012
  • 50. Sample node.js code var express = require('express'), Db = require('mongodb').Db, Server = require('mongodb').Server, Connection = require('mongodb').Connection; var host = 'localhost'; var port = Connection.DEFAULT_PORT; var db = new Db('node-mongo-examples', new Server(host, port, {}), {native_parser:false}); var app = express.createServer(); app.get('/', function(req, res){ res.send('Hello World'); }); db.open(function(err, db) { if(err) throw err app.listen(8124); }); Tuesday 3 July 2012
  • 51. node.js with mongo • https://github.com/christkv/node-mongodb-native • http://www.nodebeginner.org Tuesday 3 July 2012
  • 52. Need Help? Tuesday 3 July 2012
  • 53. • MongoDB Google User Group • New MongoDB Docs & Old MongoDB Docs • Presentations • If you’re curious :) Image Source: http://www.cannotstartoutlook.com/wp-content/uploads/2012/06/outlook-problems-help.jpg Tuesday 3 July 2012
  • 54. Credits • Credit must go to @comerford, @jonnyeight & @mikefiedler as I borrowed some of the knowledgeable slides :) Tuesday 3 July 2012