Five Things you Need to Know About Scaling

•

1 gostou•1,068 visualizações

The 5 Things
You Need To Know
About Scaling
A. Jesse Jiryu Davis 
 
Senior Python Engineer

$db.people.find({name: 'Jesse'}).explain()! {! ! "cursor" : "BasicCursor",! ! "n" : 10,! ! "nscanned" : 100000,! ! "millis" : 40,! }! ! db.people.createIndex({name: 1});! ! db.people.find({name: 'Jesse'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 10,! ! "nscanned" : 10,! ! "millis" : 0,! }! 1. Indexes$

$db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 10,! ! "millis" : 0,! }! ! db.people.createIndex({name: 1, hacks: 1});! ! db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 1,! ! "millis" : 0,! }! 1. Indexes$

$db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 10,! ! "millis" : 0,! }! ! db.people.createIndex({name: 1, hacks: 1});! ! db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1_hacks_1",! ! "n" : 1,! ! "nscanned" : 1,! ! "millis" : 0,! }! ! db.people.dropIndex({name: 1})! ! 1. Indexes$

2. File System
• ext3 is bad 
• ext4 and xfs are good

• Set noatime 
• This goes in /etc/fstab
2. File System

4. Disks
Option 1: Spinning disk.
100 seeks / sec

~100 seeks / second!
~100 seeks / second! ~100 seeks / second! ~100 seeks / second!

Primary!
Secondary!
Secondary!
MongoS!
Primary!
Secondary!
Secondary!
Primary!
Secondary!
Secondary!
Primary!
Secondary!
Secondary!
5. Shard

1. Indexes

2. File System

3.Working Set

4. Disks

5. Shard

1. Indexes

2. File System

3.Working Set

4. Disks

5. Shard
The 5 Things You Need To
Know About Scaling
A. Jesse Jiryu Davis 
bit.ly/scaling-ﬁve

Mais conteúdo relacionado

Mais procurados

Native json in the Cache' ObjectScript 2016.*Timur Safin

Linux basics by Raj MirajeRaj Mirje

Recommender Systems with Ruby (adding machine learning, statistics, etc)Marcel Caraciolo

Software Dendrology by Brandon BloomHakka Labs

PHP 1Mohammed Abutabikh

jSession #3 - Krzysztof Czajkowski - Lombok in JavajSession

GeoMapper, Python Script for Visualizing Data on Social Networks with Geo-loc...Marcel Caraciolo

Moose Lightning TalkMike Whitaker

Learning How To Use Jquery #5Takahiro Yoshimura

PHP Tutorial (funtion)Tinnakorn Puttha

20171014 tips for manipulating filesystem in julia岳華杜

Talk NullByteCon 2015Roberto Soares

Derrubando mitos em PythonDenis Costa

NoSQL & MongoDBShuai Liu

Andreas Roth - GraphQL erfolgreich im Backend einsetzenDevDay Dresden

QNlocal: Docker, Continuous Integration, WordPress e milioni di visite. Si è ...alessandro mazzoli

Semantic search for Earth Observation productsGasperi Jerome

The Animals of Rikki Tikki Tavijabezler

An (Inaccurate) Introduction to PythonNicholas Tollervey

Mais procurados (19)

Native json in the Cache' ObjectScript 2016.*

Linux basics by Raj Miraje

Recommender Systems with Ruby (adding machine learning, statistics, etc)

Software Dendrology by Brandon Bloom

PHP 1

jSession #3 - Krzysztof Czajkowski - Lombok in Java

GeoMapper, Python Script for Visualizing Data on Social Networks with Geo-loc...

Moose Lightning Talk

Learning How To Use Jquery #5

PHP Tutorial (funtion)

20171014 tips for manipulating filesystem in julia

Talk NullByteCon 2015

Derrubando mitos em Python

NoSQL & MongoDB

Andreas Roth - GraphQL erfolgreich im Backend einsetzen

QNlocal: Docker, Continuous Integration, WordPress e milioni di visite. Si è ...

Semantic search for Earth Observation products

The Animals of Rikki Tikki Tavi

An (Inaccurate) Introduction to Python

Semelhante a Five Things you Need to Know About Scaling

TDDBC お題Takuto Wada

HadoopとMongoDBを活用したソーシャルアプリのログ解析Takahiro Inoue

DEFCON 23 - Mahdi Namazifar - DETECTING RANDOM STRINGS; A LANGUAGE BASED APPR...Felipe Prado

Class 6: Lists & dictionariesMarc Gouw

Liferay Search: Best Practices to Dramatically Improve Relevance - Liferay Sy...André Ricardo Barreto de Oliveira

Running Production MongoDB Lightning Talkchrisckchang

VRA 2012, Embedded Metadata, Embedded Metadata and the do-it-yourselferVisual Resources Association

Mobile Database PersistenceEric Maxwell

Building Apps with MongoDBNate Abele

Snakes and laddersRubén Berenguel

NoSQL を Ruby で実践するための n 個の方法Tomohiro Nishimura

Good CodeKevlin Henney

Feed Normalization with Ember Data 1.0Jeremy Gillick

Python tutorialnazzf

Python tutorialShani729

Data Exploration with ElasticsearchAleksander Stensby

Getting Started: Atlas Search WebinarKaren Huaulme

CouchDB @ red dirt ruby conferenceleinweber

MongoDB + node.js で作るソーシャルゲームSuguru Namura

Strongly Typed Languages and Flexible SchemasNorberto Leite

Semelhante a Five Things you Need to Know About Scaling (20)

TDDBC お題

HadoopとMongoDBを活用したソーシャルアプリのログ解析

DEFCON 23 - Mahdi Namazifar - DETECTING RANDOM STRINGS; A LANGUAGE BASED APPR...

Class 6: Lists & dictionaries

Liferay Search: Best Practices to Dramatically Improve Relevance - Liferay Sy...

Running Production MongoDB Lightning Talk

VRA 2012, Embedded Metadata, Embedded Metadata and the do-it-yourselfer

Mobile Database Persistence

Building Apps with MongoDB

Snakes and ladders

NoSQL を Ruby で実践するための n 個の方法

Good Code

Feed Normalization with Ember Data 1.0

Python tutorial

Data Exploration with Elasticsearch

Getting Started: Atlas Search Webinar

CouchDB @ red dirt ruby conference

MongoDB + node.js で作るソーシャルゲーム

Strongly Typed Languages and Flexible Schemas

Mais de MongoDB

MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB

MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!MongoDB

MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB

MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDBMongoDB

MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...MongoDB

MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series DataMongoDB

MongoDB SoCal 2020: MongoDB Atlas Jump StartMongoDB

MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]MongoDB

MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2MongoDB

MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...MongoDB

MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!MongoDB

MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your MindsetMongoDB

MongoDB .local San Francisco 2020: MongoDB Atlas JumpstartMongoDB

MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...MongoDB

MongoDB .local San Francisco 2020: Aggregation Pipeline Power++MongoDB

MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...MongoDB

MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & GolangMongoDB

MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...MongoDB

MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...MongoDB

Mais de MongoDB (20)

MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas

MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!

MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...

MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB

MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...

MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data

MongoDB SoCal 2020: MongoDB Atlas Jump Start

MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]

MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2

MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...

MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!

MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset

MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart

MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...

MongoDB .local San Francisco 2020: Aggregation Pipeline Power++

MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...

MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive

MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang

MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...

MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...

Último

Elevate Developer Efficiency & build GenAI Application with Amazon QBhuvaneswari Subramani

Corporate and higher education May webinar.pptxRustici Software

presentation ICT roal in 21st century educationjfdjdjcjdnsjd

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Zilliz

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub

Architecting Cloud Native ApplicationsWSO2

Exploring Multimodal Embeddings with MilvusZilliz

CNIC Information System with Pakdata Cf In Pakistandanishmna97

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelDeepika Singh

FWD Group - Insurer Innovation Award 2024The Digital Insurer

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Angeliki Cooney

Platformless Horizons for Digital AdaptabilityWSO2

DBX First Quarter 2024 Investor PresentationDropbox

Understanding the FAA Part 107 License ..Christopher Logan Kennedy

TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Victor Rentea

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software

ICT role in 21st century education and its challengesrafiqahmad00786416

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93

Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vázquez

Five Things you Need to Know About Scaling

1. The 5 Things You Need To Know About Scaling A. Jesse Jiryu Davis    Senior Python Engineer

2. 1. Indexes

3. db.people.find({name: 'Jesse'}).explain()! {! ! "cursor" : "BasicCursor",! ! "n" : 10,! ! "nscanned" : 100000,! ! "millis" : 40,! }! ! db.people.createIndex({name: 1});! ! db.people.find({name: 'Jesse'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 10,! ! "nscanned" : 10,! ! "millis" : 0,! }! 1. Indexes

4. db.people.find({name: 'Jesse'}).explain()! {! ! "cursor" : "BasicCursor",! ! "n" : 10,! ! "nscanned" : 100000,! ! "millis" : 40,! }! ! db.people.createIndex({name: 1});! ! db.people.find({name: 'Jesse'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 10,! ! "nscanned" : 10,! ! "millis" : 0,! }! 1. Indexes

5. db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 10,! ! "millis" : 0,! }! ! db.people.createIndex({name: 1, hacks: 1});! ! db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 1,! ! "millis" : 0,! }! 1. Indexes

6. db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 10,! ! "millis" : 0,! }! ! db.people.createIndex({name: 1, hacks: 1});! ! db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 1,! ! "millis" : 0,! }! 1. Indexes

7. db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 10,! ! "millis" : 0,! }! ! db.people.createIndex({name: 1, hacks: 1});! ! db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 1,! ! "millis" : 0,! }! 1. Indexes

8. db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 10,! ! "millis" : 0,! }! ! db.people.createIndex({name: 1, hacks: 1});! ! db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1_hacks_1",! ! "n" : 1,! ! "nscanned" : 1,! ! "millis" : 0,! }! ! db.people.dropIndex({name: 1})! ! 1. Indexes

9. db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1",! ! "n" : 1,! ! "nscanned" : 10,! ! "millis" : 0,! }! ! db.people.createIndex({name: 1, hacks: 1});! ! db.people.find({name: 'Jesse', hacks: 'Python'}).explain()! {! ! "cursor" : "BtreeCursor name_1_hacks_1",! ! "n" : 1,! ! "nscanned" : 1,! ! "millis" : 0,! }! ! db.people.dropIndex({name: 1})! ! 1. Indexes

10. 2. File System • ext3 is bad  • ext4 and xfs are good

11. • Set noatime  • This goes in /etc/fstab 2. File System

12. 3. Working Set Size

13. 3. Working Set Size Disk! RAM!

14. 4. Disks

15. 4. Disks Option 1: Spinning disk. 100 seeks / sec

16. 4. Disks Option 2: Spinning disks!

17. ~100 seeks / second! ~100 seeks / second! ~100 seeks / second! ~100 seeks / second!

18. 4. Disks Option 3: SSD

19. 5. Shard

20. Primary! Secondary! Secondary! MongoS! Primary! Secondary! Secondary! Primary! Secondary! Secondary! Primary! Secondary! Secondary! 5. Shard

21. 1. Indexes 2. File System 3.Working Set 4. Disks 5. Shard