Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Stockage des données : quel système pour quel usage ?
Similar to Stockage des données : quel système pour quel usage ? (20)
Recently uploaded
Recently uploaded (20)
Stockage des données : quel système pour quel usage ?
- 1. #DevoxxMA @zouheircadi STOCKAGE : QUEL SYSTEME POUR QUEL USAGE
- 2. #DevoxxMA @zouheircadi QUI SUIS-JE • @ZouheirCADI • JEE architect (big data, perf., quality, ops, app, …) • Intervenant à l’ENST • Co-organisateur Devoxx France • (ancien …) Co-organisateur Paris Java User Group
- 3. #DevoxxMA @zouheircadi AGENDA • Revu des systèmes de stockage (OLAP et OLTP) • RDBMS • OLAP (Hadoop et Spark) • OLTP • Key-Value : memcached • Document : couchdb • Column family • Search • Conclusion
- 7. #DevoxxMA @zouheircadi RDBMS RelaYonal Database Management Systems were invented to let you use one set of data in mulYple ways, including ways that are unforeseen at the Yme the database is built and the 1st applicaYons are wrien. Curt Monash, analyst/blogger
- 8. #DevoxxMA @zouheircadi RDBMS • RelaYonal databases organize data in tables • Which are made of many rows. • Each row has data in each of several columns (every row in a table has the same columns) • RelaYonships are implicit Emp empno ename job deptno 7839 King President 10 7698 Blake Manager 20 deptno dname loc 10 Account NY 20 Sales CHI Dept
- 9. #DevoxxMA @zouheircadi RDBMS – KEY CONCEPTS
- 10. #DevoxxMA @zouheircadi 1er : Physical data independence PHYSICAL FILES LOGICAL MODEL fseek fopen fread © hWp://www.slideshare.net/billhoweuw/dataintensive-scalable-science
- 11. #DevoxxMA @zouheircadi 2eme : Relational algebra • Select, Project, Join • Union, Intersec`on, Difference © hWp://www.slideshare.net/billhoweuw/dataintensive-scalable-science
- 20. #DevoxxMA @zouheircadi NoSQL • NotOnlySQL • Nothing to do with SQL • Relaxa`on of transac`on constraints in distributed systems • CAP
- 21. #DevoxxMA @zouheircadi CAP • Consistency • Every read receives the most recent write or an error • Availability • Every request receives a response , without garantee that it contains the most recent version • ParYYon tolerance • The system con`nue to operate despite arbitrary par``onning due to network failure • If allowed, you might sacrifice consistency • If not, you might sacrifice availability • NOSQL may sacrifice consistency hWps://en.wikipedia.org/wiki/CAP_theorem
- 25. #DevoxxMA @zouheircadi Key dates • 2003 octobre : GFS paper released • 2004 décembre : MapReduce Simplified Data processing on large clusters • 2006 janvier : CréaYon Hadoop • 2006 octobre : Cluster Hadoop de 600machine chez Yahoo • 2007 avril: Cluster Hadoop de 1000 machine chez Yahoo hWps://en.wikipedia.org/wiki/Apache_Hadoop
- 26. #DevoxxMA @zouheircadi Map Reduce • MR • Abstrac`on • Programming model • ImplémentaYons • Open source • Hadoop • Less well known : Couchdb, Infinispan, Riak • Propriétaire : Google
- 32. #DevoxxMA @zouheircadi © hp://stackoverflow.com/quesYons/31044575/mapreduce-2-vs-yarn-applicaYons
- 38. #DevoxxMA @zouheircadi Hadoop conclusion • Données read-only avec traitements simples • Map-Reduce • Move computa`on to data • Paralleliza`on and distribu`on (High scalability) • Fault tolerance • Status and monitoring • «one person deployment » © hWps://en.wikipedia.org/wiki/MapReduce
- 43. #DevoxxMA @zouheircadi Hadoop conclusion • Map-Reduce has served a great purpose, though: many, many companies, research labs and individuals are successfully bringing Map-Reduce to bear on problems to which it is suited: brute-force processing with an opYonal aggregaYon. hWp://the-paper-trail.org/blog/the-elephant-was-a-trojan-horse-on-the-death-of-map-reduce-at-google/
- 44. #DevoxxMA @zouheircadi Hadoop conclusion • But more important in the longer term, to my mind, is the way that Map-Reduce provided the jusYficaYon for re-evaluaYng the ways in which large-scale data processing plaworms are built (and purchased!). hWp://the-paper-trail.org/blog/the-elephant-was-a-trojan-horse-on-the-death-of-map-reduce-at-google/
- 45. #DevoxxMA @zouheircadi Hadoop conclusion • It’s well known in the industry that more than 10 years ago Google invented MapReduce, the technology at the heart of first-generaYon Hadoop. It’s less well known that Google moved away from MapReduce several years ago. Today at its Google I/O 2014 … hWps://www.datanami.com/2014/06/25/google-re-imagines-mapreduce-launches-dataflow/
- 46. #DevoxxMA @zouheircadi Hadoop conclusion • … Today at its Google I/O 2014 conference, the Web giant unveiled a possible successor to MapReduce called Dataflow, which it’s selling through its hosted cloud service. hWps://www.datanami.com/2014/06/25/google-re-imagines-mapreduce-launches-dataflow/
- 51. #DevoxxMA @zouheircadi Spark - RDD Server 1 Server 2 Server 3 RDD
- 54. #DevoxxMA @zouheircadi Spark – Transformation : Map Hello World This Is Devoxx Morocco Held In Casablanca hello world this is devoxx morocco held in casablanca .map(_toLowerCase)
- 55. #DevoxxMA @zouheircadi Spark – Transformation : flatMap hello wold this is .flatMap(line=>line.split(«s+»)) hello world this is devoxx morocco held in casablanca …. devoxx
- 56. #DevoxxMA @zouheircadi Spark – Transformation : map (hello,1) (wold,1) (this,1) (is,1) .map(word=>(word,1)) …. (devoxx,1) hello wold this is …. devoxx
- 57. #DevoxxMA @zouheircadi Spark – Transformation : groupByKey (a,1) (b,1) (a,1) (a,1) (b,1) (b,1) (a,1) (a,1) (a,1) (b,1) (b,1) (b,1) (a,1) (a,1) (b,1) (b,1) (a,1) (a,1) (a,1) (b,1) (b,1) (b,1)
- 58. #DevoxxMA @zouheircadi Spark – Transformation : reduceByKey (a,1) (b,1) (a,1) (a,1) (b,1) (b,1) (a,1) (a,1) (a,1) (b,1) (b,1) (b,1) (a,1) (a,1) (a,1) (a,1) (a,1) (a,1) (a,6) (b,1) (b,1) (b,1) (b,1) (b,1) (b,1) (b,6)
- 59. #DevoxxMA @zouheircadi Spark • AcYons • func`ons that trigger computa`on and return something that isn’t an RDD • collect() : copy all elements to the driver • count() • collectAsMap() • sample() • take(n) : copy first n elements • reduce(func) : aggregates elements with func (take 2 elements, return one) • saveTextAsFile(fileName) : save to local or HDFS
- 60. #DevoxxMA @zouheircadi All in one val sc = new SparkContext() val docs = sc.textFile("hdfs://<path>") val low = docs.map(line => line.toLowerCase) val word = low.flatMap(line => line.split("s+")) val counts = words.map(word => (word,1)) val frequency = counts.reduceByKey(_ + _) val top = frequency.map(_swap).top(N) top.forEach(println)
- 73. #DevoxxMA @zouheircadi Data model © hWp://www.slideshare.net/yellow7/cassandra-backgroundandarchitecture Rela`onal DB Databases Tables Rows Columns MongoDB db Collec`ons Documents Fields Elas`cSearch Indices Types Documents Fields
- 74. #DevoxxMA @zouheircadi Column family stores • Persistent (distributed) maps
- 75. #DevoxxMA @zouheircadi Column family stores Map<RowKey, SortedMap<ColumnKey, ColumnValue>> © hWp://www.ebaytechblog.com/2012/07/16/cassandra-data-modeling-best-prac`ces-part-1/
- 76. #DevoxxMA @zouheircadi Column family stores
- 77. #DevoxxMA @zouheircadi Column family stores Map<RowKey, SortedMap<SuperColumnKey, SortedMap<ColumnKey, ColumnValue>>> © hWp://www.ebaytechblog.com/2012/07/16/cassandra-data-modeling-best-prac`ces-part-1/
- 78. #DevoxxMA @zouheircadi © hps://cloud.google.com/bigtable/docs/schema-design Column family stores (bigTable)
- 83. #DevoxxMA @zouheircadi Pourquoi explosion schema less
- 87. #DevoxxMA @zouheircadi Pourquoi explosion schema less • Start-up vs entreprises old school • (avec un TTM très court)
- 88. #DevoxxMA @zouheircadi Pourquoi explosion schema less • Allowed by business rules
- 89. #DevoxxMA @zouheircadi Pourquoi explosion schema less : 3V
- 90. #DevoxxMA @zouheircadi Contraintes à l’utilsation de NoSQL • TransacYons • On ne peut pas considérer que passer la résolu`on des conflits au client soit un progrès. • Mal nécessaire souvent dicté par le business
- 97. #DevoxxMA @zouheircadi URLOGRAPHIE • Hadoop, the definiYve guide, Third ediYon Tom White, ISBN: 978-1-449-31152-0, O'Reilly Ed. • hps://www.postgresql.org/about/ • hps://blog.codeship.com/unleash-the-power-of-storing-json-in-postgres/ • hps://opentextbc.ca/dbdesign/chapter/chapter-5-data-modelling/ • hp://coronet.iicm.edu/is/scripts/lesson03.pdf • hps://opentextbc.ca/dbdesign/chapter/chapter-3-characterisYcs-and-benefits-of-a-database/ • hp://gerardnico.com/wiki/relaYon/rdbms • hps://en.wikipedia.org/wiki/Scalability • hp://siliconangle.com/blog/2016/06/27/google-tools-up-with-its-spanner-database-looks-for-a-fight-with- aws/ • hp://www.caell.net/datastores/Datastores.pdf • hps://en.wikipedia.org/wiki/Apache_Hadoop • hps://en.wikipedia.org/wiki/MapReduce • hps://www.linkedin.com/pulse/rdbms-follows-acid-property-nosql-databases-base-does
- 98. #DevoxxMA @zouheircadi URLOGRAPHIE • hps://www.quora.com/Hadoop-Why-are-companies-invesYng-so-much-into-Hadoop-if-Google-released-the- MapReduce-paper-back-in-2004-Are-companies-just-going-to-follow-the-road-map-Google-created-Big-Table- Pregel-Dremel-etc-It-seems-to-me-that-companies-will-always-be-behind-the-curve • hp://the-paper-trail.org/blog/the-elephant-was-a-trojan-horse-on-the-death-of-map-reduce-at-google/ • hps://www.mapr.com/ebooks/spark/01-what-is-apache-spark.html • hps://www.digitalocean.com/community/tutorials/a-comparison-of-nosql-database-management-systems- and-models • hps://cloud.google.com/bigtable/docs/overview • hps://cloud.google.com/bigtable/docs/schema-design • hps://en.wikipedia.org/wiki/Dremel_(so‚ware) • hps://www.gartner.com/doc/reprints?id=1-2PMFPEN&ct=151013&st=sb • hp://www.infoworld.com/arYcle/3056637/database/nosql-chips-away-at-oracle-ibm-and-microso‚- dominance.html • hp://www.slideshare.net/billhoweuw/dataintensive-scalable-science