The ABC of Big Data
•
52 gostaram•4,658 visualizações
Introduction to Big Data Concepts and Tools
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a The ABC of Big Data
Semelhante a The ABC of Big Data (20)
Mais de André Faria Gomes
Mais de André Faria Gomes (20)
Último
Último (20)
The ABC of Big Data
- 1. Big Data ABC @andrefaria
- 3. Concepts
- 5. Key Value Stores Riak, Mecached, Berkley DB, HamsterDB, Couchbase, Voldemort, DynamoDB http://www.thoughtworks.com/insights/blog/nosql-databases-overview
- 6. Document Stores think about document databases as key-value stores where the value is examinable rich query language + indexes MongoDB, CouchDB , Terrastore, OrientDB, RavenDB http://www.thoughtworks.com/insights/blog/nosql-databases-overview
- 7. Column Family Stores Each column family can be compared to a container of rows in an RDBMS table where the key identifies the row and the row consists of multiple columns. The difference is that various rows do not have to have the same columns, and columns can be added to any row at any time without having to add it to other rows. Cassandra, HBase, Hypertable http://www.thoughtworks.com/insights/blog/nosql-databases-overview
- 8. Graph Databases Neo4J, Infinite Graph, OrientDB, FlockDB http://www.thoughtworks.com/insights/blog/nosql-databases-overview
- 9. Map Reduce
- 11. Document-oriented system Large Data Sets Records similar to JSON Automatic sharding and MapReduce Queries are written in JavaScript
- 12. Document-oriented system JavaScript Interface Multi-version concurrency control approach Client side needs to handle clashes on writes No good built-in method for horizontal scalability (but there are external solutions like BigCouch, Lounge, and Pillow)
- 13. Originally an internal Facebook project Keyspaces and column families Similar the data model used by BigTable Data is sharded and balanced automatically
- 14. Keeps the entire Database in RAM Its values can be complex data structures
- 15. BIG TABLE Structure: tables, row keys, column families, column names, timestamps, and cell values Designed to handle very large data loads by running on big clusters of commodity hardware Uses GFS as its underlying storage
- 16. Open source clone of Big Table Same structure of Big Table Uses HDFS instead of GFS
- 17. Another open source BigTable clone written in C++ Focus in High Performance
- 18. DynamoDB Key Value System Large Distributed Clusters Versioning
- 19. AWS S3 HTTP Blobs
- 20. Inspired by AWS Dynamo DB OpenSource and Commercial Versions Key Value System Large Distributed Clusters Queries in ErLang or JavaScript Consistent hashing and a gossip protocol to avoid centralized index server
- 21. Takes care of running your code across a cluster of machines. - chunking up the input data - sending it to each machine - running your code on each chunk - checking that the code ran - passing any results next stage - sorting between stages - sending each chunk of that sorted data to the right machine - writing debugging information on each job’s progress
- 22. With Hive, you can program Hadoop jobs using a SQL-like language HiveQL.
- 23. Apache Pig A procedural data processing language designed for Hadoop Provides a set of functions that help with common data processing problems
- 24. PigPen is map-reduce for Clojure, or distributed Clojure. It compiles to Apache Pig, but you don't need to know much about Pig to use it.
- 25. Hadoop for Logs The Flume project is designed to make the data gathering process easy and scalable, by running agents on the source machines that pass the data updates to collectors, which then aggregate them into large chunks that can be efficiently written as HDFS files.
- 26. The R project is both a specialized language and a toolkit of modules aimed at anyone working with statistics.
- 27. Lucene is a Java library that handles indexing and searching large collections of documents, and Solr is an application that uses the library to build a search engine server.
- 28. Mahout is an open source framework that can run common machine learning algorithms on massive datasets. The framework makes it easy to use analysis techniques to implement features such as “People who bought this also bought” recommendation engine on your own site.
- 29. ZooKeeper Coordinates work and configuration of different Clusters
- 30. Serialization As you pass data between systems and you need to store it in files at some points
- 31. JSON BSON (Binary JSON) Apache Thrift (predefine structure) Apache Avro (predefine structure)
- 32. blog.andrefaria.com datavisionary.net andrefaria.com @andrefaria