Getting your head around big data
•Transferir como PPTX, PDF•
4 gostaram•1,107 visualizações
My talk on Big Data from Dallas Day of .NET 2014
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Getting your head around big data
Semelhante a Getting your head around big data (20)
Mais de Glenn Block
Mais de Glenn Block (17)
Último
Último (20)
Getting your head around big data
- 3. 3 Make machine data accessible, usable and valuable to everyone.
- 4. Platform for Machine Data Any Machine Data HA Indexes and Storage Search and Investigation Proactive Monitoring Operational Visibility Real-time Business Insights Commodity Servers Online Services Web Services Servers Security GPS Location Storage Desktops Networks Packaged Applications Custom ApplicationsMessaging Telecoms Online Shopping Cart Web Clickstreams Databases Energy Meters Call Detail Records Smartphones and Devices RFID
- 5. DATA
- 6. 15,000 BC – Pictures Lascaux, France
- 7. 6000 BC – Symbols
- 8. 3,500 BC – Language
- 9. 1,275 BC – Papyrus
- 10. 1st - 13th Century - Codex
- 11. 13th Century – Movable type
- 12. 15th Century – Printing press
- 13. 19th to 20th century Babbage Analytical engine
- 14. 1936 – Turing machine
- 15. 1945 – ENIAC
- 16. 1947 – The first bug
- 17. 1977 - Arpanet
- 18. 1990s Internet
- 20. RFID
- 21. Cloud
- 22. Services
- 25. 90 percent of all the data in the world has been generated over the last two years source: sciencedaily.com
- 26. Every day 2.5 quintillion bytes of data is generated 1 quintillion = 1 + 18 zeros! 57.5 billion 32 GB iPads source: storagenewsletter.com
- 27. 2.7 zettabytes exist in the digital universe 1 zettabyte = 1 + 21 zeros! 42zb = All human speech digitized source: highscalability.com
- 28. How big is big?
- 29. That’s A LOT of data!
- 30. How do you harness it?
- 31. This is what big data is really about.
- 32. Asking questions and getting answers
- 33. Massive amounts of data. Machine generated VOLUME
- 34. Data is coming from a multitude of sources Mix of structured and un-structured (JSON, XML, CSV, Plain Text) Need a way to store it and and query it VARIETY
- 35. VARIETY Log files Activity Feeds Emails Device Streams Audio Files Videos
- 36. Data arrives at many different frequencies Need to be able to process real time. VELOCITY
- 37. Not all data that is stored is useful. Need to identify the useful data Need to wade through all the noise VERACITY
- 38. SOLUTIONS
- 39. Map/Reduce function map(String name, String document): // name: document name // document: document contents for each word w in document: emit (w, 1) function reduce(String word, Iterator partialCounts): // word: a word // partialCounts: a list of aggregated partial counts sum = 0 for each pc in partialCounts: sum += ParseInt(pc) emit (word, sum)
- 40. Hi scale and availability databases
- 41. Distributed processing of large datasets
- 42. Data Visualization and analysis
- 43. End to end tools
Notas do Editor
- At Splunk, our mission is to make machine data accessible, usable and valuable to everyone. Andthis overarching mission is what drives our company and product priorities.
- Splunk is the leading platform for machine data analytics with over 5,200 organizations using Splunk (as of 7/1/13) – from tens of GB to many tens of TBs of data PER DAY.Splunk software is optimized for real-time, low latency and interactivity.Splunk software reliably collects and indexes all the streaming data from IT systems and technology devices in real-time - tens of thousands of sources in unpredictable formats and types.The value from Splunking machine data is described as Operational Intelligence. This enables organizations to: 1. Find and fix problems dramatically faster2. Automatically monitor to identify issues, problems and attacks3. Gain end-to-end visibility to track and deliver on IT KPIs and make better-informed IT decisions4. Gain real-time insight from operational data to make better-informed business decisions
- The Lascaux cave paintings record the first known narrative stories. Telling stories through visualization of eventsMike Bostock,D3 and the NewYork Times VizCSS had to come from somewhere
- Jiahu, China