O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Introduction on Data Science

34.053 visualizações

Publicada em

Publicada em: Educação, Tecnologia

Introduction on Data Science

  1. 1. Data Science Module 1: Introduction to Data Science
  2. 2. LIVE On-line Class Class Recording in LMS 24/7 Post Class Support Module Wise Quiz Project Work on Large Data Base Verifiable Certificate How it Works? Slide 2 www.edureka.in/data-science
  3. 3. Slide 3 www.edureka.in/data-science  Module 1 » Introduction to Data Science  Module 2 » Basic Data Manipulation using R  Module 3 » Machine Learning Techniques using R Part -1 - Clustering - TF-IDF and Cosine Similarity - Association Rule Mining  Module 4 » Machine Learning Techniques using R Part -2 - Supervised and Unsupervised Learning - Decision Tree Classifier Course Topics  Module 5 » Machine Learning Techniques using R Part -3 - Random Forest Classifier - Naïve Bayer’s Classifier  Module 6 » Introduction to Hadoop Architecture  Module 7 » Integrating R with Hadoop  Module 8 » Mahout Introduction and Algorithm Implementation  Module 9 » Additional Mahout Algorithms and Parallel Processing in R  Module 10 » Project
  4. 4. Topics for the Day Slide 4 www.edureka.in/data-science  Big Data  Big Data Scenarios  Big Data Challenges  Introduction to Data Science  Data Science: Components Types of Data Scientists  Data Science: Core Components  Use-Cases  Introduction to Hadoop and R  R and Hadoop Integration  Machine Learning with Mahout  Assignment, Pre-work and Agenda for the Next Class  What’s Within the LMS  References
  5. 5. Objectives At the end of this module, you will be able to  Understand Big Data and its challenges  Implement Big Data in real time scenarios  List and explain the components and prospects of Data Science  Learn the implementation of Hadoop on Big data  Analyze some real world use-cases with the help of R programming Language  Understand machine learning concepts
  6. 6. Data Science Slide 6 www.edureka.in/data-science
  7. 7. Big Data Slide 7 www.edureka.in/data-science
  8. 8. What is Big Data? Lots of Data (Terabytes or Petabytes) Systems/Enterprises generate huge amount of data from Terabytes to and even Petabytes of information Slide 7 www.edureka.in/data-sciencehttp://www.today.mccombs.utexas.edu/2012/04/the-big-data-machine
  9. 9. Big Data Scenarios Slide 8 www.edureka.in/data-sciencehttp://www.clker.com/clipart-13967.html
  10. 10. Big Data Scenarios: Sports Slide 10 www.edureka.in/data-sciencehttp://www.espncricinfo.com/
  11. 11. Big Data Scenarios: Sports Sports teams are using data for tracking ticket sales and even for tracking team strategies. Advertising and marketing agencies are tracking social media to understand responsiveness to campaigns, promotions, and other advertising mediums Slide 11 www.edureka.in/data-sciencehttp://www.espncricinfo.com/
  12. 12. Big Data Scenarios : Hospital Care Slide 11 www.edureka.in/data-sciencehttp://www.majorprojects.vic.gov.au/our-projects/our-past-projects/austin-hospital
  13. 13. Big Data Scenarios : Hospital Care Hospitals are analyzing medical data and patient records to predict those patients that are likely to seek readmission within a few months of discharge. The hospital can then intervene in hopes of preventing another costly hospital stay. Medical diagnostics company analyzes millions of lines of data to develop first non-intrusive test for predicting coronary artery disease. To do so, researchers at the company analyzed over 100 million gene samples to ultimately identify the 23 primary predictive genes for coronary artery disease Slide 12 www.edureka.in/data-science
  14. 14. Big Data Scenarios : Amazon.com Slide 14 www.edureka.in/data-sciencehttp://wp.streetwise.co/wp-content/uploads/2012/08/Amazon-Recommendations.png
  15. 15. Amazon has an unrivalled bank of data on online consumer purchasing behaviour that it can mine from its 152 million customer accounts. Amazon also uses Big Data to monitor, track and secure its 1.5 billion items in its retail store that are laying around it 200 fulfilment centres around the world. Amazon stores the product catalogue data in S3. S3 can write, read and delete objects up to 5 TB of data each. The catalogue stored in S3 receives more than 50 million updates a week and every 30 minutes all data received is crunched and reported back to the different warehouses and the website. Big Data Scenarios : Amazon.com Slide 15 www.edureka.in/data-sciencehttp://wp.streetwise.co/wp-content/uploads/2012/08/Amazon-Recommendations.png
  16. 16. Big Data Scenarios: NetFlix Slide 16 www.edureka.in/data-sciencehttp://smhttp.23575.nexcesscdn.net/80ABE1/sbmedia/blog/wp-content/uploads/2013/03/netflix-in-asia.png
  17. 17. Netflix uses 1 petabyte to store the videos for streaming. BitTorrent Sync has transferred over 30 petabytes of data since its pre-alpha release in January 2013. The 2009 movie Avatar is reported to have taken over 1 petabyte of local storage at Weta Digital for the rendering of the 3D CGI effects. One petabyte of average MP3-encoded songs (for mobile, roughly one megabyte per minute), would require 2000 years to play. Big Data Scenarios: NetFlix Slide 17 www.edureka.in/data-sciencehttp://smhttp.23575.nexcesscdn.net/80ABE1/sbmedia/blog/wp-content/uploads/2013/03/netflix-in-asia.png
  18. 18. Big Data Scenarios: The Large Hadron Collider Slide 17 www.edureka.in/data-sciencehttp://www.crowdsourcing.org/article/-nasa-tries-to-free-creativity-with-big-data-challenge/19984
  19. 19. The experiments in the Large Hadron Collider produce about 15 petabytes of data per year, which are distributed over the Worldwide LHC Computing Grid. One petabyte is enough to store the DNA of the entire population of the USA - with cloning it twice. Big Data Scenarios: The Large Hadron Collider Slide 18 www.edureka.in/data-sciencehttp://en.wikipedia.org/wiki/Large_Hadron_Collider
  20. 20. IBM’s Definition IBM’s Definition – Big Data Characteristics http://www-01.ibm.com/software/data/bigdata/ Web logs Images Videos Sensor Data Audios VOLUME VELOCITY VARIETY Slide 20 www.edureka.in/data-science
  21. 21. IBM’s Definition  Structured  Unstructured  Semi structured  All the above Variety 3 Vs of Big data  Batch  Near Time  Real Time  Streams Velocity  Terabytes  Records  Transactions  Tables, files Volume IBM’s Definition – Big Data Characteristics http://www-01.ibm.com/software/data/bigdata/ Slide 21 www.edureka.in/data-science
  22. 22. What about ‘Veracity’? Slide 21 www.edureka.in/data-sciencehttp://whatsthebigdata.files.wordpress.com/2013/11/batman-on-big-data.jpg
  23. 23. HelHoello Teh!e!re!! My namiseAinsnAien.nie. I lovIeloqvueizqzueiszzaensdand puzpzluezszlaensdaIndamI am hetroemtoakmake youygouuygsutyhsintkhiannkdaanndsawnsweyr my quequestions. Slide 23 www.edureka.in/data-science Annie’s Introduction
  24. 24. Map the following to corresponding type: Structured/ Unstructured/ Semi- structured. - XML Files - Word Docs, PDF files, Text files - E-Mail body - Data from Enterprise systems (ERP, CRM etc.) Slide 24 www.edureka.in/data-science Annie’s Question
  25. 25. XML Files -> Semi-structured data Word Docs, PDF files, Text files -> Unstructured Data E-Mail body -> Unstructured Data Data from Enterprise systems (ERP, CRM etc.) -> Structured Data Slide 25 www.edureka.in/data-science Annie’s Answer
  26. 26. Big Data: Challenges Slide 25 www.edureka.in/data-sciencehttp://spinnakr.com/blog/wp-content/uploads/2013/08/Using-Big-Data-.jpg
  27. 27. Big Data Challenges Data security and Privacy High variety of Information High veracity of Data Data Acquisition High velocity of processed Data Information search and Analytics High volume of Data Information storage and Analytics Slide 26 www.edureka.in/data-science Big Data: Challenges
  28. 28. Slide 27 www.edureka.in/data-sciencehttp://thesocietypages.org/sociologylens/files/2013/09/BIgDataDilbert_Cartoon.jpg
  29. 29. Data Science Slide 28 www.edureka.in/data-sciencehttp://escience.washington.edu/blog/uw-berkeley-nyu-collaborate-378m-data-science-initiative
  30. 30. Data Science “More data usually beats better algorithms,” Such as: Recommending movies or music based on past preferences. Slide 30 www.edureka.in/data-science
  31. 31. No matter how extremely unpleasant your algorithm is, they can often be beaten simply by having more data (and a less sophisticated algorithm). Big Data is here Bad News We are struggling to store and analyze it. Good News Data Science Slide 31 www.edureka.in/data-science
  32. 32. Data Science: Components Slide 31 www.edureka.in/data-sciencehttp://abstrusegoose.com/55
  33. 33. Data Science Visualization Domain Expertise Statistics Data Engineering Advanced Computing Data Science: Components Slide 33 www.edureka.in/data-science
  34. 34. Data Science: Prospects Slide 34 www.edureka.in/data-science
  35. 35. Types of Data Scientists Based on clustering the ways that data is handled by Data Scientists, the following 4 categories can be created:  Data Businesspeople are the product and profit-focused data scientists. They’re leaders, managers, and entrepreneurs, but with a technical bent. A common educational path is an engineering degree paired with an MBA.  Data Creatives are eclectic jacks-of-all-trades, able to work with a broad range of data and tools. They may think of themselves as artists or hackers, and excel at visualization and open source technologies.  Data Developers are focused on writing software to do analytic, statistical, and machine learning tasks, often in production environments. They often have computer science degrees, and often work with so-called “big data”.  Data Researchers apply their scientific training, and the tools and techniques they learned in academia, to organizational data. They may have PhDs, and their creative applications of mathematical tools yields valuable insights and products. Slide 34 www.edureka.in/data-sciencehttp://datacommunitydc.org/blog/2013/06/there-is-more-than-one-kind-of-data-scientist/
  36. 36. Relationships - Four Categories and the Five Skill Groups Slide 35 www.edureka.in/data-sciencehttp://datacommunitydc.org/blog/wp-content/uploads/2012/08/SkillsSelfIDMosaic-edit-500px.png
  37. 37. Data Science: Core Components Data Science Data Architecture Tool: Hadoop Machine Learning Tool: Mahout Analytics Tool: R Slide 37 www.edureka.in/data-science
  38. 38. Use-Cases Slide 38 www.edureka.in/data-science
  39. 39. No one Knows How to Use it Slide 39 www.edureka.in/data-science
  40. 40. Use-Case Implementation: Techniques Used A Problem Dataset Analysis Results Slide 40 www.edureka.in/data-science
  41. 41. Understanding the Machine Learning algorithm to be used Implementing Machine Learning in Hadoop on Big Data Visualisation of the analysis Understanding the problem statement and defining the solution Exploring ways to integrate R with Hadoop Implementing Machine Learning algorithm in R on the smaller dataset Use-Case Implementation:Process Flow Diagram Slide 41 www.edureka.in/data-science
  42. 42. Domain of the Dataset: Communications and Media. However, the application of the algorithm is not limited to only Communications and Media. The technique is useful for any domain which requires organizing documents to improve retrieval and support browsing. Problem Statement: A top media company wants to browse through the popular news from a collection that appeared on the Reuters newswire in 1987. Clustering / Grouping documents based on their contents will make the analysis easier. Media Use-Case The Reuters-21578 data set composition Slide 42 www.edureka.in/data-science
  43. 43. Media Use-Case: K-means Clustering First we will understand the implementation of the technique in R on a smaller dataset Then we will understand how to achieve document clustering on Big Data using Mahout libraries on Hadoop K-Means Clustering can be implemented on this dataset Communications and Media Dataset to be Clustered based on their contents R Implementation Hadoop Implementation Machine Learning Implementation Content-wise Clustered/Grouped documents Slide 43 www.edureka.in/data-science
  44. 44. Domain of the Dataset: Products and Retail. However, the application of the algorithm is not limited to only Products and Retail. The technique can be applied wherever we want to discover the co-occurrence relationship amongst various activities. Problem Statement: Market Basket Analysis. A retail outlet wants understand the purchase behavior of a buyer. This information will enable the retailer to understand the buyer's needs. The analysis might tell a retailer that customers often purchase shampoo and conditioner together, so putting both items on promotion at the same time would create a significant increase in profit, while a promotion involving just one of the items would likely drive sales of the other. Market Basket Use-Case Market Basket Analysis 98% of people who purchased items A and B also purchased item C Slide 44 www.edureka.in/data-science
  45. 45. Market Basket Use-Case: Association Rule Mining Product and Retail Dataset Understand the implementation of the technique on a smaller dataset Understand how to achieve the same on Big Data using Mahout libraries on Hadoop The technique used is Affinity Analysis or Association Rule Mining R Implementation Hadoop Implementation Machine Learning Implementation Market Basket Analysis Slide 45 www.edureka.in/data-science
  46. 46. Slide 45 www.edureka.in/data-science Domain of the Dataset: Life Science and Health Care. However, the application of the algorithm is not limited to only Life Science and Health Care . The technique can be applied wherever we want to forecast the occurrence of a event on the basis of certain conditions. Problem Statement: A health care organization wants to forecast the onset of diabetes mellitus in Indians using certain set of attributes of patients as input such as: Plasma glucose concentration Diastolic blood pressure Triceps skin fold thickness etc. Health Care Use-Case http://www.thenewstribe.com/2013/11/15/diabetes-is-killing-one-patient-every-six-seconds/
  47. 47. Slide 46 www.edureka.in/data-science Understand the basic implementation of the technique on a smaller dataset using R Achieve parallel processing on the same algorithm using a parallel processing library provided by Revolution R. Understand how to achieve the same on Big Data using Mahout libraries on Hadoop The technique used is Affinity Analysis or Association Rule Mining. R Implementation Hadoop Implementation Machine Learning Implementation Forecast the onset of diabetes mellitus in Indians Life Science and Health Care Dataset with some attributes of patients as input. Health Care Use-Case: Parallel Processing
  48. 48. Slide 47 www.edureka.in/data-science Domain of the Dataset: Social Media. However, the application of the algorithm is not limited to only Social Media. The technique can be applied wherever we want to put documents into category without going through the contents of all the documents. Problem Statement: A Social Media research firm wants to know the trends of topics discussed on Twitter. For easy analysis it wants to classify them in the following categories:  apparel (clothes, shoes, watches, …)  art (Book, DVD, Music, …)  camera  event (travel, concert, …)  health (beauty, spa, …)  home (kitchen, furniture, garden, …)  tech (computer, laptop, tablet, …) http://www.mobigyaan.com/images/stories/Miscellaneous/mobigyaan-twitter-chat.jpg Social Media Use-Case
  49. 49. Social Media Use-Case: Naïve Bayes Classifier Understand the basic implementation of the technique on a smaller dataset using R. Understand how to achieve the same on Big Data using Mahout libraries on Hadoop. The technique used is Naïve Bayes Classifier. Social Media dataset R Implementation Hadoop Implementation Machine Learning Implementation Categorical classification of the tweets Slide 49 www.edureka.in/data-science
  50. 50. Going forward with the class, we will throw some light on the concepts of Hadoop, R and Machine Learning respectively. These topics will be vividly covered in their respective modules during the course. Data Science: Core Components Slide 50 www.edureka.in/data-science
  51. 51. Introduction to Hadoop Slide 51 www.edureka.in/data-science
  52. 52.  Apache Hadoop is a framework that allows for the distributed processing of large data sets across clusters of commodity computers using a simple programming model.  It is an Open-source Data Management with scale-out storage & distributed processing.  In 2004, Google published a paper on a process called MapReduce.  MapReduce framework provides a and associated implementation to data. parallel processing model process huge amount of  Therefore, an implementation of MapReduce framework was adopted by an Apache open source project named Hadoop. Introduction to Hadoop Slide 52 www.edureka.in/data-science
  53. 53. Hadoop Key Characteristics Scalable Reliable Economical Flexible Robust Ecosystem Hadoop Key Characteristics Slide 53 www.edureka.in/data-science
  54. 54. Hadoop Core Components Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker Data Node Task Tracker MapReduce Engine HDFS Cluster Job Tracker Admin Node Name node Slide 54 www.edureka.in/data-science
  55. 55. Hadoop is a framework that allows for the distributed processing of: - Small Data Sets - Large Data Sets Slide 55 www.edureka.in/data-science Annie’s Question
  56. 56. Large Data Sets. It is also capable to process small data-sets however to experience the true power of Hadoop one needs to have data in Tb’s because this where RDBMS takes hours and fails whereas Hadoop does the same in couple of minutes. Slide 56 www.edureka.in/data-science Annie’s Answer
  57. 57. For setting-up Hadoop on your system you can follow the “Hadoop Installation Guide” present in the LMS. Slide 57 www.edureka.in/data-science
  58. 58. Analytics with R Slide 58 www.edureka.in/data-science
  59. 59. Analytics with R Slide 58 www.edureka.in/data-sciencehttp://www.r-project.org/
  60. 60. R : Characteristics Slide 60 www.edureka.in/data-science  R is open source and free.  R has lots of packages and multiple ways of doing the same thing.  By default stores memory in RAM.  R has the most advanced graphics. You need much better programming skills.  R has GUI to help make learning easier.  Customization needs command line.  R can connect to many database and data types.
  61. 61. Comparing R and others http://r4stats.com/articles/popularity/ Comparing R Slide 61 www.edureka.in/data-science
  62. 62. Comparing R with Base SAS* /SAS Stat* Slide 61 www.edureka.in/data-science*Copyright © 2012 SAS Institute Inc., SAS Campus Drive, Cary, North Carolina 27513, USA. All rights reserved. R Base SAS* /SAS Stat* R is open source and free Base SAS* , SAS/Stat*, SAS/ET*, SAS/OR*, SAS/Graph* are expensive relatively because of annual licenses Open source R has support from email lists, twitter, stack overflow SAS Institute* products have dedicated support and extensive documentation R is slower on the desktop than base SAS for datasets ~4-5 gb By default R stores memory in RAM, so we can use the cloud R has much better graphics You need much better programming skills You can create custom functions in R easily Customization needs command line R has multiple GUI that are free SAS GUI are more expensive
  63. 63. Annie’s Question R Provides support in terms of: 1. Dedicated Support and Documentation 2. Email-lists, twitter, etc. Slide 63 www.edureka.in/data-science
  64. 64. Annie’s Answer Answer: 2. Email-lists, twitter, etc. Slide 64 www.edureka.in/data-science
  65. 65. Annie’s Question Custom functions can be easily created in : 1. SAS 2. R Slide 65 www.edureka.in/data-science
  66. 66. Annie’s Answer Answer: 1. R Slide 66 www.edureka.in/data-science
  67. 67. Annie’s Question Most of the functions in R are written in : - Java - R - C - Fortran Slide 67 www.edureka.in/data-science
  68. 68. Annie’s Answer Most of the user-visible functions in R are written in R. It is possible for the user to interface to procedures written in the C, C++, or FORTRAN languages for efficiency. Slide 68 www.edureka.in/data-science
  69. 69. Introduction to R Programming language www.r-project.org/about.html Slide 69 www.edureka.in/data-science  History  Open Source  Official Website  R Journal  Evolution  Free  R Core  Current State  Widely Recognized  Creators
  70. 70. R and Hadoop Integration  R and Hadoop are a natural match in Big Data Analytics and visualization.  One of the most well-known R packages to support Hadoop functionalities is : RHadoop  Rhadoop was developed by Revolution Analytics.  RHadoop is a collection of three R packages: rmr, rhdfs and rhbase.  rmr package provides Hadoop MapReduce functionality in R, rhdfs provides HDFS management in R and rhbase provides HBase database management from within R. file + Slide 70 www.edureka.in/data-science
  71. 71. For setting-up R on your system you can follow the “R Installation Guide” present in the LMS. Slide 71 www.edureka.in/data-science
  72. 72. Machine Learning Slide 72 www.edureka.in/data-science
  73. 73. Machine Learning: Mahout  Machine Learning is a class of algorithms which is data-driven, i.e. unlike "normal" algorithms it is the data that "tells" what the "good answer" is. Example: An hypothetical non-machine learning algorithm for face recognition in images would try to define what a face is (round skin-like-colored disk, with dark area where you expect the eyes etc). A machine learning algorithm would not have such coded definition, but will "learn-by-examples": you'll show several images of faces and not-faces and a good algorithm will eventually learn and be able to predict whether or not an unseen image is a face. Slide 72 www.edureka.in/data-sciencehttp://endthelie.com/2012/08/24/fbi-sharing-facial-recognition-software-with-police-departments-across-america/
  74. 74. Mahout Overview Mahout is about scalable Machine Learning Mahout has functionality for many of today’s common machine learning tasks Machine Learning is all over the web today MapReduce magic in action Slide 74 www.edureka.in/data-science
  75. 75. Hadoop and MapReduce magic in action Write intelligent applications using Apache Mahout LinkedIn Recommendations https://cwiki.apache.org/confluence/display/MAHOUT/Powered+By+Mahout Machine Learning: LinkedIn Recommendations Slide 75 www.edureka.in/data-science
  76. 76. Annie’s Question Mahout Algorithms for clustering, classification and collaborative filtering are implemented on top of Apache Hadoop using : - Flume - MapReduce - Sqoop - Hive Slide 76 www.edureka.in/data-science
  77. 77. Annie’s Answer Mahout Algorithms are implemented on top of Apache Hadoop using the Map/Reduce paradigm. Slide 77 www.edureka.in/data-science
  78. 78. 1. Install R with the help of “R Installation Steps” guide in the LMS. This is a step wise guide which will help you in installing and setting up R on your system Slide 78 www.edureka.in/data-science Assignment
  79. 79. Agenda for Next Class Slide 79 www.edureka.in/data-science In the next class you will be able to  Understand what is R  Describe why R is used?  Implement R Programming Concepts  Learn Data Import Techniques  Analyze the Processing of Data
  80. 80. Pre-work Go through the “R Essentials for Data Science” section in the LMS. Watch the recordings present in the section to gain an understanding of the R environment. Slide 80 www.edureka.in/data-science
  81. 81. What’s Within the LMS This section will give some prerequisites on R Programming Language. This section will give you an insight of machine learning This section will give you an insight of data science course This section provides an insight in Hadoop HDFS and MapReduce framework to manage data. Old Batch recordings – Handy for you Click here to expand and view all the elements of this Module Slide 81 www.edureka.in/data-science
  82. 82. What’s Within the LMS Recording of the Class Presentation Assignment Quiz Slide 82 www.edureka.in/data-science
  83. 83. What’s Within the LMS Further Reading Pre-work Installation Guide Slide 83 www.edureka.in/data-science
  84. 84. References Slide 84 www.edureka.in/data-science http://www.today.mccombs.utexas.edu/2012/04/the-big-data-machine http://www.espncricinfo.com/ http://www.majorprojects.vic.gov.au/our-projects/our-past-projects/austin-hospital http://wp.streetwise.co/wp-content/uploads/2012/08/Amazon-Recommendations.png http://smhttp.23575.nexcesscdn.net/80ABE1/sbmedia/blog/wp-content/uploads/2013/03/netflix-in-asia.png http://www.crowdsourcing.org/article/-nasa-tries-to-free-creativity-with-big-data-challenge/19984 http://whatsthebigdata.files.wordpress.com/2013/11/batman-on-big-data.jpg http://spinnakr.com/blog/wp-content/uploads/2013/08/Using-Big-Data-.jpg http://thesocietypages.org/sociologylens/files/2013/09/BIgDataDilbert_Cartoon.jpg http://abstrusegoose.com/55 http://www.thenewstribe.com/2013/11/15/diabetes-is-killing-one-patient-every-six-seconds/ http://www.mobigyaan.com/images/stories/Miscellaneous/mobigyaan-twitter-chat.jpg http://www.r-project.org/ http://endthelie.com/2012/08/24/fbi-sharing-facial-recognition-software-with-police-departments-across-america/
  85. 85. edureka ! /•