O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Apache Drill and Zeppelin: Two Promising Tools You've Never Heard Of

Study after study shows that data preparation and other data janitorial work consume 50-90% of most data scientists’ time. Apache Drill is a very promising tool which can help address this. Drill works with many different forms of “self describing data” and allows analysts to run ad-hoc queries in ANSI SQL against that data. Unlike HIVE or other SQL on Hadoop tools, Drill is not a wrapper for Map-Reduce and can scale to clusters of up to 10k nodes.

  • Entre para ver os comentários

Apache Drill and Zeppelin: Two Promising Tools You've Never Heard Of

  1. 1. @OPENDATASCI Apache Drill & Zeppelin: Two Promising Tools You’ve Never Heard Of @OPENDATASCI OPEN DATA SCIENCE CONFERENCE Charles Givre @cgivre, givre_charles@bah.com S A N F R A N C I S C O | 2 0 1 5
  2. 2. @OPENDATASCI Who am I?
  3. 3. Charles Givre @cgivre givre_charles@bah.com
  4. 4. @OPENDATASCI Apache Zeppelin
  5. 5. @OPENDATASCI
  6. 6. @OPENDATASCI Drill is the first schema free SQL Engine
  7. 7. @OPENDATASCI Drill allows you to query self-describing data wherever it is, using standard SQL.
  8. 8. @OPENDATASCI Drill allows you to query self-describing data wherever it is, using standard SQL.
  9. 9. @OPENDATASCI Drill allows you to query self-describing data wherever it is, using standard SQL.
  10. 10. @OPENDATASCI Open Source
  11. 11. @OPENDATASCI Flexible and Extensible
  12. 12. @OPENDATASCI Scale
  13. 13. @OPENDATASCI + =
  14. 14. @OPENDATASCI +( )+ =
  15. 15. @OPENDATASCI +( )+ = SELECT <fields> FROM dfs.strata.`file1.csv.` f1 JOIN dfs.strata.`file2.json` f2 ON f1.id = f2.id WHERE <something is true> GROUP BY f1.field1 ORDER BY f2.name DESC
  16. 16. @OPENDATASCI ETL
  17. 17. @OPENDATASCI
  18. 18. @OPENDATASCI
  19. 19. @OPENDATASCI
  20. 20. @OPENDATASCI { “employee_id":1, "full_name":"Sheri Nowmer”, “first_name":"Sheri", “last_name”:”Nowmer", … "management_role":"Senior Management” }
  21. 21. @OPENDATASCI SELECT management_role, COUNT( employee_id ) as roleCount FROM cp.`employee.json` GROUP BY management_role ORDER BY roleCount DESC
  22. 22. @OPENDATASCI
  23. 23. @OPENDATASCI
  24. 24. @OPENDATASCI
  25. 25. @OPENDATASCI import requests import pandas as pd import json url = "http://localhost:8047/query.json" employee_query = """SELECT management_role, COUNT( employee_id ) as roleCount FROM cp.`employee.json` GROUP BY management_role ORDER BY roleCount DESC""" data = {"queryType" : "SQL", "query": employee_query } data_json = json.dumps(data) headers = {'Content-type': ‘application/json'} response = requests.post(url, data=data_json, headers=headers) df = pd.DataFrame( response.json()['rows'] ) print( df.head() )
  26. 26. @OPENDATASCI management_role roleCount 0 Store Full Time Staf 764 1 Store Temp Staff 264 2 Store Management 100 3 Middle Management 17 4 Senior Management 10 http://bit.ly/1MJZ7QX : Accessing Drill via REST http://bit.ly/1GWbNIq: Accessing Drill via ODBC
  27. 27. @OPENDATASCI Performance
  28. 28. @OPENDATASCI
  29. 29. @OPENDATASCI   Drill SQL-on-Hadoop (Hive, Impala, etc.) Use case Self-service, in-situ, SQL-based analytics Data warehouse offload Data sources Hadoop, NoSQL, cloud storage (including multiple instances) A single Hadoop cluster Data model Schema-free JSON (like MongoDB) Relational User experience Point-and-query Ingest data → define schemas → query Deployment model Standalone service or co-located with Hadoop or NoSQL Co-located with Hadoop Data management Self-service IT-driven SQL ANSI SQL SQL-like 1.0 availability Q2 2015 Q2 2013 or earlier https://drill.apache.org/faq/
  30. 30. @OPENDATASCI https://drill.apache.org
  31. 31. @OPENDATASCI Apache Zeppelin
  32. 32. @OPENDATASCI What is it?
  33. 33. @OPENDATASCI A web based notebook that enables interactive data analytics
  34. 34. @OPENDATASCI Built in integration with
  35. 35. @OPENDATASCI
  36. 36. @OPENDATASCI Zeppelin makes exploring data easy
  37. 37. @OPENDATASCI
  38. 38. @OPENDATASCI val bankText = sc.textFile("/Users/cgivre/Downloads/bank/bank-full.csv") case class Bank(age:Integer, job:String, marital : String, education : String, balance : Integer) val bank = bankText.map(s=>s.split(";")).filter(s=>s(0)!=""age"").map( s=>Bank(s(0).toInt, s(1).replaceAll(""", ""), s(2).replaceAll(""", ""), s(3).replaceAll(""", ""), s(5).replaceAll(""", "").toInt ) ) bank.toDF().registerTempTable("bank")
  39. 39. @OPENDATASCI %sql SELECT marital, COUNT( 1 ) AS statusCount FROM bank GROUP BY marital ORDER BY statusCount DESC
  40. 40. @OPENDATASCI
  41. 41. @OPENDATASCI
  42. 42. @OPENDATASCI
  43. 43. @OPENDATASCI
  44. 44. @OPENDATASCI %sql SELECT education, age, AVG( balance) as avgBalance FROM bank WHERE age > 20 AND age < 35 GROUP BY age, education ORDER BY age ASC
  45. 45. @OPENDATASCI
  46. 46. @OPENDATASCI
  47. 47. @OPENDATASCI
  48. 48. @OPENDATASCI %sql SELECT education, age, AVG( balance) as avgBalance FROM bank WHERE age > ${minimumAge=30} AND age < ${maximumAge=35} GROUP BY age, education ORDER BY age ASC
  49. 49. @OPENDATASCI
  50. 50. @OPENDATASCI
  51. 51. @OPENDATASCI %sql SELECT age, count(1) value FROM bank WHERE marital="${marital=single,single|divorced|married}" GROUP BY age ORDER BY age
  52. 52. @OPENDATASCI
  53. 53. @OPENDATASCI
  54. 54. @OPENDATASCI https://zeppelin.incubator.apache.org/
  55. 55. @OPENDATASCI Questions? givre_charles@bah.com @cgivre

×