BETTER TOGETHER: USING SPARK AND REDSHIFT TO COMBINE YOUR DATA WITH PUBLIC DATASETS
•
3 gostaram•6,133 visualizações
The document discusses combining data from Jawbone with public datasets using Spark and Redshift. It describes how to model the problem, find and understand the different data sources, validate and fuse the data while accounting for confounding variables. It then provides examples of using Redshift and Spark to analyze the combined dataset and identify relationships between factors like temperature and daily steps. Overall, the key message is that data fusion can provide powerful insights but requires addressing challenges like noisy data and understanding the problem domain.
Recomendados
Recomendados
Mais conteúdo relacionado
Destaque
Destaque (9)
Semelhante a BETTER TOGETHER: USING SPARK AND REDSHIFT TO COMBINE YOUR DATA WITH PUBLIC DATASETS
Semelhante a BETTER TOGETHER: USING SPARK AND REDSHIFT TO COMBINE YOUR DATA WITH PUBLIC DATASETS (20)
Mais de C4Media
Mais de C4Media (20)
Último
Último (20)
BETTER TOGETHER: USING SPARK AND REDSHIFT TO COMBINE YOUR DATA WITH PUBLIC DATASETS
- 1. BETTER TOGETHER USING SPARK AND REDSHIFT TO COMBINE YOUR DATA WITH PUBLIC DATASETS EUGENE MANDEL (@EUGMANDEL) JAWBONE QCON SF 2014
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /hadoop-redshift-spark
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 6. %OFPEOPLEAWAKEAT3:25 DISTANCE FROM EPICENTER (MILES)
- 8. DATA FUSION IS THE PROCESS OF INTEGRATION OF MULTIPLE DATA AND KNOWLEDGE REPRESENTING THE SAME REAL-WORLD OBJECT INTO A CONSISTENT, ACCURATE, AND USEFUL REPRESENTATION. (WIKIPEDIA)
- 9. DATA FUSION - HOW TO FIND THE ELEPHANT IMAGE SOURCE: HTTP://COMMONS.WIKIMEDIA.ORG/WIKI/FILE%3ABLIND_MEN_AND_ELEPHANT.PNG
- 10. DATA FUSION POWERFUL BUT HARD DATA IS NOISY DOMAIN UNDERSTANDING IS KEY
- 11. LET’S TALK ABOUT THE WEATHER
- 14. FIND THE DATA
- 17. HOURLY DAILY
- 18. DATA GENERATION PROCESS NETWORK OF WEATHER STATIONS FREQUENCY OF MEASUREMENTS - HOURLY TO DAILY ! COLLABORATION WITH INTERNATIONAL AGENCIES ! AGGREGATION AND QA BY NCDC !
- 19. UNDERSTAND THE DOMAIN WEATHER STATION TIME: 2014-07-09 13:04:00 AIR TEMP: 86°F PRECIPITATION: 3CM
- 20. QA THE DATA
- 21. BUT ISN’T IT DONE?
- 22. AIR TEMP: 105°F BAKERSFIELD, CA JULY 17, 15:00 DULUTH, MN JAN 12, 05:00 …MAYBE NOT!
- 23. DATA VALIDATION DOMAIN KNOWLEDGE ! COMPARE MULTIPLE SOURCES - E.G. CLIMATE ! MANUAL REVIEW OF FLAGGED DATA POINTS
- 24. JOIN
- 25. DOMAIN SPECIFIC HOW? WEATHER STATION B LAT: 39.35 LON: -74.44 TIME: 2014-07-09 13:00:00 AIR TEMP: 60°F WEATHER STATION A LAT: 39.36 LON: -74.45 TIME: 2014-07-09 13:04:00 AIR TEMP: 74°F ELEVATION: 30FT ELEVATION: 120FT
- 26. DO THE DATASETS INTERSECT ENOUGH? COVERAGE PLACES ! TIMES ! USERS
- 28. CONFOUNDING VARIABLES WEEKDAYS/WEEKENDS ! DAYLIGHT ! RAIN/SNOW WHAT ELSE AFFECTS ACTIVITY?
- 30. AMAZON REDSHIFT RELATIONAL ANALYTICAL DATABASE BY AMAZON ! COMPLEX QUERIES ON LARGE DATASETS IN SECONDS ! SQL INTERFACE (POSTGRES) ! MANAGED CLUSTER
- 32. IN-MEMORY DATA PROCESSING FRAMEWORK ! MODELS COMPUTATION AS A GRAPH OF RDDS (RESILIENT DISTRIBUTED DATASETS) ! FUNCTIONAL PROGRAMMING MODEL (SCALA, PYTHON) ! SQL ! CAN READ FROM SAME SOURCES AS HADOOP
- 34. PICK YOUR OWN ADVENTURE SILVER BULLET? PROGRAMMER-FRIENDLY ! END-TO-END SOLUTION ! SELF-DOCUMENTING SPARK REDSHIFT EASY TO SHARE DATA WITH NON-DEVELOPERS ! MANAGED - EASY SCALING !
- 35. WHAT DID WE FIND?
- 36. IDEAL TEMP FOR MOVEMENT DAILYSTEPS MAX TEMP (F)
- 37. AND NOW BY STATE… DAILYSTEPS MAX TEMP (F)
- 38. HOURLY STEPS BY AIR TEMP WEEKENDS
- 39. LESS CHOICE = SMALLER EFFECT WEEKDAYS
- 40. DATA FUSION POWERFUL BUT HARD DATA IS NOISY DOMAIN UNDERSTANDING IS KEY
- 42. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/hadoop- redshift-spark