用小數據看 台灣大數據生態系的 社群流轉 Observe Changes of Taiwan Big Data Communities with Small Data Shared at 2018-06-13 < 交通大學數據科學與工程研究所 Seminar >

1. 2018-06-13 ( ) 15:30~16:20
1001
Initiator and Chair, TDEA
Data Architect, TenMax
AVP, Product Management, Etu
Associate Researcher, NCHC
Observe Changes of Taiwan Big Data Communities with Small Data
Small Data
, 2013 2018 Indeed
( D i g i t a l Transformation)
Social Network Analysis

2. My Journey of “Innovation”
( aka “From Zero to One” )
用小數據看
台灣大數據生態系的
社群流轉
Observe Changes of Taiwan Big Data
Communities with Small Data
Jazz Yao-Tsung Wang
Initiator and Chair, TDEA
Data Architect, TenMax
Shared at 2018-06-13 < Seminar >

3. Hello!
I am Jazz Wang
Co-Founder of Hadoop.TW
Initiator and Chair of Taiwan Data Engineering Association (TDEA)
Hadoop Evangelist since 2008.
Open Source Promoter. System Admin (Ops).
- 11 years (2002/08 ~ 2014/02) Associate Researcher in HPC field.
- 2 years (2014/03 ~ 2016/04) Assistant Vice President (AVP),
Product Management of ‘Big Data Platform Management’
- 2 years (2016/04 ~ Now) Data Architect of Real-Time Bidding
You can find me at @jazzwang_tw or
https://fb.com/groups/dataengineering.tw
https://slideshare.net/jazzwang
3

4. 4
Hadoop ~ ~
2006 2 2008 4
~$ whois hadoop.tw
Domain Name: hadoop.tw
Record created on 2008-04-28 (YYYY-MM-DD)

5. 5
~
- Data Engineer – Vendor Site – – –
- Data Scientist – User Site – – –

6. 6
● ● ● ●
HR
Do you believe in “Data-Driven” ??

7. 7
/
Q1 -
- ? ? ? ? ?
Q2 - (Dimensions) (Measures)
- ? ? ?
- ? ? ?

8. 8
Hadoop
• Indeed Hadoop
•
• 2013/02/07~2018/06/11
http://www.indeed.com/jobtrends/q-Hadoop.html
http://tw.indeed.com/jobs?q=Hadoop&l=Taiwan

9. 9
Indeed
•
• —
• 1998 :)
• PS. / ( )
• Unix, Linux DOS AI/DO Linux …
https://plus.nctu.edu.tw/courses/22241

10. 10
Hadoop
• 1,255 ( Gmail mbox Python
• 5,647
• 500 (

11. - Indeed.mbox
11https://takeout.google.com/settings/takeout/custom/gmail

12. mbox Python
12https://github.com/jazzwang/show-mbox
Binary File Git Repo
Indeed.mbox
CSV Git Repo

13. (1)
80%
13
Job Company
1. 2016-11-17 E-mail HTML
2.2016-11-17 CSS
class=“sg-paragraph-large”
class=“sg-paragraph-large db”
case
3. 2016-11-17 HTML Tag “n” “ ”

14. (2)
80%
14
Job Location
1. 2016-11-17 E-mail HTML
2.2016-11-17 CSS
class=“sg-paragraph-large db”
3. 2016-11-17 HTML Tag “n” “ ”
4. 2016-11-17 HTML Tag ( - )

15. (3)
80%
15
Job Title
1. 2016-11-17 E-mail HTML
2. 2016-11-17 HTML Tag ‘ “> ‘
3. 2016-11-17 HTML Tag “n” “ ”

16. Tableau
16
Hadoop E-mail
PS. URL ( )
2017 11 2016 11
2018 Hadoop

17. 17

18. 18

19. 19

20. https://public.tableau.com/profile/jazz.wang#!/vizhome/Hadoop_Job_Taiwan_2018-06/Dashboard1 20
197
2018

21. 2016 12
21

22. 22

23. ▷ Top 15
○
■ → → /
→ → → (?)
▷ Top 15
○ Job Title URL ?
○ Job Title URL
→ /
23

24. Tableau Public
▷ Mac OS brew cask install tableau-public
24https://public.tableau.com/profile/jazz.wang#!/vizhome/Hadoop_Job_Taiwan_2018-06/Dashboard1

25. Google Data Studio
25
https://datastudio.google.com

26. ## Part 1 : Indeed Job Analytics
1. http://blog.chunnorris.cc/2015/04/python-2x-unicode.html
2. https://stackoverflow.com/questions/3806874/how-to-merge-two-files-line-
by-line-in-bash
3. https://stackoverflow.com/questions/466345/converting-string-into-
datetime
4. https://stackoverflow.com/questions/19351541/excluding-unwanted-
results-of-findall-using-beautifulsoup
5. https://pythonadventures.wordpress.com/2010/10/11/chomp-functionality-
in-python/
6. https://www.cnblogs.com/pylemon/archive/2011/05/18/2050179.html
26

27. 27
/
Q1 -
- ? ? ? ? ?
Q2 - (Dimensions) (Measures)
- ? ? ?
- ? ? ?

28. “
LinkedIn Profile
28
(HR 4.0?)

29. 29
Hadoop
• Cloudera (CCDH/CCAH)
• LinkedIn
•
• 2015-05-12 LinkedIn API
• LinkedIn
•
• 62 Cloudera LinkedIn

30. 30
LinkedIn Advanced People Search
• 3837
• Cloudera (119) Apple ( 79)
Cognizant (68) eBay (55)
Hortonworks (47)
•
• IT

31. 31
Hadoop
• Hadoop
• (2012) → / / / → / / (2015)
• (SI)

32. { }
32
解讀雲端大數據新趨勢
Big Data Stack on The Cloud
Shared at 2018-05-16 < iThome Cloud Summit 2018 >

33. 33

34. 34

35. —
35
https://www.facebook.com/groups/dataengineering.tw https://www.facebook.com/groups/datasci.tw

36. Facebook Graph API 2018
36
https://github.com/HadoopTW/communities-dashboard
Facebook
Graph API 2.10
3.0
: Selenium

37. 37
Hadoop.TW
•
• Hadoop.TW 5,534 (@2018-06-13)
• Facebook Graph API
• HTML

38. Selenium Python Code
38
https://github.com/jazzwang/snippet/tree/master/selenium_lab

39. 39
Hadoop.TW
• 2013 SNA
• http://www.slideshare.net/jazzwang/ss-50602379
•
• (post) (comment) (like)
•

40. ▷ Selenium Facebook member
○ “ (group_name)”, “ (user_id)”, “ (join_time)”
▷ https://github.com/HadoopTW/communities-dashboard Gephi
Social Network Analytics
○ https://gephi.org/
▷ member
▷ D3.js Sankey Diagram
○ https://d3js.org/
○ http://benlogan1981.github.io/VerticalSankey/UBS.html
40

41. ## Part 2 : Hadoop.TW member Analytics
1. https://www.youtube.com/watch?v=49Mwqbu2cMo
2. https://github.com/mozilla/geckodriver/releases
41

42. ▷ ( )
○ -
○ -
○ -
▷ ( )
○ Python / Shell Script
■ Data Science at the Command Line
https://www.datascienceatthecommandline.com/
○ Crawler - Ex. Selenium, Scrapy
○ Data Visualization - Ex. Tableau, Google Data Studio, D3.js
▷ -
○ Intern
○ Kaggle
■ Ex. Streaming Data Input → Data Quality / Data Governance
→ Datawarehouse → BI Report
42

43. 43
https://goo.gl/2z9BGK

44. Thanks!
Any questions?
You can find me at @jazzwang_tw or
https://fb.com/groups/dataengineering.tw
https://slideshare.net/jazzwang
44