At BarCamp Macau 2014, I give a talk about open source software, open source job board for drupal CMS, scrapy + django, and open source community.

1. Use open source
software
to develop ideas
at work
Sammy Fung
BarCamp Macau 2014

2. Sammy Fung
● Python Developer
– Perl → PHP → Python
● President at OpenSource.HK
● Community Volunteer in HK & Asia
– GNOME.Asia, Mozilla......

3. 將來市場領導者必會是
使用及貢獻開放源碼的
企業
Future market leader must be the one which
use and contribute open source.

4. 自由
FREEDOM

5. “ Free software means
software that respects
users' freedom and
community. ”
http://www.gnu.org/philosophy/free-sw.html

6. 4 Essential Freedoms
● Freedom 0: Run the program.
● Freedom 1: Open Source to study and change
it.
● Freedom 2: Re-distribute the program.
● Freedom 3: Distribute your modified versions
as free software.

7. Ecosystem in
Technology
Innovation

8. Ecosystem in Technology Innovation
● Operating Systems
● Servers with Applications
● Clients

9. Operating System
● Linux
– Server, Desktop, Embedded System.
– Mobile (Android).
● 1983: GNU projects developed many free software
programs replacing proprietary programs on UNIX
platforms.
● 1991: Linus Torvalds released first Linux kernel on
internet, then developers integrate with GNU softwares
(GNU/Linux) and distribute a new OS.
● Documentary Movie: Revolution OS (2001).

10. Web Server
● Apache Software Foundation.
● Apache HTTP Server (Web Server).
● Most popular HTTP server since April 1996.
● 1995: First release
● 2009: 1st web server serving 100 million websites.
● August 2014: 47.92% market share (1st)
– http://news.netcraft.com/archives/2014/08/27/august-2014-
web-server-survey.html
● Documentary Movie: Revolution OS (2001).

11. Applications
● Developing Web Applications and Mobile Applications
running on OS platforms.
● Huge amount of free software (open source software)
are esstential and used to develop, manage and
operate web & mobile apps and their platforms.
● Programming Languages:
– PHP, Python, Ruby......
● Web Frameworks, CMS, Blog......:
– Wordpress, Drupal, Django, Joomla......

12. Web Browser
● Mozilla Foundation (Found in 2003).
● Firefox Web Browser.
● Market Share: 15-20% (2014), 25-30% (2011).
● 1998: Netscape Communications Corp (creator of Netscape
Navigator web browser) created Mozilla project to coordinate open
source version of internet suite.
– Internet Suite: Web + Mail / Newsgroup + Web Composer + IRC.
● 2002: 1st release Firefox web browser (experimental branch of
Mozilla project).
● Sept 2014: Firefox 32.
● Documentary Movie: Code Rush

13. Example 1
Open Source
Job Board

14. 很多年前
Many years ago...

15. 找工作
Looking for a job

16. 一個香港徵才網站
At a popular job site

17. 傳回錯誤!
Got error!

18. 原因?
Why?

19. 你會猜到的!
You should get it!

21. 高!
High!

22. 市場佔有率
…...in Market Share

23. Also......

24. 高!
High!

25. ASP
CFM Nginx+PHP

26. ASP

27. 市場佔有率
…...in Market Share

28. 所以......
So......

29. ...我就決定...
...I decided...

30. 做開放源碼的
徵才網站軟件!
Develop open source
job board software

31. 從 Drupal 6 模組開始
Development starting from
Drupal 6 module

32. 但沒有發表
Never launched

33. 在今年
In this year

34. 轉移到 Drupal 7 模組
Porting from Drupal 6 to
Drupal 7

35. API 改變
API Changed

36. 修改了不少 codes
Changed some codes

37. GitHub: sammyfung / drupal7-job

38. 張貼徵才
Job Posting

39. 分類
Job Category

40. 公司資料
Company
Information

41. 每個分類也有
獨立RSS
RSS Feeds to
each job category.

42. 開放源碼
Open Source

43. 你就可以有擁有你
自己的徵才網站
You can own your
job board now

44. 如何?
How ?

45. 安裝Linux
網頁伺服器
Install a Linux
web server.

46. 安裝Drupal
內容管理系統
Install Drupal Content
Management System

47. 安裝這徵才網站
的Drupal模組
Install and enable this job
board module for Drupal

48. 完成!
Done!

49. job.opensource.hk

50. 如果你有任何職位需要懂
任何一種開源軟件技術...
If you got any job
positions which requires
any one of open source
software skill sets....

51. 就貼去
Post it to
job.opensource.hk

52. 免費!
It's Free of Charge!

53. 將來發展
Future
Roadmap

54. 推出Django
模組
Release a
Django module

55. 推出Wordpres
模組?
Release Wordpress
module ?

56. 推出Javascript
模組?
Release Javascript
module ?

57. 商業化
Commercialize

58. Example 2
Open Source
Weather Data
Parser

59. http://www.twitter.com/weatherhk

60. WeatherHK
● Reports
– Hourly Current Weather Information
– Weather Forecast
● Warning
– Tropical Signal issuing, updates, pre-warning.

61. WeatherHK
● Interview by MetroPop in 2009.

62. http://sammy.hk/projects/tctrack/tctrack.php

63. TCTrack
● Plot paths of any active Tropical Cyclones
– On Google Map
● Probably the 1st Tropical Cyclones map in Hong Kong using Google Map
API.
– sammy.hk → Weather Undergroud Hong Kong → HKO
– Current Paths.
– Forecast Paths.
– Tweets JTWC updates to Twitter.
– Data Sources:
● JTWC
● HKO

64. Open Data

66. Five Star Open Data
1.make your stuff available on the Web (whatever format)
under an open license.
2.make it available as structured data (e.g., Excel instead of
image scan of a table)
3.use non-proprietary formats (e.g., CSV instead of Excel)
4.use URIs to denote things, so that people can point at your
stuff.
5.link your data to other data to provide context.
5stardata.info by Tim Berners-Lee, the inventor of the Web.

67. Web Scraping
a computer software
technique of extracting
information from websites.
(Wikipedia)

68. Web Scraping
with Scrapy

69. Scrapy
● Python web scraping framework.
● Scrap websites and extract structured data.
● From data mining to monitoring and
automated testing.

70. Scrapy Features
● Define your own data structures.
● Write spiders to extract data.
● Built-in XPath selectors to extracting data.
● Built-in JSON, CSV, XML output.
● Interactive shell console, telnet console, logging......
● Scrapyd
– a daemon providing Web API and simple Web UI.
– http://localhost:6800

71. Scrapy Installation
$ apt-get install python python-virtualenv
python-pip
$ virtualenv env
$ source env/bin/activate
$ pip install scrapy

72. Creating a Scrapy Project
● Creating a new scrapy project.
– $ scrapy startproject <project name>
● Define your data structure
● Write your first spider
– Test with scrapy shell console
● Output / Store collected data
– Output with built-in supported formats
– Store to database / object store.

73. Scrapy + Django
● Django
– Python web framework
● Installing Django via pip
– $ pip install django
● Scrap with Scrapy and store data with Django

74. Creating django project
● Creating a Django project.
– $ django-admin.py startproject <project name>
● Define django settings.
– <project name>/settings.py
– Create database, tables and first django user.
● $ python manage.py syncdb
– Define installed django apps with
INSTALLED_APPS array.

75. Creating a Django App
● Create your own django app.
– $ python manage.py startapp <new app name>
● Define data model at models.py
– $ python manage.py syncdb
● Activate django admin UI at admin.py
– Add URL router to access admin UI.

76. Define django data model
class WeatherData(models.Model):
reporttime = models.DateTimeField()
station = models.CharField(max_length=3)
temperture = models.FloatField(null=True,
blank=True)
humidity = models.IntegerField(null=True,
blank=True)

77. Enable django admin ui
● Adding to INSTALLED_APPS at settings.py
– django.contrib.admin
● Adding URL router at urls.py
– $ python manage.py runserver
● Access admin UI
– http://127.0.0.1:8000/admin

78. Scrapy + Django
● Define django environment at scrapy settings.
– Load django configuration.
● Use Scrapy DjangoItem class
– Insteads of Item and Field class
– Define which django data model should be linked
with.
● Query and insert data at scrapy pipelines.

79. hk0weather

80. hk0weather
● Weather Data Project.
– https://github.com/sammyfung/hk0weather
– convert weather information to JSON data from
HKO webpages.
– python + scrapy + django

81. hk0weather
● Hong Kong Weather Data.
– 20+ HKO weather stations in Hong Kong.
– Regional weather data.
– Rainfall data.
– Weather forecast report.

82. Hk0weather installation
● Setup and activate a python virtual enviornment,
and install scrapy and django with pip.
● Clone hk0weather from GitHub
– $ git clone https://github.com/sammyfung/hk0weather.git
● Setup database connection at Django and create
database, tables and first django user.
● Scrap regional weather data
– $ scrapy crawl regionalwx -t json -o regional.json

84. How can you
use & participate
open source
software ?

85. 社群
Community

87. Hong Kong Linux User Group
● http://www.linux.org.hk/
● Re-established in 1997.
● Linux Talks, Seminiars and Workshops.
● Linux Cafe meetups.
● Linux Exhibitions and Demostrations.

89. Open Source Hong Kong
● http://www.opensource.hk
● Established in 2008.
● Events: Workshops, Hackathons, Conferences.

90. Monthly Events -
Open Source Workshops

91. OpenSource.HK Hackathons

92. Hong Kong Open Source
Conferences

93. Hong Kong Creative Open
Technology Association
● http://www.cota.hk
● Established in 2014.
● HK registered limited company in guarantee.
● Non-profit organization. (in final process)
● Promoting Open Standards, Free and Open Source
Software, Open Hardware, Free Culture Works, Open
Content and Creative Commons.
● Work on projects and events.
– eg. Hong Kong Open Source Conference 2014.

94. 自由
FREEDOM

95. “ Free software means
software that respects
users' freedom and
community. ”
http://www.gnu.org/philosophy/free-sw.html

96. 將來市場領導者必會是
使用及貢獻開放源碼的
企業
Future market leader must be the one which
use and contribute open source.

97. Welcome to invite me to......
● Meetings, Talks, Workshops and Events
– at your company and schools.
● Projects
– Development
– Consultation
● Let's get a name card.

98. Thank You!
sammy@sammy.hk