1. Web Scraping 1-2-3 with
Python + Scrapy
Sammy Fung
sammy.hk, gownjob.com

2. Today Agenda
● Some Cases
● Python and Scrapy

3. Web Scraping
● a computer software technique of extracting
information from websites. (Wikipedia)
● for business, hobbies, research.......
● NOT talk about business cases today.

4. CableTV & NOWTV Programme
(Past)
● 2004.
● slow, slow, slow, or worst - can't connect.
● use Flash.

5. HK Observatory and Joint Typhoon
Warning Center
● no easy data exchange format, eg.
RSS/Atom.
● We won't check websites everyday.

6. Transportation - KMB, PTES
● no map view on KMB website for a bus route
in the past.
● Exteremly Poor, Ugly (or much worse) map
UI on PTES.

7. My experiences on web scraping
● 2004: php
● year after: python
● recent year: python with scrapy

8. Document Types
● HTML, XML,......
● Text
● Others, eg. pictures, videos,......

9. Web Scraping
● Look for right URLs to scrap.
● Look for right content from webpages.
● Saving data into data store.
● When to run the web scraping program ?

10. What is Scrapy ?
● An open source web scraping framework for
Python.
● Scrapy is a fast high-level screen scraping
and web crawling framework, used to crawl
websites and extract structured data from
their pages. It can be used for a wide range
of purposes, from data mining to monitoring
and automated testing.

11. Features of Scrapy
● define data you want to scrapy
● write spider to extract data
● Built-in: selecting and extracting data from
HTML and XML
● Built-in: JSON, CSV, XML output
● Interactive shell console
● Built-in: web service, telnet console, logging
● Others

12. Installation of Scrapy
● pip
● APT repo
● RPM
● tarball (binary/source)

13. Create new scrapy project
$ scrapy startproject mybot
mybot/
mybot/scrapy.cfg
mybot/mybot/items.py
mybot/mybot/pipeline.py
mybot/mybot/settings.py
mybot/mybot/spiders/myspider.py
etc.......

14. items.py
from scrapy.item import Item, Field
class HKOCurrentItem(Item):
time = Field()
station = Field()
temperature = Field()
humidity = Field()
#......

15. spiders/hko_spider.py (1/5)
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from weatherhk.items import HKOCurrentItem
import datetime, re

16. spiders/hko_spider.py (2/5)
class HKOCurrentSpider(BaseSpider):
name = "HKOCurrentSpider"
#allowed_domains = ["www.weather.gov.hk"]
start_urls = [
"http://www.weather.gov.
hk/textonly/forecast/chinesewx.htm"
]

17. spiders/hko_spider.py (3/5)
def parse(self, response):
hxs = HtmlXPathSelector(response)
stations = []
# Getting weather data from each stations.
tx = hxs.select("//pre[1]/text()").re('[^n]*n')

18. spiders/hko_spider.py (4/5)
for i in tx:
if re.search(u'd 度',i):
data = HKOCurrentItem()
data['time'] = int(dt)
data['station'] = self.station.code(i)
data['temperature'] = int(re.findall(u'd+',i)
[0])
stations.append(data)

19. spiders/hko_spider.py (5/5)
return stations

20. pipelines.py (1/2)
class HKOCurrentPipeline(object):
def process_item(self, item, spider):
station = self.db[item['station']]
storeditem = dict(item.__dict__)['_values']

21. pipelines.py (2/2)
try:
if 'temperature' in storeditem:
lasttime = station.find({'temperature': {'$gt':
0}}).sort('time', -1).limit(1)
if lasttime[0]['time'] != storeditem['time']:
id = self.insert(storeditem)
return item