1. Scraping content from web for
location-based mobile app.

2. Nguyen Hong Diep
founder, magik.vn

3. Summary
1. Web Scraping
– Definitions
– Value added
– Analysis a Sample Case
2. Scrapy Framework
– Overview
– Architecture
– A simple Scrapy program.
3. Build a auto scraping system for location-based apps
– Extract LatLng from address
– Extract phone number
– Realtime update & continuous 24/7
– Prevent duplication data
– Deploy without a dedicated server or VPS

4. Web crawler
Internet bot that systematically browses
the World Wide Web,
typically for web indexing.
Sources: wikipedia.org

5. Scrape
Crawl websites and
extract structured data from pages.
Sources: wikipedia.org

6. Added Value?

7. giamua.com – “groupon”

8. baomoi.com

9. Added Value?
same user experience
but
more content than

10. oizoioi.vn
Price comparison for electronic

11. Added Value?
make
new knowledge
from many informations
Wisdom
Knowledge
Information
Data
DIKW Hierachy

12. Nha Tro Tot

14. Added Value?
The smartphone revolution
new platform
need
new user experienced
Source: www.widexconnect.ca

15. And mores
Sources : Laban.vn

16. Analysis a sample case
(1) collect [home for sales] records
from Web
(2) from many websites in Vietnam
(3) as soon as they posted
(4) continuous 24 / 7
Need

17. Step 1: Listing sources

18. Step 2: build general database

19. Step 3: Ctrl+C, Ctrl+V
• For every sites:
– Find listing latest records webpage link.
– For every record :
• Check if new record
– Copy & paste fields into a new record in my DB.

20. Step 3: Ctrl+C, Ctrl+V

21. Bước 3 : Let’s Scrapy

22. Scrapy Framework
• Overview
• Architecture
• Xpath
• Make a simple Scrapy program.

23. • Scrapy is a fast high-level screen
scraping and web crawling
framework.
• Open-source, 100% Python => Portable

24. Scrapy’s github info
• From 2008
• Stats

25. Architecture
Source: http://doc.scrapy.org/en/0.12/topics/architecture.html

26. XPath
Navigate through
elements and attributes
in an XML document.

27. Simple Scrapy Program
• (1) Pick a website
– http://www.mininova.org/today
• (2) Define the data you want to
scrape

28. Simple Scrapy Program (cont.)
• (3) Write a Spider to extract the data

30. Simple Scrapy Program (cont.)
(4) Run the spider to extract the data
(5) Review scraped data

31. Build a auto scraping system for
location-based apps
• Extract LatLng from address
• Extract phone number
• Realtime update & continuous 24/7
• Prevent duplication data
• Deploy without a dedicated server or
VPS

32. Extract LatLng from address
• Use Google Geocode
• https://maps.googleapis.com/maps/api/geocode/json?addr
ess=xxx&sensor=true_or_false&key=API_KEY

33. Extract LatLng from address (cont.)

34. Extract LatLng from address (cont.)

35. Extract Phone Number
• Libphonenumber’s python port.
• Sample

36. “Real time” update and
continuous 24/7.
• Task Schedule
(Windows)
• Cron jobs
(Linux)

37. Prevent duplication data
• Make a middleware for ignore exists
Item. IgnoreExistsMiddleW
are

38. Without a dedicated server or
VPS
• Problems: my server-side is on a cpanel
web hosting => can’t deploy scrapy
• Solutions:
– Make a web services for sync new record
data.
• /get_head_revision
• /sync
– Scrapy run on my PC, then sync with server.