Some tips for crawling processes.

1. Crawling the world
@mmoreram

2. Apparently...

3. Nobody
uses parsing in their applications

4. Not even
Chuck Norris

5. Many bussinesses
need crawling

6. Crawling brings you
knowledge
Knowledge is
power

7. And power is
Money

8. What is crawling?
Or
parsing

9. Crawling
We download just an url with a request (HTML, XML…)
We manipulate response by searching the desired data,
like links, headers or any kind of text or label
Once we have needed content, we can just update our
database and take following decisions, for example
parsing some found links.
and that’s it!

10. “Machines will do what humans
do before they realize”
–Marc Morera, yesterday

11. Let’s see an example
Step by step-

13. chicplace.com
Our goal is parse all available products, saving name,
description, price, shop and categories
We will use linear strategy. There are some kind of
strategies when a site must be parsed
Let’s see all available strategies

14. Parsing Strategies
Linear. Just one script. If any page fails (crawling error,
server timeout, …) some kind of exception could be
thrown and catched.
Advantages: Just an script is needed. Easier? Not even
close…
Problems: Cannot be distributed. Just one script for 1M
requests. Memory problem?

15. Parsing Strategies
Distributed. One script for each case. If any page fails
can be recovered by simply execute himself again.
Advantages: All cases are encapsulated in an individual
script, low memory. Can be easily distributed by using
queues.
Problems: Any

16. Crawling steps
Analyzing. Think like Google does. find the fastest way
through the labyrinth
Scripting. Build scripts using queues for distributed
strategy. Each queue means one page
Running. keep in mind the impact of your actions. DDOS
attack, copyright

17. Analyzing
Every parsing process should be evaluated as a simple
crawler. For example Google
How to access to all needed pages with the lowest
server impact
Usually, all serious websites are designed to easily
access to all pages within 3 clicks

18. Analyzing
We will use category map to just access to
all available products

19. Analyzing
Each category will list all available products

20. Analyzing
Do we need also to parse product page?
In fact, we do. We already have name, price and
category, but we also need description and shop
So we have main page to parse all category links, we
have category page with all product ( can be paginated )
and we need also product page to get all information
Product page is responsible for saving all data in DDBB

21. Scripting
We will use distributed strategy, using queues and
supervisord
Supervisord is responsible for managing X instances of
a process running at the same time.
Using distributed queue system, we will have 3 workers.

22. Worker?
Yep, worker. Using a queue system, a worker is like a
box ( script ) with a parameters ( input values ), that just
do something.
We have 3 kind of workers. One of them, the
CategoryWorker will just receive a category url, will
parse related content ( HTML ) and will detect all
products. Each product will generate a new instance for
ProductWorker

23. Running
We just enable all workers and forces first to run.
First worker will find all categories urls and will
enqueue them into a queue named categories-queue
Second worker ( for example 10 instances ) will just
consume categories-queue looking for urls and parsing
their content.
Their content means just products urls

24. Running
Each url is enqueued to another queued named
products-queue
Third and last worker ( 50 instances ) just consume this
queue, parses their content and get needed data ( name,
description, shop, category and price.

25. OK. Call me God

26. but…

27. “Don't shoot the messenger”
–Some bored man

28. warning!
50 workers requesting chicplace in parallel. This is a big
problem
@Gonzalo (CTO) will be angry and he will detect
something is happening
So, we must be careful to not alert him or just prevent
us discover

29. Warning
do not try this at home

30. Be invisible
To be invisible we just can parse all site slowly ( days )
To be faster we just can mask our IP using Proxies
( How about different proxy for every request? )
To be faster we just can user some reversed Proxy, like
TOR.
To be stupid we can just parse chicplace with our IP
( most companies will not even notice )

31. They are attacking
me !

32. “
And whatever you ask in prayer,
you will receive, if you have faith”
–Matthew 21:22

33. My pray!
A good crawling implementation is infallible
Server will receive dozens of requests per second and
will not recognize any pattern to discriminate crawler
requests from simple user requests
So…?

34. Welcome to amazing world of
Crawling

35. Where no one is
SAVE