AsyncIO can speed up web scraping by allowing asynchronous requests and parsing to occur concurrently without blocking. It uses coroutines and an event loop to schedule tasks. For scraping, URLs can be fetched asynchronously using aiohttp. Results are gathered after tasks complete without waiting sequentially. Performance can be monitored using tools like aiomonitor which provide a task process list and console. MongoDB can be used to save crawling results with the batch ID and track success/error counts.

1. AsyncIO
To Speed Up Your Crawler
Linggar Primahastoko
x@linggar.asia
http://linggar.asia
Python ID (Jogja)
@Prism Jogja
03 Maret 2018

2. Who am I?

Typing Python frequently

Sometimes writing PHP

Also playing with Golang

Member of Sherina Fans Club mailing list since
2003

3. AsyncIO?

5. Understanding The Terms

7. Sync Async

Task are executed one by one

One process runs after the
previous process is complete

Task start and continue
running while the execution
moves on to a new task

Async tasks do not block the
operation

Usually run in background

8. https://www.semantics3.com/blog/the-buzzwords-in-product-apis-explained-7b79b2cc0d7b

9. Parallelism Concurrency

Making progress in parallel

Executing more than one task
at the same time

More than one task at the
same time

Sharing time slices from the
same core

10. Multithreading VS Multiprocessing

11. Multithreading

Share Heap Memory

Each thread do specific task

Own code, own stack memory

Limited By GIL (Global Interpreter Lock)

12. Multiprocessing

Separate Memory Space

Multiple CPU Cores

Bypass GIL Limitation

13. Why AsyncIO?

14. Reasons

Better event distribution

Better task management

15. AsyncIO

Coroutine: Like Generator, but coroutine
consume data. Coroutine can pause and
resume. It’s a way of pausing a function and
returning a series of values periodically

Future / Task : A Future is an object that is
supposed to have a result in the future. A Task
is a subclass of Future that wraps a coroutine.
When the coroutine finishes, the result of the
Task is realized. Subclass of Future that wraps
coroutine.

Event Loop: Central Executor in AsyncIO. An

16. Event Loop
https://medium.com/@gauravbaliyan/a-good-use-of-asyncio-in-python-and-django-8aa7bc401e5f

17. Web Scraping

18. Requirements

requests: http://docs.python-requests.org/en/master/
HTTP for Humans

beautifulsoup4: https://pypi.python.org/pypi/beautifulsoup4
HTML / XML Parser

19. Purpose

Get all URLS from a single page

Do it fast

20. Request and parse

Requests
page = requests.get(main_url)

Parse
bs = BeautifulSoup(page.text, 'html.parser')

Get Urls
urls = []
url_list = bs.find_all('a', href=True)
for url in url_list:
if url['href'].startswith('http'):
urls.append(url['href'])

21. Using AsyncIO

22. Requirements

AsyncIO:
https://docs.python.org/3/library/asyncio.html

Aiohttp: https://aiohttp.readthedocs.io/en/stable/

Python 3.6
Because we use async def func() and await
syntax, we need Python 3.5++

23. Coroutine

async def fetch(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
html = await response.read()
print("{}, length: {}".format(url, len(html)))
return {
'resp': response,
'html': html,
'url': url,
}

24. ensure_future
Schedule the execution of a coroutine object:
wrap it in a future. Return a Task object.
futures = [asyncio.ensure_future(fetch(url)) for url in urls]

25. Results
results = []
for future in futures:
try:
resp = await future
results.append(resp)
except Exception as error:
print(url, error)
continue

26. Start Event Loop
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))
loop.close()

27. Comparing
Sync Request
Async Request

28. Monitoring

29. aiomonitor
if __name__ == '__main__':
loop = asyncio.get_event_loop()
with aiomonitor.start_monitor(loop=loop):
loop.run_until_complete(main(loop))
aiomonitor is Python 3.5+ module that adds monitor and cli capabilities
for asyncio application. Idea and code borrowed from curio project.
Task monitor that runs concurrently to the asyncio loop (or fast drop in
replacement uvloop) in a separate thread as result monitor will work
even if event loop is blocked for some reason.
https://github.com/aio-libs/aiomonitor

31. aiomonitor Process List

32. https://github.com/aio-libs/aiomonitor/blob/master/aiomonitor/monitor.py
def start(self) -> None:
assert not self._closed
assert not self._started
self._started = True
h, p = self._host, self._port
self._ui_thread.start()
if self._console_enabled:
log.info('Starting console at %s:%d', h, p)
self._console_future = init_console_server(
self._host, self._console_port, self._locals, self._loop)
https://github.com/aio-libs/aiomonitor/blob/master/aiomonitor/utils.py
def init_console_server(host: str,
port: int,
locals: OptLocals,
loop: Loop) -> Future:
def _factory(streams=None):
return aioconsole.AsynchronousConsole(
locals=locals, streams=streams, loop=loop)
coro = aioconsole.start_interactive_server(
host=host, port=port, factory=_factory, loop=loop)
console_future = asyncio.run_coroutine_threadsafe(coro, loop=loop)
return console_future

33. Process List
def _command_ps(self, sout: IO[str]) -> None:
headers = ('Task ID', 'State', 'Task')
table_data = [headers]
for task in sorted(asyncio.Task.all_tasks(loop=self._loop), key=id):
taskid = str(id(task))
if task:
t = 'n'.join(wrap(str(task), 80))
table_data.append((taskid, task._state, t))
table = AsciiTable(table_data)
sout.write(table.table)
sout.write('n')
sout.flush()

34. Monitor Crawling Results

35. Flow

Using MongoDB to save crawling results

Insert to batch collection and get batch ID

Start tasks

List all pending tasks

If there is any error, save to error collection

Iterate finished task to get the result and save
to news collection

Update batch with the number of success and
error results

36. List Pending Tasks
task_ids = {}
for task in sorted(asyncio.Task.all_tasks(loop=loop), key=id):
t = str(task)
taskid = str(id(task))
task_ids[taskid] = task._state
Save Results
tasks = asyncio.Task.all_tasks(loop=loop)
for task in tasks:
try:
if 'result' in dir(task):
task_result = {
'id': id(task),
'batch_id': batch_id,
'before': task_ids[str(id(task))],
'after': task._state,
'url': task.result()['url']
}
db.news.insert(task_result)
except:
pass

38. Another Things

Schedule a call

call_soon

call_later

Semaphore

BoundedSemaphore

etc