This document discusses recoding Google in Python. It summarizes Google's early use of Python for its crawler before switching to C++ for performance reasons. It then outlines the major components of a web search engine (crawler, parser, index, searcher/ranker) and discusses Python libraries and techniques that could be used to implement each component today, noting improved stability, performant C/Cython libraries, and algorithmic changes. It calls for contributors to help build out additional features and discusses open challenges.

1. Et si on recodait Google en Python ?
PyCon-FR 2016
sylvain@sylvainzimmer.com

5. transparence
reproductibilité

7. https://uidemo.commonsearch.org

8. https://explain.commonsearch.org/?q=python&g=en

10. Google's early Python code
https://www.quora.com/Why-did-Google-move-from-Python-to-C++-for-use-in-its-crawler
Python (1.2 IIRC) would occasionally just core dump while running the
crawler. It was completely stock, no C++ modules compiled in or dynamically
linked, just bog standard.
[...] no unit tests, and its "system tests" were minimal at best, absent at worst.
[...] there was originally some controversy about the switch. However, when
the new C++ system was turned on and used fewer machines to crawl 5x
faster with higher reliability, the practical question was settled.
Python was "abandoned" from the core search stack around 2000.

11. Qu'est-ce qui a changé depuis ?
• Stabilité & écosystème
• Librairies performantes en C / Cython
• Evolution des bottlenecks
• PyPy?

13. http://infolab.stanford.edu/~backrub/google.html
The Anatomy of a Large-Scale Hypertextual Web Search Engine (1998)
Crawler
Parser
Index
SearcherRanker

14. Crawler

15. http://scrapy.org

16. http://github.com/cocrawler/cocrawler

18. http://commoncrawl.org

19. Parser

20. HTML parsers
• BeautifulSoup & derivés.
• lxml
• html5lib
• Gumbo!

21. https://github.com/google/gumbo-parser

22. Extensions C en Python
Mémoire gérée par PythonMémoire gérée par l'extension C
PyObject
ctypes

24. Cython!
• Faire le gros du travail en C
• Eviter la conversion de données au maximum
• Générer une extension C pour Python facilement

27. https://github.com/sylvinus/cython-simple-examples

28. Gumbocy
• HTML envoyé au C en UTF-8, sans conversion
• Parcours de l'arbre en Cython
• Gestion de la visibilité & du boilerplate
• Attributs & tags ignorables, ...
https://github.com/commonsearch/gumbocy

29. https://github.com/commonsearch/urlparse4

30. Autres analyses
• Détection de langue : cld2
• Détection charset : cchardet + metatags/headers
• Cleaning titres & metadata

31. Index

32. https://pypi.python.org/pypi/Whoosh/

33. http://lucene.apache.org/

34. https://www.elastic.co

35. Ranker

36. Formule du ranking
rank = f( static_score , dynamic_score( query ) )
Alexa
DMOZ
Blacklists
PageRank
...
ElasticSearch & Lucene
TF-IDF
BM25

38. https://about.commonsearch.org/developer/get-started

39. Searcher

40. Go version:
https://github.com/commonsearch/cosr-front
https://github.com/commonsearch/cosr-back/blob/master/cosrlib/searcher.py

41. Frontend

42. https://uidemo.commonsearch.org

43. http://infolab.stanford.edu/~backrub/google.html
The Anatomy of a Large-Scale Hypertextual Web Search Engine (1998)
Crawler
Parser
Index
SearcherRanker

44. Qu'est-ce qui manque ?

45. Architecture
• 2-pass search (host clustering, result diversity)
• Indexation continue
• Infoboxes
• Pubs
• Verticaux (images, vidéos, news, science, ...)
• ...

46. Encore plus de fun
Spam / Relevance
Sustainability
Outreach
API
...

47. Ca vous tente?
https://about.commonsearch.org/contributing
https://github.com/commonsearch
contact@commonsearch.org
slack.commonsearch.org