Anatomy of a Large Django site

Anatomy of a large
Django site

Andy McKay Mozilla

mozilla

Python
Zope and Plone
...now at Mozilla

mozilla

Using Django
http://www.djangoproject.com

Credit: http://www.ﬂickr.com/photos/abiavati/3110357974/
mozilla

1. About the site
2. Performance
3. Localisation
4. Reuse

mozilla

1. About the site

mozilla

All code is open:
https://github.com/jbalogh/zamboni

mozilla

All* bugs are open:
https://bugzilla.mozilla.org

mozilla

Convert from
CakePHP (remora)
to Django (zamboni)

mozilla

Credit: http://www.ﬂickr.com/photos/improbcat/4177702580/
mozilla

Changing one URL at a time
from CakePHP to Django
General trend to move
away from PHP and do
more Python and Django

mozilla

How large?
250k+ addons
150 mn views month
500 mn API hits day

mozilla

Lines of code
PHP 40k
Python 18k

mozilla

Lines of code
PHP 40k
Python 18k
Unit tests 15k

mozilla

running both php and
python side by side. a
few issues on that

No pages go out until
they are faster

mozilla

3 zeus load balancers
24 django (and php)
1 mysql + 4 slaves
3 memcached
3 sphinx
1 rabbitmq + 2 celeryd
1 redis master + 1 slave
Credit: http://www.ﬂickr.com/photos/tbridge/15300843/
mozilla

2. Performance

mozilla

As usual, database bottleneck

mozilla

Cache machine
http://bit.ly/cache-machine

Credit: http://www.ﬂickr.com/photos/mwichary/4063534688/
mozilla

from django.db import models
import caching.base

class Addon(caching.base.CachingMixin,
models.Model):
...
status = models.IntegerField()
objects = caching.base.CachingManager()
available as a mixin

need to addin the custom
manager

mozilla

>>> Addon.objects.filter(status=public)
>>> len(connection.queries)
13

mozilla

13

13

mozilla

Invalidation

mozilla

md5(‘select... a’) [addon 3615]

mozilla


addon 3615 md5(‘select... a’)

mozilla


md5(‘select... b’) [addon 3615, addon 1685]

md5(‘select... a’)
addon 3615
md5(‘select... b’)

mozilla

Memcached
rules = cache.get(3615)
rules.add('select...')
cache.set(3615, rules)

mozilla

Redis
redis.SADD(3615, ‘select...’)

mozilla

Home page
20+ addons
400+ sql queries

mozilla

add-on version

version

version files
standard answer in
django is select-related

files

files

mozilla

django: select_related()
http://bit.ly/select-related

mozilla

simonw

Transformer
http://bit.ly/queryset-transform

mozilla

@staticmethod
def transformer(addons):
addon_dict = dict((a.id, a) for a in addons)
vs = filter(None, (a.current_version_id for a in addons)
versions = list(Version.objects.filter(id__in=vs))
for version in versions:
addon_dict[version.addon_id].current_version = version

slightly outdated
example

mozilla

big SQL statements... :(

14313 character

Home page
20+ addons
~14 sql queries

mozilla

Update
Called on startup
about:conﬁg extensions.update.url

mozilla

Incoming 8,000 req/sec
Uncached 1,600 req/sec
Im used to Plone in the
bad old days

mozilla

Incoming 8,000 req/sec
Uncached 1,600 req/sec
PHP 550 req/sec
Im used to Plone in the
bad old days

mozilla

v1
Plain
Django

PHP Python

mozilla

v2
Min. SQL
queries

PHP Python

mozilla

v3
Django and
raw SQL
max-requests 200,
actually we hit 210

PHP Python

mozilla

v4
WSGI
no Django

PHP Python

mozilla

v5
Pooling,
optimised
queries
Thats 700 req/sec
PHP Python which translates into

mozilla

Reducing the SQL queries...
doesn’t always help

mozilla

mySQL query cache is fast

mozilla

Celery
http://celeryproject.org

Credit: http://www.ﬂickr.com/photos/chiotsrun/3843988392/
mozilla

Push things async
email
image processing
add-on validation
specifically fixing data
changes bet ween php
and python

mozilla

from celeryutils import task

@task
def update_tag(tag, **kw):
tag.update_stat()

mozilla

from celeryutils import task

@task(rate_limit='60/h')
def update_tag(tag, **kw):
tag.update_stat()

mozilla

from tasks import update_tag

update_tag.delay(tag)

mozilla

Measurement

mozilla

Timing Middleware
http://bit.ly/timing-ware

Credit: http://www.ﬂickr.com/photos/wwarby/3297205226/
mozilla

3. Localization

mozilla

show site in arabic?

40+ languages
including rtl

mozilla

content translated

and

templates

Database strings

mozilla

class Addon(caching.base.CachingMixin,
models.Model):
...
name = models.ForeignKey(Translation)

mozilla

addon.name = 'name'
addon.save()

mozilla

addon.name = 'name'
addon.save()

addon.name = {'fr': 'la nomme'}
addon.save()

mozilla

Django
{% blocktrans with app=request.APP %}
Add-ons for {{ app }}
{% endblocktrans %}

mozilla

Jinja2
http://jinja.pocoo.org/
{{ _('Add-ons for {0}')|f(request.APP) }}

mozilla

Python Unicode hell
UnicodeDecodeError: 'ascii' codec can't
decode byte 0xd0 in position 16: ordinal
not in range(128)

mozilla

Bleach
http://pypi.python.org/pypi/

Credit: http://www.ﬂickr.com/photos/maisonbisson/3350954463/
mozilla

>>> bleach.clean('an
<script>evil()</script>
example')

'an <script>evil()</
script> example'

mozilla

>>> bleach.linkify('an http://
ex.com url')

'an <a href="http://ex.com"
rel="nofollow">http://ex.com</
a> url'

mozilla

Javascript tests

mozilla

django-qunit
http://bit.ly/django-qunit

kumar

mozilla

test('English', function() {
    z.refreshL10n('en-us');
    equals($('textarea:visible', this.sandbox).text().trim(),
           'Firebug integrates with Firefox to put ' +
'a wealth of development tools...');
});

mozilla

test('Japanese', function() {
    z.refreshL10n('ja');
    equals($('textarea:visible', this.sandbox).text().trim(),
           'Firebug Web ' +

' Firefox' +
           ' ');
});

mozilla

So we use hudson for CI,
but haven’t got the
automated tests in yet

Hoping to do this via
jstestnet

Use HudsonJenkins
http://bit.ly/jstestnet

mozilla

pep 8
py flakes
MacCabe

Flake8
http://bit.ly/ﬂake8
Credit: http://www.ﬂickr.com/photos/nebarnix/357779131/
mozilla

~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.py
apps/editors/tasks.py:1: 'datetime' imported but unused
apps/editors/tasks.py:3: 'stat' imported but unused

mozilla

Playdoh
http://bit.ly/mozilla-playdoh

fred wenzel

Credit: http://www.ﬂickr.com/photos/ahmee/97960570/
mozilla

Basis for new projects

mozilla

Celery support
Jinja2 support
Simple migrations

By default:
SHA-512 password hashing
X-Frame-Options: DENY
secure and httponly ﬂags on cookies
fred wenzel

mozilla

Take inspiration from...
but not the best for you
for example jinja2 which
makes integration with
lots of django addons
possible, but a bit harder

mozilla

Questions?
@andymckay
andym@mozilla.com
andym on irc.freenode.net, irc.mozilla.org

mozilla

Anatomy of a Large Django site

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Anatomy of a Large Django site

Semelhante a Anatomy of a Large Django site (20)

Anatomy of a Large Django site