SlideShare uma empresa Scribd logo
1 de 22
Baixar para ler offline
High Performance Django

David Cramer
http://www.davidcramer.net/
http://www.ibegin.com/
Curse
•  Peak daily traffic of approx. 15m pages, 150m hits.

•  Average monthly traffic 120m pages, 6m uniques.

•  Python, MySQL, Squid, memcached, mod_python, lighty.

•  Most developers came strictly from PHP (myself included).

•  12 web servers, 4 database servers, 2 squid caches.
iBegin
•  Massive amounts of data, 100m+ rows.

•  Python, PHP, MySQL, mod_wsgi.

•  Small team of developers.

•  Complex database partitioning/synchronization tasks.

•  Attempting to not branch off of Django. 
Areas of Concern
•  Database (ORM)

•  Webserver (Resources, Handling Millions of Reqs)

•  Caching (Invalidation, Cache Dump)

•  Template Rendering (Logic Separation)

•  Profiling
Tools of the Trade
•  Webserver (Apache, Nginx, Lighttpd)

•  Object Cache (memcached)

•  Database (MySQL, PostgreSQL, …)

•  Page Cache (Squid, Nginx, Varnish)

•  Load Balancing (Nginx, Perlbal)
How We Did It

•  “Primary” web servers serving Django using mod_python.

•  Media servers using Django on lighttpd.

•  Static served using additional instances of lighttpd.

•  Load balancers passing requests to multiple Squids.

•  Squids passing requests to multiple web servers.
Lessons Learned
•  Don’t be afraid to experiment. You’re not limited to a one.

•  mod_wsgi is a huge step forward from mod_python.

•  Serving static files using different software can help.

•  Send proper HTTP headers where they are needed.

•  Use services like S3, Akamai, Limelight, etc..
Webserver Software
Python Scripts             Static Content
•  Apache (wsgi, mod_py,   •  Apache
   fastcgi)                •  Lighttpd
•  Lighttpd (fastcgi)      •  Tinyhttpd
•  Nginx (fastcgi)         •  Nginx
Reverse Proxies            Software Load Balancers
•  Nginx                   •  Nginx
•  Squid                   •  Perlbal
•  Varnish
Database (ORM)
•  Won’t make your queries efficient. Make your own indexes.

•  select_related() can be good, as well as bad.

•  Inherited ordering (Meta: ordering) will get you.

•  Hundreds of queries on a page is never a good thing.

•  Know when to not use the ORM.
Handling JOINs
class Category(models.Model):
     name       = models.CharField()
     created_by = models.ForeignKey(User)

class Poll(models.Model):
     name         = models.CharField()
     category     = models.ForeignKey(Category)
     created_by = models.ForeignKey(User)

# We need to output a page listing all Poll's with
# their name and category's name.

def a_bad_example(request):
     # We have just caused Poll to JOIN with User and Category,
     # which will also JOIN with User a second time.
     my_polls = Poll.objects.all().select_related()
     return render_to_response('polls.html', locals(), request)

def a_good_example(request):
     # Use select_related explicitly in each case.
     poll = Poll.objects.all().select_related('category')
     return render_to_response('polls.html', locals(), request)
Template Rendering
•  Sandboxed engines are typically slower by nature.

•  Keep logic in views and template tags.

•  Be aware of performance in loops, and groupby (regroup).

•  Loaded templates can be cached to avoid disk reads.

•  Switching template engines is easy, but may not give you

  any worthwhile performance gain.
Template Engines
Caching
•  Two flavors of caching: object cache and browser cache.

•  Django provides built-in support for both.

•  Invalidation is a headache without a well thought out plan.

•  Caching isn’t a solution for slow loading pages or improper indexes.

•  Use a reverse proxy in between the browser and your web servers:

   Squid, Varnish, Nginx, etc..
Cache With a Plan
•  Build your pages to use proper cache headers.

•  Create a plan for object cache expiration, and invalidation.

•  For typical web apps you can serve the same cached page

  for both anonymous and authenticated users.

•  Contain commonly used querysets in managers for

  transparent caching and invalidation.
Cache Commonly Used Items
def my_context_processor(request):
    # We access object_list every time we use our context processors so
    # it makes sense to cache this, no?
    cache_key = ‘mymodel:all’
    object_list = cache.get(cache_key)
    if object_list is None:
         object_list = MyModel.objects.all()
         cache.set(cache_key, object_list)
    return {‘object_list’: object_list}

# Now that we are caching the object list we are going to want to invalidate it
class MyModel(models.Model):
    name = models.CharField()

    def save(self, *args, **kwargs):
        super(MyModel, self).save(*args, **kwargs)
        # save it before you update the cache
        cache.set(‘mymodel:all’, MyModel.objects.all())
Profiling Code
•  Finding the bottleneck can be time consuming.

•  Tools exist to help identify common problematic areas.

   –  cProfile/Profile Python modules.

   –  PDB (Python Debugger)
Profiling Code With cProfile
import sys
try: import cProfile as profile
except ImportError: import profile
try: from cStringIO import StringIO
except ImportError: import StringIO
from django.conf import settings

class ProfilerMiddleware(object):
   def can(self, request):
     return settings.DEBUG and 'prof' in request.GET and (not settings.INTERNAL_IPS or request.META['REMOTE_ADDR'] in
       settings.INTERNAL_IPS)
   def process_view(self, request, callback, callback_args, callback_kwargs):
     if self.can(request):
         self.profiler = profile.Profile()
       args = (request,) + callback_args
       return self.profiler.runcall(callback, *args, **callback_kwargs)
  def process_response(self, request, response):
    if self.can(request):
        self.profiler.create_stats()
       out = StringIO()
       old_stdout, sys.stdout = sys.stdout, out
       self.profiler.print_stats(1)
       sys.stdout = old_stdout
       response.content = '<pre>%s</pre>' % out.getvalue()
     return response
http://localhost:8000/?prof
Profiling Database Queries
from django.db import connection
class DatabaseProfilerMiddleware(object):
  def can(self, request):
    return settings.DEBUG and 'dbprof' in request.GET 
       and (not settings.INTERNAL_IPS or 
       request.META['REMOTE_ADDR'] in settings.INTERNAL_IPS)

  def process_response(self, request, response):
    if self.can(request):
        out = StringIO()
        out.write('timetsqln')
        total_time = 0
        for query in reversed(sorted(connection.queries, key=lambda x: x['time'])):
         total_time += float(query['time'])*1000
         out.write('%st%sn' % (query['time'], query['sql']))

       response.content = '<pre style=quot;white-space:pre-wrapquot;>%d queries executed in %.3f secondsnn%s</pre>' %
    (len(connection.queries), total_time/1000, out.getvalue())
    return response
http://localhost:8000/?dbprof
Summary
•  Database efficiency is the typical problem in web apps.

•  Develop and deploy a caching plan early on.

•  Use profiling tools to find your problematic areas. Don’t pre-

  optimize unless there is good reason.

•  Find someone who knows more than me to configure your

  server software. 
Thanks!

Slides and code available online at:
http://www.davidcramer.net/djangocon

Mais conteúdo relacionado

Mais procurados

Beyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the codeBeyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the codeWim Godden
 
Security: Odoo Code Hardening
Security: Odoo Code HardeningSecurity: Odoo Code Hardening
Security: Odoo Code HardeningOdoo
 
Rails-like JavaScript Using CoffeeScript, Backbone.js and Jasmine
Rails-like JavaScript Using CoffeeScript, Backbone.js and JasmineRails-like JavaScript Using CoffeeScript, Backbone.js and Jasmine
Rails-like JavaScript Using CoffeeScript, Backbone.js and JasmineRaimonds Simanovskis
 
JavaScript and the AST
JavaScript and the ASTJavaScript and the AST
JavaScript and the ASTJarrod Overson
 
Cache Money Talk: Practical Application
Cache Money Talk: Practical ApplicationCache Money Talk: Practical Application
Cache Money Talk: Practical ApplicationWolfram Arnold
 
Deploying
DeployingDeploying
Deployingsoon
 
AngularJS Services
AngularJS ServicesAngularJS Services
AngularJS ServicesEyal Vardi
 
Angular.js Fundamentals
Angular.js FundamentalsAngular.js Fundamentals
Angular.js FundamentalsMark
 
AngularJS Compile Process
AngularJS Compile ProcessAngularJS Compile Process
AngularJS Compile ProcessEyal Vardi
 
Meet Magento Sweden - Magento 2 Layout and Code Compilation for Performance
Meet Magento Sweden - Magento 2 Layout and Code Compilation for PerformanceMeet Magento Sweden - Magento 2 Layout and Code Compilation for Performance
Meet Magento Sweden - Magento 2 Layout and Code Compilation for PerformanceIvan Chepurnyi
 
Node.js in action
Node.js in actionNode.js in action
Node.js in actionSimon Su
 
Hidden Treasures in Project Wonder
Hidden Treasures in Project WonderHidden Treasures in Project Wonder
Hidden Treasures in Project WonderWO Community
 
RubyEnRails2007 - Dr Nic Williams - Keynote
RubyEnRails2007 - Dr Nic Williams - KeynoteRubyEnRails2007 - Dr Nic Williams - Keynote
RubyEnRails2007 - Dr Nic Williams - KeynoteDr Nic Williams
 
Class-based views with Django
Class-based views with DjangoClass-based views with Django
Class-based views with DjangoSimon Willison
 
Workshop 1: Good practices in JavaScript
Workshop 1: Good practices in JavaScriptWorkshop 1: Good practices in JavaScript
Workshop 1: Good practices in JavaScriptVisual Engineering
 
HTML5: friend or foe (to Flash)?
HTML5: friend or foe (to Flash)?HTML5: friend or foe (to Flash)?
HTML5: friend or foe (to Flash)?Remy Sharp
 

Mais procurados (20)

Rails on Oracle 2011
Rails on Oracle 2011Rails on Oracle 2011
Rails on Oracle 2011
 
Beyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the codeBeyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the code
 
Django Heresies
Django HeresiesDjango Heresies
Django Heresies
 
Security: Odoo Code Hardening
Security: Odoo Code HardeningSecurity: Odoo Code Hardening
Security: Odoo Code Hardening
 
Rails-like JavaScript Using CoffeeScript, Backbone.js and Jasmine
Rails-like JavaScript Using CoffeeScript, Backbone.js and JasmineRails-like JavaScript Using CoffeeScript, Backbone.js and Jasmine
Rails-like JavaScript Using CoffeeScript, Backbone.js and Jasmine
 
JavaScript and the AST
JavaScript and the ASTJavaScript and the AST
JavaScript and the AST
 
ES2015 workflows
ES2015 workflowsES2015 workflows
ES2015 workflows
 
Cache Money Talk: Practical Application
Cache Money Talk: Practical ApplicationCache Money Talk: Practical Application
Cache Money Talk: Practical Application
 
Deploying
DeployingDeploying
Deploying
 
Scala active record
Scala active recordScala active record
Scala active record
 
AngularJS Services
AngularJS ServicesAngularJS Services
AngularJS Services
 
Angular.js Fundamentals
Angular.js FundamentalsAngular.js Fundamentals
Angular.js Fundamentals
 
AngularJS Compile Process
AngularJS Compile ProcessAngularJS Compile Process
AngularJS Compile Process
 
Meet Magento Sweden - Magento 2 Layout and Code Compilation for Performance
Meet Magento Sweden - Magento 2 Layout and Code Compilation for PerformanceMeet Magento Sweden - Magento 2 Layout and Code Compilation for Performance
Meet Magento Sweden - Magento 2 Layout and Code Compilation for Performance
 
Node.js in action
Node.js in actionNode.js in action
Node.js in action
 
Hidden Treasures in Project Wonder
Hidden Treasures in Project WonderHidden Treasures in Project Wonder
Hidden Treasures in Project Wonder
 
RubyEnRails2007 - Dr Nic Williams - Keynote
RubyEnRails2007 - Dr Nic Williams - KeynoteRubyEnRails2007 - Dr Nic Williams - Keynote
RubyEnRails2007 - Dr Nic Williams - Keynote
 
Class-based views with Django
Class-based views with DjangoClass-based views with Django
Class-based views with Django
 
Workshop 1: Good practices in JavaScript
Workshop 1: Good practices in JavaScriptWorkshop 1: Good practices in JavaScript
Workshop 1: Good practices in JavaScript
 
HTML5: friend or foe (to Flash)?
HTML5: friend or foe (to Flash)?HTML5: friend or foe (to Flash)?
HTML5: friend or foe (to Flash)?
 

Destaque

Activity streams Lightning Talk, DjangoCon 2011, Day3
Activity streams Lightning Talk, DjangoCon 2011, Day3Activity streams Lightning Talk, DjangoCon 2011, Day3
Activity streams Lightning Talk, DjangoCon 2011, Day3Steve Ivy
 
Django, Framework Python para desenvolvimento web
Django, Framework Python para desenvolvimento webDjango, Framework Python para desenvolvimento web
Django, Framework Python para desenvolvimento webMayron Cachina
 
Useful Django 1.4
Useful Django 1.4Useful Django 1.4
Useful Django 1.4hirokiky
 
My pyhack 1301
My pyhack 1301My pyhack 1301
My pyhack 1301hirokiky
 
django-meio-easytags Lightining Talk @ DjangoCon US 2011
django-meio-easytags Lightining Talk @ DjangoCon US 2011django-meio-easytags Lightining Talk @ DjangoCon US 2011
django-meio-easytags Lightining Talk @ DjangoCon US 2011Vinicius Mendes
 
Introducing Django
Introducing DjangoIntroducing Django
Introducing Djangozerok
 

Destaque (8)

Activity streams Lightning Talk, DjangoCon 2011, Day3
Activity streams Lightning Talk, DjangoCon 2011, Day3Activity streams Lightning Talk, DjangoCon 2011, Day3
Activity streams Lightning Talk, DjangoCon 2011, Day3
 
Django, Framework Python para desenvolvimento web
Django, Framework Python para desenvolvimento webDjango, Framework Python para desenvolvimento web
Django, Framework Python para desenvolvimento web
 
Useful Django 1.4
Useful Django 1.4Useful Django 1.4
Useful Django 1.4
 
My pyhack 1301
My pyhack 1301My pyhack 1301
My pyhack 1301
 
Welcome to the Django
Welcome to the DjangoWelcome to the Django
Welcome to the Django
 
django-meio-easytags Lightining Talk @ DjangoCon US 2011
django-meio-easytags Lightining Talk @ DjangoCon US 2011django-meio-easytags Lightining Talk @ DjangoCon US 2011
django-meio-easytags Lightining Talk @ DjangoCon US 2011
 
Introducing Django
Introducing DjangoIntroducing Django
Introducing Django
 
Django Intro
Django IntroDjango Intro
Django Intro
 

Semelhante a High Performance Django 1

PyGrunn 2017 - Django Performance Unchained - slides
PyGrunn 2017 - Django Performance Unchained - slidesPyGrunn 2017 - Django Performance Unchained - slides
PyGrunn 2017 - Django Performance Unchained - slidesArtur Barseghyan
 
SproutCore and the Future of Web Apps
SproutCore and the Future of Web AppsSproutCore and the Future of Web Apps
SproutCore and the Future of Web AppsMike Subelsky
 
[Coscup 2012] JavascriptMVC
[Coscup 2012] JavascriptMVC[Coscup 2012] JavascriptMVC
[Coscup 2012] JavascriptMVCAlive Kuo
 
D Trace Support In My Sql Guide To Solving Reallife Performance Problems
D Trace Support In My Sql Guide To Solving Reallife Performance ProblemsD Trace Support In My Sql Guide To Solving Reallife Performance Problems
D Trace Support In My Sql Guide To Solving Reallife Performance ProblemsMySQLConference
 
mongodb-introduction
mongodb-introductionmongodb-introduction
mongodb-introductionTse-Ching Ho
 
Automated Frontend Testing
Automated Frontend TestingAutomated Frontend Testing
Automated Frontend TestingNeil Crosby
 
Future of Web Apps: Google Gears
Future of Web Apps: Google GearsFuture of Web Apps: Google Gears
Future of Web Apps: Google Gearsdion
 
Nanoformats
NanoformatsNanoformats
Nanoformatsrozario
 
All I Need to Know I Learned by Writing My Own Web Framework
All I Need to Know I Learned by Writing My Own Web FrameworkAll I Need to Know I Learned by Writing My Own Web Framework
All I Need to Know I Learned by Writing My Own Web FrameworkBen Scofield
 
Good practices for PrestaShop code security and optimization
Good practices for PrestaShop code security and optimizationGood practices for PrestaShop code security and optimization
Good practices for PrestaShop code security and optimizationPrestaShop
 
Php classes in mumbai
Php classes in mumbaiPhp classes in mumbai
Php classes in mumbaiaadi Surve
 
Sinatra and JSONQuery Web Service
Sinatra and JSONQuery Web ServiceSinatra and JSONQuery Web Service
Sinatra and JSONQuery Web Servicevvatikiotis
 
Django Multi-DB in Anger
Django Multi-DB in AngerDjango Multi-DB in Anger
Django Multi-DB in AngerLoren Davie
 
6 tips for improving ruby performance
6 tips for improving ruby performance6 tips for improving ruby performance
6 tips for improving ruby performanceEngine Yard
 
Behind the curtain - How Django handles a request
Behind the curtain - How Django handles a requestBehind the curtain - How Django handles a request
Behind the curtain - How Django handles a requestDaniel Hepper
 

Semelhante a High Performance Django 1 (20)

PyGrunn 2017 - Django Performance Unchained - slides
PyGrunn 2017 - Django Performance Unchained - slidesPyGrunn 2017 - Django Performance Unchained - slides
PyGrunn 2017 - Django Performance Unchained - slides
 
SproutCore and the Future of Web Apps
SproutCore and the Future of Web AppsSproutCore and the Future of Web Apps
SproutCore and the Future of Web Apps
 
[Coscup 2012] JavascriptMVC
[Coscup 2012] JavascriptMVC[Coscup 2012] JavascriptMVC
[Coscup 2012] JavascriptMVC
 
D Trace Support In My Sql Guide To Solving Reallife Performance Problems
D Trace Support In My Sql Guide To Solving Reallife Performance ProblemsD Trace Support In My Sql Guide To Solving Reallife Performance Problems
D Trace Support In My Sql Guide To Solving Reallife Performance Problems
 
mongodb-introduction
mongodb-introductionmongodb-introduction
mongodb-introduction
 
Automated Frontend Testing
Automated Frontend TestingAutomated Frontend Testing
Automated Frontend Testing
 
Openstack 簡介
Openstack 簡介Openstack 簡介
Openstack 簡介
 
Future of Web Apps: Google Gears
Future of Web Apps: Google GearsFuture of Web Apps: Google Gears
Future of Web Apps: Google Gears
 
Nanoformats
NanoformatsNanoformats
Nanoformats
 
All I Need to Know I Learned by Writing My Own Web Framework
All I Need to Know I Learned by Writing My Own Web FrameworkAll I Need to Know I Learned by Writing My Own Web Framework
All I Need to Know I Learned by Writing My Own Web Framework
 
Django tricks (2)
Django tricks (2)Django tricks (2)
Django tricks (2)
 
What's new in Django 1.2?
What's new in Django 1.2?What's new in Django 1.2?
What's new in Django 1.2?
 
Django Pro ORM
Django Pro ORMDjango Pro ORM
Django Pro ORM
 
Good practices for PrestaShop code security and optimization
Good practices for PrestaShop code security and optimizationGood practices for PrestaShop code security and optimization
Good practices for PrestaShop code security and optimization
 
Php classes in mumbai
Php classes in mumbaiPhp classes in mumbai
Php classes in mumbai
 
Sinatra and JSONQuery Web Service
Sinatra and JSONQuery Web ServiceSinatra and JSONQuery Web Service
Sinatra and JSONQuery Web Service
 
Django Multi-DB in Anger
Django Multi-DB in AngerDjango Multi-DB in Anger
Django Multi-DB in Anger
 
Django Show
Django ShowDjango Show
Django Show
 
6 tips for improving ruby performance
6 tips for improving ruby performance6 tips for improving ruby performance
6 tips for improving ruby performance
 
Behind the curtain - How Django handles a request
Behind the curtain - How Django handles a requestBehind the curtain - How Django handles a request
Behind the curtain - How Django handles a request
 

Mais de DjangoCon2008

Mais de DjangoCon2008 (6)

Why I Hate Django
Why I Hate DjangoWhy I Hate Django
Why I Hate Django
 
Pinax
PinaxPinax
Pinax
 
Satchmo
SatchmoSatchmo
Satchmo
 
Reusable Apps
Reusable AppsReusable Apps
Reusable Apps
 
What’S New In Newforms Admin
What’S New In Newforms AdminWhat’S New In Newforms Admin
What’S New In Newforms Admin
 
High Performance Django
High Performance DjangoHigh Performance Django
High Performance Django
 

Último

Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdfJamie (Taka) Wang
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 

Último (20)

Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 

High Performance Django 1

  • 1. High Performance Django David Cramer http://www.davidcramer.net/ http://www.ibegin.com/
  • 2. Curse •  Peak daily traffic of approx. 15m pages, 150m hits. •  Average monthly traffic 120m pages, 6m uniques. •  Python, MySQL, Squid, memcached, mod_python, lighty. •  Most developers came strictly from PHP (myself included). •  12 web servers, 4 database servers, 2 squid caches.
  • 3. iBegin •  Massive amounts of data, 100m+ rows. •  Python, PHP, MySQL, mod_wsgi. •  Small team of developers. •  Complex database partitioning/synchronization tasks. •  Attempting to not branch off of Django. 
  • 4. Areas of Concern •  Database (ORM) •  Webserver (Resources, Handling Millions of Reqs) •  Caching (Invalidation, Cache Dump) •  Template Rendering (Logic Separation) •  Profiling
  • 5. Tools of the Trade •  Webserver (Apache, Nginx, Lighttpd) •  Object Cache (memcached) •  Database (MySQL, PostgreSQL, …) •  Page Cache (Squid, Nginx, Varnish) •  Load Balancing (Nginx, Perlbal)
  • 6. How We Did It •  “Primary” web servers serving Django using mod_python. •  Media servers using Django on lighttpd. •  Static served using additional instances of lighttpd. •  Load balancers passing requests to multiple Squids. •  Squids passing requests to multiple web servers.
  • 7. Lessons Learned •  Don’t be afraid to experiment. You’re not limited to a one. •  mod_wsgi is a huge step forward from mod_python. •  Serving static files using different software can help. •  Send proper HTTP headers where they are needed. •  Use services like S3, Akamai, Limelight, etc..
  • 8. Webserver Software Python Scripts Static Content •  Apache (wsgi, mod_py, •  Apache fastcgi) •  Lighttpd •  Lighttpd (fastcgi) •  Tinyhttpd •  Nginx (fastcgi) •  Nginx Reverse Proxies Software Load Balancers •  Nginx •  Nginx •  Squid •  Perlbal •  Varnish
  • 9. Database (ORM) •  Won’t make your queries efficient. Make your own indexes. •  select_related() can be good, as well as bad. •  Inherited ordering (Meta: ordering) will get you. •  Hundreds of queries on a page is never a good thing. •  Know when to not use the ORM.
  • 10. Handling JOINs class Category(models.Model): name = models.CharField() created_by = models.ForeignKey(User) class Poll(models.Model): name = models.CharField() category = models.ForeignKey(Category) created_by = models.ForeignKey(User) # We need to output a page listing all Poll's with # their name and category's name. def a_bad_example(request): # We have just caused Poll to JOIN with User and Category, # which will also JOIN with User a second time. my_polls = Poll.objects.all().select_related() return render_to_response('polls.html', locals(), request) def a_good_example(request): # Use select_related explicitly in each case. poll = Poll.objects.all().select_related('category') return render_to_response('polls.html', locals(), request)
  • 11. Template Rendering •  Sandboxed engines are typically slower by nature. •  Keep logic in views and template tags. •  Be aware of performance in loops, and groupby (regroup). •  Loaded templates can be cached to avoid disk reads. •  Switching template engines is easy, but may not give you any worthwhile performance gain.
  • 13. Caching •  Two flavors of caching: object cache and browser cache. •  Django provides built-in support for both. •  Invalidation is a headache without a well thought out plan. •  Caching isn’t a solution for slow loading pages or improper indexes. •  Use a reverse proxy in between the browser and your web servers: Squid, Varnish, Nginx, etc..
  • 14. Cache With a Plan •  Build your pages to use proper cache headers. •  Create a plan for object cache expiration, and invalidation. •  For typical web apps you can serve the same cached page for both anonymous and authenticated users. •  Contain commonly used querysets in managers for transparent caching and invalidation.
  • 15. Cache Commonly Used Items def my_context_processor(request): # We access object_list every time we use our context processors so # it makes sense to cache this, no? cache_key = ‘mymodel:all’ object_list = cache.get(cache_key) if object_list is None: object_list = MyModel.objects.all() cache.set(cache_key, object_list) return {‘object_list’: object_list} # Now that we are caching the object list we are going to want to invalidate it class MyModel(models.Model): name = models.CharField() def save(self, *args, **kwargs): super(MyModel, self).save(*args, **kwargs) # save it before you update the cache cache.set(‘mymodel:all’, MyModel.objects.all())
  • 16. Profiling Code •  Finding the bottleneck can be time consuming. •  Tools exist to help identify common problematic areas. –  cProfile/Profile Python modules. –  PDB (Python Debugger)
  • 17. Profiling Code With cProfile import sys try: import cProfile as profile except ImportError: import profile try: from cStringIO import StringIO except ImportError: import StringIO from django.conf import settings class ProfilerMiddleware(object): def can(self, request): return settings.DEBUG and 'prof' in request.GET and (not settings.INTERNAL_IPS or request.META['REMOTE_ADDR'] in settings.INTERNAL_IPS) def process_view(self, request, callback, callback_args, callback_kwargs): if self.can(request): self.profiler = profile.Profile() args = (request,) + callback_args return self.profiler.runcall(callback, *args, **callback_kwargs) def process_response(self, request, response): if self.can(request): self.profiler.create_stats() out = StringIO() old_stdout, sys.stdout = sys.stdout, out self.profiler.print_stats(1) sys.stdout = old_stdout response.content = '<pre>%s</pre>' % out.getvalue() return response
  • 19. Profiling Database Queries from django.db import connection class DatabaseProfilerMiddleware(object): def can(self, request): return settings.DEBUG and 'dbprof' in request.GET and (not settings.INTERNAL_IPS or request.META['REMOTE_ADDR'] in settings.INTERNAL_IPS) def process_response(self, request, response): if self.can(request): out = StringIO() out.write('timetsqln') total_time = 0 for query in reversed(sorted(connection.queries, key=lambda x: x['time'])): total_time += float(query['time'])*1000 out.write('%st%sn' % (query['time'], query['sql'])) response.content = '<pre style=quot;white-space:pre-wrapquot;>%d queries executed in %.3f secondsnn%s</pre>' % (len(connection.queries), total_time/1000, out.getvalue()) return response
  • 21. Summary •  Database efficiency is the typical problem in web apps. •  Develop and deploy a caching plan early on. •  Use profiling tools to find your problematic areas. Don’t pre- optimize unless there is good reason. •  Find someone who knows more than me to configure your server software. 
  • 22. Thanks! Slides and code available online at: http://www.davidcramer.net/djangocon