O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Do more than one thing at the same time, the Python way

6.961 visualizações

Publicada em

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Do more than one thing at the same time, the Python way

  1. 1. DO MORE THAN ONE THING AT THE TIME the Python way! Jaime Buelta
  2. 2. WHY?
  3. 3. SLICE A PROBLEM TOSOLVE IT USING MORE RESOURCES
  4. 4. SAME THING FOR DIFFERENT ACTORS
  5. 5. DIFFERENT THINGS FOR THE SAME ACTOR
  6. 6. DOING MORE THAN ONE THING IS TOUGH
  7. 7. CHOOSE WISELY
  8. 8. THREADS
  9. 9. THREADS IN PYTHON module threading module thread
  10. 10. THREAD EXAMPLEimport threadingITERATIONS = 1000000class Example1(threading.Thread): def __init__(self, num): self.num = num super(Example1, self).__init__() def run(self): for i in xrange(ITERATIONS): passdef main(): for j in xrange(10): t = Example1(j) t.start()if __name__ == __main__: main()
  11. 11. TIMERSfrom threading import TimerDELAYED_TIME = 10.5def delayed(): print This call is delayedt = Timer(10.5, delayed)t.start()t.cancel() # Cancels the execution
  12. 12. PROCESSES
  13. 13. YE OLDE FORK
  14. 14. MULTIPROCESS MODULEimport multiprocessingITERATIONS = 1000000class Example1(multiprocessing.Process): def __init__(self, num): self.num = num super(Example1, self).__init__() def run(self): for i in xrange(ITERATIONS): passdef main(): for j in xrange(10): t = Example1(j) t.start()if __name__ == __main__: main()
  15. 15. OS ARE GREAT AT MULTITASKING
  16. 16. PROCESS COMMUNICATION NEEDS TO BE STRUCTURED but that is not necessarily a bad thing
  17. 17. ASYNCHRONOUS PROGRAMMING
  18. 18. Thread 1 Thread 2
  19. 19. Task 1 Task 2 Task 3 The task will releasewaiting control once they areready! blocked waiting for an done! input from IO waiting ready! Callbackdone! done!
  20. 20. death by callback
  21. 21. EVENTLETNUM_URLS = 1000URL = http://www.some_address.com/urls = [URL] * NUM_URLSimport eventletfrom eventlet.green import urllib2def fetch(url): return urllib2.urlopen(url).read()pool = eventlet.GreenPool()for body in pool.imap(fetch, urls): do_something_with_result(body)
  22. 22. Asynchronousprograming is greatwhen the tasks are IO - Bound So the CPU is basically waiting...
  23. 23. Asynchronous programing is notgood when tasks are CPU - BoundIf one tasks enters on an infinite loop, the whole system is blocked
  24. 24. YESTERDAY THERE WAS ATALK ABOUT ASYNC PYTHON PROGRAMMING Hope you attended, I did. If you don’t, you can watch it online later
  25. 25. THE INFAMOUS GIL
  26. 26. It doesn’t allow to run two threads at the same time, even if theOS will do it.Only one thread run. The rest will be blocked.
  27. 27. 2 core machineThread AThread B
  28. 28. 100,000,000 iterations40 s30 s20 s10 s 0s 1 10 100 1000 10000 4 core machine
  29. 29. IS IT REALLYA PROBLEM?
  30. 30. I WANT TO WATCH THIS YOUTUBE BUT I’M ALREADY LISTENING TO MUSIC AT THE SAME TIMEEEH, NOT AS BIG AS IT LOOKS
  31. 31. GIL MAKES CONCURRENTPROGRAMMING MUCH EASIER And the problems are quite limited in practice
  32. 32. BUT MAYBE YOUR PROGRAM IS ONE OF THE FEW Avoid problems not using threads, but processes
  33. 33. threading multiprocess sequential100,000ms 10,000ms 1,000ms 100ms 10ms 1ms 1000 10000 100000 1000000 10000000
  34. 34. Great, detailed talk “Understanding GIL” by David Beazlyhttp://www.dabeaz.com/python/UnderstandingGIL.pdf
  35. 35. AVOID PROBLEMS
  36. 36. SIMPLE ARCHITECTURE
  37. 37. ALL PYTHON OPERATIONS ARE ATOMIC Hey, that’s what the GIL is for
  38. 38. LOCKING TOBE USED WITH EXTREME CAUTION If you need to set exclusive sections, you are probably doing it wrong
  39. 39. BUT WHEN I DO, I USE WITHfrom threading import Lockmy_lock = Lock()def some_function(args): with my_lock: protected_section()
  40. 40. USE QUEUES (AND PIPES)
  41. 41. PROCESSTHE TASK WITHWORKERS
  42. 42. LIMIT THE NUMBERS!!!
  43. 43. THREAD COORDINATION IS HELL
  44. 44. Main periodic thread task A task B task C task DMain periodic threadMain periodic thread
  45. 45. but that’s probably not the best use of Python
  46. 46. QUESTIONS? THANKS FOR YOUR ATTENTION @jaimebuelta wrongsideofmemphis.wordpress.com

×