Large Scale Processing with Django

•Transferir como ODP, PDF•

15 gostaram•3,816 visualizações

Udi Bauman

A short presentation for PyWeb-IL 8th meeting.

Tecnologia Negócios

Large-scale processing using Django Mashing clouds, queues & workflows PyWeb-IL 8 th meeting Udi h Bauman (@dibau_naum_h) Tikal Knowledge (http://tikalk.com)

Use-case 1: automated data integration service

Use-case 2: social media analysis service

Web apps vs. Back-end Services ,[object Object]

Web back-ends become thinner - just services

Applications become service providers, usually over HTTP

All reasons for using Django for almost any back-end offering services

May involve continuous communications, & not just request-response

Reliability & high-availability are usually more important with non-human users

Lots of communication with other back-ends

Addressing the needs of back-end services ,[object Object]

Continuous communication (e.g., TCP-based) is possible, can be abstracted with XMPP

Clouds & auto-scaling enable high-availability

Can use SOAP/REST for protocols against other back-ends

Even when processing took days to complete, Django was 100% robust

Mais conteúdo relacionado

Mais procurados

Spark Pitfalls meetup UnderscoreILlioron22

Apache kafka- Onkar KadamOnkar Kadam

Continuously Updating Query Results over Real-Time Linked DataRuben Taelman

Prometheus with Grafana - AddWeb SolutionAddWeb Solution Pvt. Ltd.

Message Queues a basic overviewGeshan Manandhar

How SignalR and SQL Dependency Work Process in ASP.NET with SQL Server?Raju Ahmed

Mais procurados (6)

Spark Pitfalls meetup UnderscoreIL

Apache kafka- Onkar Kadam

Continuously Updating Query Results over Real-Time Linked Data

Prometheus with Grafana - AddWeb Solution

Message Queues a basic overview

How SignalR and SQL Dependency Work Process in ASP.NET with SQL Server?

Semelhante a Large Scale Processing with Django

Adventures building 3 realtime single-page apps 6 different waysHenrik Joreteg

Event driven-archMohammed Shoaib

PeerToPeerComputing (1)MurtazaB

APIs & Underlying Protocols #APICraftSFAndrei Savu

djypllh5r1gjbaekxgwv-signature-cc6692615bbc55079760b9b0c6636bc58ec509cd0446cb...Dr. Thippeswamy S.

AWS Big Data in everyday use at YleRolf Koski

云计算及其应用lantianlcdx

BISSA: Empowering Web gadget Communication with Tuple SpacesSrinath Perera

Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...Nati Shalom

Crash Course in Open Source Cloud ComputingMark Hinkle

101 mistakes FINN.no has made with Kafka (Baksida meetup)Henning Spjelkavik

Concurrency and parallel in .netMohammad Hossein Karami

thread_ multiprocessor_ scheduling_a.pptnaghamallella

Cloud computingZeeshan Bilal

Application Modernisation through Event-Driven Microservices confluent

Scaling Streaming - Concepts, Research, Goalskamaelian

Distributed Systems in Data EngineeringAdetimehin Oluwasegun Matthew

SaaS Enablement of your existing application (Cloud Slam 2010)Nati Shalom

AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...GeeksLab Odessa

Cloud CRamin Rastin ✮✮✮✮✮

Semelhante a Large Scale Processing with Django (20)

Adventures building 3 realtime single-page apps 6 different ways

Event driven-arch

PeerToPeerComputing (1)

APIs & Underlying Protocols #APICraftSF

djypllh5r1gjbaekxgwv-signature-cc6692615bbc55079760b9b0c6636bc58ec509cd0446cb...

AWS Big Data in everyday use at Yle

云计算及其应用

BISSA: Empowering Web gadget Communication with Tuple Spaces

Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...

Crash Course in Open Source Cloud Computing

101 mistakes FINN.no has made with Kafka (Baksida meetup)

Concurrency and parallel in .net

thread_ multiprocessor_ scheduling_a.ppt

Cloud computing

Application Modernisation through Event-Driven Microservices

Scaling Streaming - Concepts, Research, Goals

Distributed Systems in Data Engineering

SaaS Enablement of your existing application (Cloud Slam 2010)

AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...

Cloud C

Mais de Udi Bauman

13Udi Bauman

Intro to-django-for-media-companiesUdi Bauman

Django course final-projectUdi Bauman

Django course summaryUdi Bauman

Nonrelational DatabasesUdi Bauman

Ship Early Ship Often With DjangoUdi Bauman

Django Article V0Udi Bauman

Python Django Intro V0.1Udi Bauman

Django And AjaxUdi Bauman

Udi Google Dev DayUdi Bauman

Intro To DjangoUdi Bauman

Mais de Udi Bauman (12)

Intro to-django-for-media-companies

Django course final-project

Django course summary

Nonrelational Databases

Ship Early Ship Often With Django

Django Article V0

Python Django Intro V0.1

Django And Ajax

Udi Google Dev Day

Intro To Django

Último

Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK

CNv6 Instructor Chapter 6 Quality of Servicegiselly40

Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersThousandEyes

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal

A Domino Admins Adventures (Engage 2024)Gabriella Davis

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Boost PC performance: How more available memory can improve productivityPrincipled Technologies

🐬 The future of MySQL is Postgres 🐘RTylerCroy

Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700

Slack Application Development 101 Slidespraypatel2

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

Large Scale Processing with Django

1. Large-scale processing using Django Mashing clouds, queues & workflows PyWeb-IL 8 th meeting Udi h Bauman (@dibau_naum_h) Tikal Knowledge (http://tikalk.com)

3. Addressing Scalability

4. Experience with Django

5. Use-case 1: automated data integration service

6. Use-case 2: social media analysis service

7. Recommendations

8. Links

10. Web back-ends become thinner - just services

11. Applications become service providers, usually over HTTP

12. All reasons for using Django for almost any back-end offering services

13.

14. Usually involve long processing

15. May involve continuous communications, & not just request-response

16. Reliability & high-availability are usually more important with non-human users

17. Lots of communication with other back-ends

18.

19. Workflow Engines manage long processing

20. Continuous communication (e.g., TCP-based) is possible, can be abstracted with XMPP

21. Clouds & auto-scaling enable high-availability

22. Can use SOAP/REST for protocols against other back-ends

23.

24. Even when processing took days to complete, Django was 100% robust

25.

26. Large data

27. Protocols against other back-ends

28.

29. Integrating data & services across several back-end systems

30.

31.

32. Used a great library called Suds

33.

34.

35.

36. Used REST interface to push & pull messages with message queues, such as ActiveMQ

37. Used Celery for AMQP-based message queues

38.

39. According to metadata, different format handlers were invoked

40.

41.

42.

43.

44. Exports logic to Java-based scripts

45.

46. Used Nicolas Toll's workflow engine, which allows users to define & manage complex workflows

47.

48. Added a graphical dashboard

49.

50.

51. Used task-names for flexible handlers invocation

52. Used periodic tasks for driving the workflow engine

53.

54. Horizontal & vertical scaling

55. Reliable & easy to manage

56. Message queues allow distributing load horizontally

57. Used script-based auto-scaling – starting new instances based on load

58.

59.

60.

61.

62.

63. Recommend people to follow based on interests

64.

65. Objects are organized in hierarchical layers

66. Objects have autonomous micro behavior aggregating to the macro behavior of the system

67. Layers are organized in spatial grids, which enable easy sharding & parallel processing

68.

69. Dbpedia

70. ConceptNet

71.

72.

73. Celery & RabbitMQ are used

74. JSON is used as message payload

75.

76. most of the data processing

77. Very intensive

78.

79. Not always-on

80. Requires 3 rd party libraries, not limited to Python

81.

82.

83. Can be started just upon need

84.

85.

86.

87. Frameworks such as Fuzed can be integrated with Python/Django

88. We're working on it as a coding session & hope to deliver a prototype soon

89.

90. Suds

91. Workflow

92. django-xmpp

93. Fuzed

94. Google Chart API

95. Talend

96. Thanks! @dibau_naum_h

Large Scale Processing with Django

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (6)

Semelhante a Large Scale Processing with Django

Semelhante a Large Scale Processing with Django (20)

Mais de Udi Bauman

Mais de Udi Bauman (12)

Último

Último (20)

Large Scale Processing with Django