Large Scale Processing with Django

•Transferir como ODP, PDF•

15 gostaram•3,816 visualizações

Udi Bauman

A short presentation for PyWeb-IL 8th meeting.

Tecnologia Negócios

Large-scale processing using Django Mashing clouds, queues & workflows PyWeb-IL 8 th meeting Udi h Bauman (@dibau_naum_h) Tikal Knowledge (http://tikalk.com)

Use-case 1: automated data integration service

Use-case 2: social media analysis service

Web apps vs. Back-end Services ,[object Object]

Web back-ends become thinner - just services

Applications become service providers, usually over HTTP

All reasons for using Django for almost any back-end offering services

May involve continuous communications, & not just request-response

Reliability & high-availability are usually more important with non-human users

Lots of communication with other back-ends

Addressing the needs of back-end services ,[object Object]

Continuous communication (e.g., TCP-based) is possible, can be abstracted with XMPP

Clouds & auto-scaling enable high-availability

Can use SOAP/REST for protocols against other back-ends

Even when processing took days to complete, Django was 100% robust

Mais conteúdo relacionado

Mais procurados

Spark Pitfalls meetup UnderscoreILlioron22

Apache kafka- Onkar KadamOnkar Kadam

Continuously Updating Query Results over Real-Time Linked DataRuben Taelman

Prometheus with Grafana - AddWeb SolutionAddWeb Solution Pvt. Ltd.

Message Queues a basic overviewGeshan Manandhar

How SignalR and SQL Dependency Work Process in ASP.NET with SQL Server?Raju Ahmed

Mais procurados (6)

Spark Pitfalls meetup UnderscoreIL

Apache kafka- Onkar Kadam

Continuously Updating Query Results over Real-Time Linked Data

Prometheus with Grafana - AddWeb Solution

Message Queues a basic overview

How SignalR and SQL Dependency Work Process in ASP.NET with SQL Server?

Semelhante a Large Scale Processing with Django

Adventures building 3 realtime single-page apps 6 different waysHenrik Joreteg

Event driven-archMohammed Shoaib

PeerToPeerComputing (1)MurtazaB

APIs & Underlying Protocols #APICraftSFAndrei Savu

djypllh5r1gjbaekxgwv-signature-cc6692615bbc55079760b9b0c6636bc58ec509cd0446cb...Dr. Thippeswamy S.

AWS Big Data in everyday use at YleRolf Koski

云计算及其应用lantianlcdx

BISSA: Empowering Web gadget Communication with Tuple SpacesSrinath Perera

Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...Nati Shalom

Crash Course in Open Source Cloud ComputingMark Hinkle

101 mistakes FINN.no has made with Kafka (Baksida meetup)Henning Spjelkavik

Concurrency and parallel in .netMohammad Hossein Karami

thread_ multiprocessor_ scheduling_a.pptnaghamallella

Cloud computingZeeshan Bilal

Application Modernisation through Event-Driven Microservices confluent

Scaling Streaming - Concepts, Research, Goalskamaelian

Distributed Systems in Data EngineeringAdetimehin Oluwasegun Matthew

SaaS Enablement of your existing application (Cloud Slam 2010)Nati Shalom

AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...GeeksLab Odessa

Cloud CRamin Rastin ✮✮✮✮✮

Semelhante a Large Scale Processing with Django (20)

Adventures building 3 realtime single-page apps 6 different ways

Event driven-arch

PeerToPeerComputing (1)

APIs & Underlying Protocols #APICraftSF

djypllh5r1gjbaekxgwv-signature-cc6692615bbc55079760b9b0c6636bc58ec509cd0446cb...

AWS Big Data in everyday use at Yle

云计算及其应用

BISSA: Empowering Web gadget Communication with Tuple Spaces

Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...

Crash Course in Open Source Cloud Computing

101 mistakes FINN.no has made with Kafka (Baksida meetup)

Concurrency and parallel in .net

thread_ multiprocessor_ scheduling_a.ppt

Cloud computing

Application Modernisation through Event-Driven Microservices

Scaling Streaming - Concepts, Research, Goals

Distributed Systems in Data Engineering

SaaS Enablement of your existing application (Cloud Slam 2010)

AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...

Cloud C

Mais de Udi Bauman

13Udi Bauman

Intro to-django-for-media-companiesUdi Bauman

Django course final-projectUdi Bauman

Django course summaryUdi Bauman

Nonrelational DatabasesUdi Bauman

Ship Early Ship Often With DjangoUdi Bauman

Django Article V0Udi Bauman

Python Django Intro V0.1Udi Bauman

Django And AjaxUdi Bauman

Udi Google Dev DayUdi Bauman

Intro To DjangoUdi Bauman

Mais de Udi Bauman (12)

Intro to-django-for-media-companies

Django course final-project

Django course summary

Nonrelational Databases

Ship Early Ship Often With Django

Django Article V0

Python Django Intro V0.1

Django And Ajax

Udi Google Dev Day

Intro To Django

Último

From Family Reminiscence to Scholarly Archive .Alan Dix

Artificial intelligence in cctv survelliance.pptxhariprasad279825

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada

SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero

How to write a Business Continuity PlanDatabarracks

Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxnull - The Open Security Community

New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada

Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays

Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed

Take control of your SAP testing with UiPath Test SuiteDianaGray10

What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett

Anypoint Exchange: It’s Not Just a Repo!Manik S Magar

Story boards and shot lists for my a level piececharlottematthew16

Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro

DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy

Powerpoint exploring the locations used in television show Time Clashcharlottematthew16

Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University

Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge

Large Scale Processing with Django

1. Large-scale processing using Django Mashing clouds, queues & workflows PyWeb-IL 8 th meeting Udi h Bauman (@dibau_naum_h) Tikal Knowledge (http://tikalk.com)

3. Addressing Scalability

4. Experience with Django

5. Use-case 1: automated data integration service

6. Use-case 2: social media analysis service

7. Recommendations

8. Links

10. Web back-ends become thinner - just services

11. Applications become service providers, usually over HTTP

12. All reasons for using Django for almost any back-end offering services

13.

14. Usually involve long processing

15. May involve continuous communications, & not just request-response

16. Reliability & high-availability are usually more important with non-human users

17. Lots of communication with other back-ends

18.

19. Workflow Engines manage long processing

20. Continuous communication (e.g., TCP-based) is possible, can be abstracted with XMPP

21. Clouds & auto-scaling enable high-availability

22. Can use SOAP/REST for protocols against other back-ends

23.

24. Even when processing took days to complete, Django was 100% robust

25.

26. Large data

27. Protocols against other back-ends

28.

29. Integrating data & services across several back-end systems

30.

31.

32. Used a great library called Suds

33.

34.

35.

36. Used REST interface to push & pull messages with message queues, such as ActiveMQ

37. Used Celery for AMQP-based message queues

38.

39. According to metadata, different format handlers were invoked

40.

41.

42.

43.

44. Exports logic to Java-based scripts

45.

46. Used Nicolas Toll's workflow engine, which allows users to define & manage complex workflows

47.

48. Added a graphical dashboard

49.

50.

51. Used task-names for flexible handlers invocation

52. Used periodic tasks for driving the workflow engine

53.

54. Horizontal & vertical scaling

55. Reliable & easy to manage

56. Message queues allow distributing load horizontally

57. Used script-based auto-scaling – starting new instances based on load

58.

59.

60.

61.

62.

63. Recommend people to follow based on interests

64.

65. Objects are organized in hierarchical layers

66. Objects have autonomous micro behavior aggregating to the macro behavior of the system

67. Layers are organized in spatial grids, which enable easy sharding & parallel processing

68.

69. Dbpedia

70. ConceptNet

71.

72.

73. Celery & RabbitMQ are used

74. JSON is used as message payload

75.

76. most of the data processing

77. Very intensive

78.

79. Not always-on

80. Requires 3 rd party libraries, not limited to Python

81.

82.

83. Can be started just upon need

84.

85.

86.

87. Frameworks such as Fuzed can be integrated with Python/Django

88. We're working on it as a coding session & hope to deliver a prototype soon

89.

90. Suds

91. Workflow

92. django-xmpp

93. Fuzed

94. Google Chart API

95. Talend

96. Thanks! @dibau_naum_h

Large Scale Processing with Django

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (6)

Semelhante a Large Scale Processing with Django

Semelhante a Large Scale Processing with Django (20)

Mais de Udi Bauman

Mais de Udi Bauman (12)

Último

Último (20)

Large Scale Processing with Django