Let's Build a Service Oriented Data Pipeline!

•Transferir como PPTX, PDF•

0 gostou•2,546 visualizações

Like many software projects, data pipelines built by Business Intelligence teams often start out as quickly built monoliths, but over time they can be made simpler and more maintainable by splitting them into a series of loosely coupled jobs, stealing ideas from service oriented architecture. In this presentation we'll look at the challenges we've faced scaling Data Analytics at Hootsuite, then move into a live coding session, where we'll stitch together a data pipeline as a series of Scala apps, deployed to AWS Lambda, connected using Airbnb's open source Airflow tool.

Tecnologia

Let’s Build a Service Oriented
Data Pipeline!
June 2016
Software Developer | Hootsuite
Yasha Podeswa

Now: Software Developer at Hootsuite
Me!

Introduce a problem that requires a new data pipeline
Design it in a service oriented style
Build it on stage!
This Talk

Passive Aggressive Inc. just
cancelled their subscription!
Desperate Dan in trouble!
The Problem

What We’re Starting With
Things Users
Did

What We’re Starting With
Things
Organizations
Did

High Level Plan
JSON files
Calculate stats
about
organizations
DB

High Level Plan
JSON files
Calculate stats
about
organizations
DB
Extract
Transform
Load

JSON files
Calculate stats
about
organizations
DB
Clean and organize
data
Calculate stats per
organization

JSON files
Calculate stats
about
organizations
DB
Clean and organize
data
Calculate stats per
organization
Useful for lots of
things!

JSON files
Calculate stats
about
organizations
DB
Clean and organize
data
Calculate stats per
organization
Shouldn’t run until
dependent job done

Need a “Service” Communication and
Orchestration Layer!

First App Event Cleaning and Loading
Read logs from S3, clean and sort into different types of
events, load into data warehouse
Vanilla Scala app
AWS Lambda

Second App Organization Stat Calculation
Read cleaned/sorted events from data warehouse, calculate
stats about organization, load stats to data warehouse
Vanilla Scala app
AWS Lambda

Third App Airflow
Hook up the Lambda apps in a dependency graph
● Scheduling
● Retries
● Monitoring

Steal my code!
https://github.com/yashap/etl-load-events
https://github.com/yashap/etl-organization-stats
https://github.com/yashap/airflow

Mais conteúdo relacionado

Destaque

Integrating NiFi and Flink

Bryan Bende

Data is being collected more and more every year. Cloud applications, including IoT, web, and mobile send torrents of bits at our data centers that have to be processed and stored. In addition, users expect an always-on experience, with little room for error. Numerous companies are successfully doing this every day. In this webinar, you will learn about the convergence of complementary technologies: Spark, Mesos, Akka, Cassandra and Kafka (SMACK), how Apache Kafka can help you get your data under control and the critical role Kafka plays in your data pipeline. Webinar recording: https://youtu.be/uwYlwLyv-1s Webinar Q&A will be posted shortly.

Webinar - How to Build Data Pipelines for Real-Time Applications with SMACK &...

DataStax

Writing a Jenkins / Hudson plugin

Anthony Dahanne

Curator intro

Jordan Zimmerman

While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. Many organizations understand the use cases around their data – fraud detection, quality of service and technical operations, user behavior analysis, for example – but are not necessarily data infrastructure experts. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes an hour of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. Attendees will leave this session knowing not just which open source projects go into a system such as this, but how they work together, what tradeoffs and decisions need to be addressed, and how to present a single general purpose data platform to multiple applications. This session should be attended by data infrastructure engineers and architects planning, building, or maintaining similar systems.

Building an Event-oriented Data Platform with Kafka, Eric Sammer

confluent

In this session, we will cover our experience working with Apache NiFi, an easy to use, powerful, and reliable system to process and distribute a large volume of data. The first part of the session will be an introduction to Apache NiFi. We will go over NiFi main components and building blocks and functionality. In the second part of the session, we will show our use case for Apache NiFi and how it's being used inside our Data Processing infrastructure.

Data ingestion and distribution with apache NiFi

Lev Brailovskiy

This is the slide deck from my keynote at the EA User Event in Brussels, September 2015. Micro-services and micro-services architecture are the next hype in software development. Websites and blogs are full of introducing posts, the first books are being written and the first conferences organized. There’s big promises of scalability, flexibility and replaceability of individual elements in your landscape. However, when you are knee deep in the mud as a software architect at an insurance, it is very hard to find help on how to design applications and components in a micro-services architecture. During this talk Sander will show how he used Enterprise Architect to model the micro services architecture, and will explain the difficulties and the lessons learned, using many real-life examples.

Modeling Microservices

Sander Hoogendoorn

Building a real-time streaming platform using Kafka Connect + Kafka Streams

confluent

Apache kafka

Rahul Jain

Kafka Streams is a new stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a natural DSL for writing stream processing applications. As such it is the most convenient yet scalable option to analyze, transform, or otherwise process data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Spark Streaming or Storm, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka.

Introduction to Kafka Streams

Guozhang Wang

Introduction to Apache Kafka

Jeff Holoman

The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus, Rim...

confluent

reveal.js 3.0.0

Hakim El Hattab

Introduction to Kafka and Zookeeper

Rahul Jain

Destaque (14)

Integrating NiFi and Flink

Webinar - How to Build Data Pipelines for Real-Time Applications with SMACK &...

Writing a Jenkins / Hudson plugin

Curator intro

Building an Event-oriented Data Platform with Kafka, Eric Sammer

Data ingestion and distribution with apache NiFi

Modeling Microservices

Building a real-time streaming platform using Kafka Connect + Kafka Streams

Apache kafka

Introduction to Kafka Streams

Introduction to Apache Kafka

The Enterprise Service Bus is Dead! Long live the Enterprise Service Bus, Rim...

reveal.js 3.0.0

Introduction to Kafka and Zookeeper

Semelhante a Let's Build a Service Oriented Data Pipeline!

SPS Warsaw 2017 - Building applications with PowerApps, Microsoft flow and Of...

Ahmad Najjar

Microsoft Flow is a brand new SaaS offering, for automating workflows across the growing number of applications and SaaS services that business users rely on. This session introduces Microsoft Flow and how it can help you save time, and make your business highly visible by partnering with Microsoft's growing ecosystem including O365 and SharePoint Online. It demonstrates how you can create an automated workflow between your favorite O365 apps and services to get notifications, synchronize files, collect data, and more!

SPS Brno 2017 - Go with the Microsoft flow

Ahmad Najjar

Introduction to Microsoft Power BI

CCG

Jira - Solving Reporting Problems using eazyBI

Marlon Palha

Synapse NanoApps

Synapse Information Ltd

Microsoft Business Intelligence - Practical Approach & Overview

Li Ken Chong

Intro to PowerApps and Flow

April Dunnam

Microsoft Flow is a new service (currently in preview) that helps you to build connection between different services and apps. You can create several workflows using available templates in O365 to do specific tasks like sending e-mail notifications to an app or sending messages at specific schedules. In this session you will get an overview of what Microsoft Flow can do for you and how to create your own custom workflows easily using easy to learn visual designer tools.

Automating your tasks with microsoft flow

Dipti Chhatrapati

Power BI has in its DNA the goal of enabling everybody to experience their data any way, anywhere—in seconds and at global scale. Power BI offers a set of capabilities that are uniquely enabled by its global and cloud nature: The ability to harness data from Excel spreadsheets, on-premises data sources through the data gateway, big data, streaming data, and cloud services. It doesn’t matter what type of data you want or where it lives, Power BI allows you to connect to hundreds of data sources. Out-of-the box SaaS content packs that deliver a curated experience with pre-built dashboards to get you up and running quickly. We have hundreds of ISVs building content packs to cater to the needs of millions of Power BI users. Unmatched, unique ways for users to experience their data with speed and agility: Live dashboards that maintain a real-time pulse on the business and provide critical insights. Natural language query that enables users to simply and intuitively ask questions of their data, including through Cortana. Custom visuals that bring data to life and surface intelligence hidden in the sea of data, with our community leveraging the Power BI visualization stack to create new ways to visualize data in a way that makes more sense. (Now available in the Office store.) Integration of Power BI with the Microsoft stack. Power BI is part of larger ecosystem that integrates with services like Microsoft Teams, Office 365, and Dynamics 365. These services are aware of Power BI, are wired to Power BI, and enable you to use Power BI in the context of your work. Anywhere access to insights. Whether in the office or on-the-go, Power BI provides anywhere access to insights with dashboards accessible via the desktop, on the web, or across mobile devices. Inside Excel, embedded—we have hundreds of ISVs embedding Power BI in their offerings.

Microsoft Power BI Technical Overview

David J Rosenthal

Introduction to Data Analytics

Dr. C.V. Suresh Babu

Ha100 notes units 1 and 2 sp08

Duskydope Rao

PowerApps and Flow are services for building and using custom business apps that connect to your data and work across the web and mobile - without the time and expense of custom software development. However, there are scenarios where the need to customize and develop extra features for workflows are vital, in order to take workflows to the next level. This session demonstrates key advanced scenarios addressing the next level workflows using PowerApps and Flow, featuring Azure Api Apps and Azure Functions.

SPS Brno 2017 - PowerApps & Microsoft Flow: Advanced scenarios

Ahmad Najjar

SharePoint 2010: Insights into BI

J.D. Wade

The slide deck for the Power Platform Presentation in SQL Saturday Redmond 2019. We have reviewed the power Platform Components, why is it better together and how to make it happen. During the demo all the options of implementation between the Power Apps and PowerBI were demonstrated. Including the data visualization changes with new data feed. Use some of the following ideas in your organization and POC's for more complex implementations.

SQL Saturday Redmond The Power Platform

Berkovich Consulting

The Briefing Room with Dr. Robin Bloor and WhereScape Live Webcast on April 1, 2014 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=7b23b14b532bd7be60a70f6bd5209f03 In the Big Data shuffle, everyone is looking at Hadoop as “the answer” to collect interesting data from a new set of sources. While Hadoop has given organizations the power to gather more information assets than ever before, the question still looms: which data, regardless of source, structure, volume and all the rest, are significant for affecting business value – and how do we harness it? One effective approach is to bolster the data warehouse environment with a solution capable of integrating all the data sources, including Hadoop, and automating delivery of key information into the rights hands. Register for this episode of The Briefing Room to hear veteran Analyst Robin Bloor as he explains how a rapidly changing information landscape impacts data management. He will be briefed by Mark Budzinski of WhereScape, who will tout his company’s data warehouse automation solutions. Budzinski will discuss how automation can be the cornerstone for closing the gap between those responsible for data management and the people driving business decisions. Visit InsideAnlaysis.com for more information.

The Right Data Warehouse: Automation Now, Business Value Thereafter

Inside Analysis

SPS Dubai 2017 - PowerApps & Microsoft Flow: Advanced scenarios

Ahmad Najjar

Projects are dynamic. Staying ahead of the curve means having real-time access to your project data and it’s leading indicators for actionable intelligence. A real-time, flexible platform Tivitie Insights service enables organizations to visualize critical Microsoft Project Online data in real-time, spot trends as they occur and produce actionable facts for better decision making in a fraction of the time.

Project Insights for Data Driven Decisions

Michelle Manimtim

Loading text data from SAP source systems

Marcelo Honores

In this session, you learn about the latest and hottest features of Amazon Redshift. Join Vidhya Srinivasan, General Manager of Amazon Redshift, to take a deep dive into the architecture and inner workings of Amazon Redshift. You discover how the recent availability, performance, and manageability improvements we’ve made can significantly enhance your end user experience. You also get a glimpse of what we are working on and our plans for the future.

AWS re:Invent 2016: What’s New with Amazon Redshift (BDA304)

Amazon Web Services

SPS London 2017 - Building applications with PowerApps, Microsoft flow and Of...

Ahmad Najjar

Semelhante a Let's Build a Service Oriented Data Pipeline! (20)

SPS Warsaw 2017 - Building applications with PowerApps, Microsoft flow and Of...

SPS Brno 2017 - Go with the Microsoft flow

Introduction to Microsoft Power BI

Jira - Solving Reporting Problems using eazyBI

Synapse NanoApps

Microsoft Business Intelligence - Practical Approach & Overview

Intro to PowerApps and Flow

Automating your tasks with microsoft flow

Microsoft Power BI Technical Overview

Introduction to Data Analytics

Ha100 notes units 1 and 2 sp08

SPS Brno 2017 - PowerApps & Microsoft Flow: Advanced scenarios

SharePoint 2010: Insights into BI

SQL Saturday Redmond The Power Platform

The Right Data Warehouse: Automation Now, Business Value Thereafter

SPS Dubai 2017 - PowerApps & Microsoft Flow: Advanced scenarios

Project Insights for Data Driven Decisions

Loading text data from SAP source systems

AWS re:Invent 2016: What’s New with Amazon Redshift (BDA304)

SPS London 2017 - Building applications with PowerApps, Microsoft flow and Of...

Último

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Orbitshub

Discover the innovative features and strategic vision that keep WSO2 an industry leader. Explore the exciting 2024 roadmap of WSO2 API management, showcasing innovations, unified APIM/APK control plane, natural language API interaction, and cloud native agility. Discover how open source solutions, microservices architecture, and cloud native technologies unlock seamless API management in today's dynamic landscapes. Leave with a clear blueprint to revolutionize your API journey and achieve industry success!

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

DBX First Quarter 2024 Investor Presentation

Dropbox

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

Vector Search -An Introduction in Oracle Database 23ai.pptx

Remote DBA Services

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

CNIC Information System with Pakdata Cf In Pakistan

danishmna97

ICT role in 21st century education and its challenges

rafiqahmad00786416

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

Retrieval augmented generation (RAG) is the most popular style of large language model application to emerge from 2023. The most basic style of RAG works by vectorizing your data and injecting it into a vector database like Milvus for retrieval to augment the text output generated by an LLM. This is just the beginning. One of the ways that we can extend RAG, and extend AI, is through multilingual use cases. Typical RAG is done in English using embedding models that are trained in English. In this talk, we’ll explore how RAG could work in languages other than English. We’ll explore French, Chinese, and Polish.

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Zilliz

Passkeys: Developing APIs to enable passwordless authentication Cody Salas, Sr Developer Advocate | Solutions Architect - Yubico Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

apidays

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Let's Build a Service Oriented Data Pipeline!

1. Let’s Build a Service Oriented Data Pipeline! June 2016 Software Developer | Hootsuite Yasha Podeswa

2. Before: Oceanographer Me!

3. Now: Software Developer at Hootsuite Me!

4. Introduce a problem that requires a new data pipeline Design it in a service oriented style Build it on stage! This Talk

5. Passive Aggressive Inc. just cancelled their subscription! Desperate Dan in trouble! The Problem

6. Want to Build a Tool Like This

7. Want to Build a Tool Like This

8. Want to Build a Tool Like This

10. What We’re Starting With

11. What We’re Starting With Things Users Did

12. What We’re Starting With Things Organizations Did

13. What We’re Starting With Crap

14. High Level Plan JSON files Calculate stats about organizations DB

15. High Level Plan JSON files Calculate stats about organizations DB Extract Transform Load

16. High Level Plan JSON files Calculate stats about organizations DB Extract Transform Load

17. JSON files Calculate stats about organizations DB Clean and organize data Calculate stats per organization

18. JSON files Calculate stats about organizations DB Clean and organize data Calculate stats per organization Useful for lots of things!

19. JSON files Calculate stats about organizations DB Clean and organize data Calculate stats per organization Shouldn’t run until dependent job done

20. Need a “Service” Communication and Orchestration Layer!

21.

22. Let’s build it!

23. First App Event Cleaning and Loading Read logs from S3, clean and sort into different types of events, load into data warehouse Vanilla Scala app AWS Lambda

24. Second App Organization Stat Calculation Read cleaned/sorted events from data warehouse, calculate stats about organization, load stats to data warehouse Vanilla Scala app AWS Lambda

25. Third App Airflow Hook up the Lambda apps in a dependency graph ● Scheduling ● Retries ● Monitoring

26. Steal my code! https://github.com/yashap/etl-load-events https://github.com/yashap/etl-organization-stats https://github.com/yashap/airflow

27. Questions?

Let's Build a Service Oriented Data Pipeline!

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (14)

Semelhante a Let's Build a Service Oriented Data Pipeline!

Semelhante a Let's Build a Service Oriented Data Pipeline! (20)

Último

Último (20)

Let's Build a Service Oriented Data Pipeline!