Distilling Insights @ Appsflyer (Data Architecture)

•

3 gostaram•1,888 visualizações

Arnon Rotem-Gal-Oz

Appsflyer's data architecture

Tecnologia

Distilling insights @
Arnon Rotem-‐Gal-‐Oz
Chief Data Officer

Kafka
Columnar Database
(Redshift- evaluating Vertica)
IMDG
(Ignite - evaluating Geode)
Secor
Spark
Aggregations
SparkSQL
(evaluating
Drill,
Presto)
SQL
SQL
Raw
(sequence
ﬁles)
DW
(parquet
ﬁles)
DM
(Aggregations)
Application
dashboard
Self-serve
BI
(TBD)
Spark
ETL
Spark
Spark
ML
Latest Events
Scoring
exploration
Agg.
logic
Internal
tools
installs clicksinapplaunches
Accounts

Data’s hierarchy of needs*
*With apologies to Maslow
Acted
upon
presented
Distilled
Usable
Accessible
Exist

“Malting”
Just slap SQL on everything
Accessible

RT insights
Predictive
Prescriptive
Dashboards
whatnot
presented

Enrich with foreign
keys before persisting

Mais conteúdo relacionado

Mais procurados

Bridging the Gap Between Datasets and DataFramesDatabricks

The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Mo...Spark Summit

Data Warehousing with Spark Streaming at ZalandoDatabricks

Cloud Experience: Data-driven Applications Made Simple and FastDatabricks

Processing genetic data at scaleMark Schroering

Sparkflows Use CasesJayant Shekhar

Building the Autodesk Design Graph-(Yotto Koga, Autodesk)Spark Summit

End-to-End Spark/TensorFlow/PyTorch Pipelines with Databricks DeltaDatabricks

Spark WorshopJuan Pedro Moreno

Unifying your data management with HadoopJayant Shekhar

Big Data Meets Learning Science: Keynote by Al EssaSpark Summit

Jump Start into Apache Spark (Seattle Spark Meetup)Denny Lee

Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in A...Databricks

Stream Processing: Choosing the Right Tool for the JobDatabricks

Shifting Data Science into High GearSpark Summit

DMM.comラボはなぜSparkを採用したのか？レコメンドエンジン開発の裏側をお話します！leverages_event

Vectorized R Execution in Apache SparkDatabricks

Field Notes from Expeditions in the Cloud-(Matt Wood, Amazon Web Services)Spark Summit

How to Boost 100x Performance for Real World Application with Apache Spark-(G...Spark Summit

New Developments in the Open Source Ecosystem: Apache Spark 3.0, Delta Lake, ...Databricks

Mais procurados (20)

Bridging the Gap Between Datasets and DataFrames

The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Mo...

Data Warehousing with Spark Streaming at Zalando

Cloud Experience: Data-driven Applications Made Simple and Fast

Processing genetic data at scale

Sparkflows Use Cases

Building the Autodesk Design Graph-(Yotto Koga, Autodesk)

End-to-End Spark/TensorFlow/PyTorch Pipelines with Databricks Delta

Spark Worshop

Unifying your data management with Hadoop

Big Data Meets Learning Science: Keynote by Al Essa

Jump Start into Apache Spark (Seattle Spark Meetup)

Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in A...

Stream Processing: Choosing the Right Tool for the Job

Shifting Data Science into High Gear

DMM.comラボはなぜSparkを採用したのか？レコメンドエンジン開発の裏側をお話します！

Vectorized R Execution in Apache Spark

Field Notes from Expeditions in the Cloud-(Matt Wood, Amazon Web Services)

How to Boost 100x Performance for Real World Application with Apache Spark-(G...

New Developments in the Open Source Ecosystem: Apache Spark 3.0, Delta Lake, ...

Semelhante a Distilling Insights @ Appsflyer (Data Architecture)

Apache spark - Architecture , Overview & librariesWalaa Hamdy Assy

Introduction to apache spark Aakashdata

Vertica And Spark: Connecting Computation And DataRui Liu

Vertica And Spark: Connecting Computation And DataSpark Summit

HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...Chetan Khatri

Spark4poovarasu maniandan

DataEngConf SF16 - Spark SQL WorkshopHakka Labs

Making Apache Spark Better with Delta LakeDatabricks

Fossasia 2018-chetan-khatriChetan Khatri

Spark SQL - 10 Things You Need to KnowKristian Alexander

A Tale of Two APIs: Using Spark Streaming In ProductionLightbend

What's New in Spark 2?Eyal Ben Ivri

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)Helena Edelson

SparkR: Enabling Interactive Data Science at Scalejeykottalam

SparkR: Enabling Interactive Data Science at Scale on HadoopDataWorks Summit

Running Presto and Spark on the Netflix Big Data PlatformEva Tse

(BDT303) Running Spark and Presto on the Netflix Big Data PlatformAmazon Web Services

Parallelizing Existing R PackagesCraig Warman

20170126 big data processingVienna Data Science Group

Netflix - Productionizing Spark On Yarn For ETL At Petabyte ScaleJen Aman

Semelhante a Distilling Insights @ Appsflyer (Data Architecture) (20)

Apache spark - Architecture , Overview & libraries

Introduction to apache spark

Vertica And Spark: Connecting Computation And Data

HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...

Spark4

DataEngConf SF16 - Spark SQL Workshop

Making Apache Spark Better with Delta Lake

Fossasia 2018-chetan-khatri

Spark SQL - 10 Things You Need to Know

A Tale of Two APIs: Using Spark Streaming In Production

What's New in Spark 2?

Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)

SparkR: Enabling Interactive Data Science at Scale

SparkR: Enabling Interactive Data Science at Scale on Hadoop

Running Presto and Spark on the Netflix Big Data Platform

(BDT303) Running Spark and Presto on the Netflix Big Data Platform

Parallelizing Existing R Packages

20170126 big data processing

Netflix - Productionizing Spark On Yarn For ETL At Petabyte Scale

Mais de Arnon Rotem-Gal-Oz

Taking ML to production - a journeyArnon Rotem-Gal-Oz

Apache sparkArnon Rotem-Gal-Oz

Fallacies of Distributed Computing Arnon Rotem-Gal-Oz

Docker & Kubernetes introArnon Rotem-Gal-Oz

Docker IntroArnon Rotem-Gal-Oz

Data security @ the personal levelArnon Rotem-Gal-Oz

Microservices - it's déjà vu all over againArnon Rotem-Gal-Oz

Big data in the cloud - welcome to cost oriented designArnon Rotem-Gal-Oz

Big data OverviewArnon Rotem-Gal-Oz

Hadoop YARN overviewArnon Rotem-Gal-Oz

SAFArnon Rotem-Gal-Oz

REST presentationArnon Rotem-Gal-Oz

SOA & Big DataArnon Rotem-Gal-Oz

Why the JVM?Arnon Rotem-Gal-Oz

Building reliable systems from unreliable componentsArnon Rotem-Gal-Oz

Azure migrationArnon Rotem-Gal-Oz

Things to think about while architecting azure solutionsArnon Rotem-Gal-Oz

Soa Arnon Rotem-Gal-Oz

RestArnon Rotem-Gal-Oz

SOA patterns Arnon Rotem-Gal-Oz

Mais de Arnon Rotem-Gal-Oz (20)

Taking ML to production - a journey

Apache spark

Fallacies of Distributed Computing

Docker & Kubernetes intro

Docker Intro

Data security @ the personal level

Microservices - it's déjà vu all over again

Big data in the cloud - welcome to cost oriented design

Big data Overview

Hadoop YARN overview

SAF

REST presentation

SOA & Big Data

Why the JVM?

Building reliable systems from unreliable components

Azure migration

Things to think about while architecting azure solutions

Soa

Rest

SOA patterns

Último

TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc

Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed

Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro

Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation

Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada

New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada

Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren

How to write a Business Continuity PlanDatabarracks

Powerpoint exploring the locations used in television show Time Clashcharlottematthew16

Advanced Computer Architecture – An IntroductionDilum Bandara

Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson

Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm

Story boards and shot lists for my a level piececharlottematthew16

DevEX - reference for building teams, processes, and platformsSergiu Bodiu

"ML in Production",Oleksandr BaganFwdays

Take control of your SAP testing with UiPath Test SuiteDianaGray10

From Family Reminiscence to Scholarly Archive .Alan Dix

Search Engine Optimization SEO PDF for 2024.pdfRankYa

DMCC Future of Trade Web3 - Special EditionDubai Multi Commodity Centre

Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar

Distilling Insights @ Appsflyer (Data Architecture)

1. Distilling insights @ Arnon Rotem-‐Gal-‐Oz Chief Data Officer

4. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence ﬁles) DW (parquet ﬁles) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts

5. Data’s hierarchy of needs* *With apologies to Maslow Acted upon presented Distilled Usable Accessible Exist

6. Exist

7. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence ﬁles) DW (parquet ﬁles) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts

8. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence ﬁles) DW (parquet ﬁles) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts

9. Working off of RAW data

10. “Malting” Just slap SQL on everything Accessible

11. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence ﬁles) DW (parquet ﬁles) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts

12. Fermenting Usable

13. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence ﬁles) DW (parquet ﬁles) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts

14. Distilling Distilled

15. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence ﬁles) DW (parquet ﬁles) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts

16. RT insights Predictive Prescriptive Dashboards whatnot presented

17. Sidetrack: On use of Spark

18. Hadoop & Mesos

19.

20. Land data in a queue

21. All data is time-‐series

22. Enrich with foreign keys before persisting

23. Analyze and balance jobs

24. Not everything is big data

25. We’re hiring…. jobs@appsflyer.com