Data Intensive Applications with Apache Flink

•

0 gostou•655 visualizações

A brief introduction to Apache Flink and an overview of the current possibilities it offers to develop Machine Learning solutions.

Software

Milan – July 13 2016
Data Intensive Applications with Apache Flink
Simone Robutti
Machine Learning Engineer at Radicalbit
@SimoneRobutti

Agenda
1. Brief Introduction to Apache Flink
○ Why
○ What
○ How
2. Machine Learning on Flink
○ Present landscape
○ Future of the Ecosystem
3. Closing notes on Radicalbit (shameless plug ahead)

100% Buzzword-free guaranteed
Big Data
Machine
Intelligence
Web-scale
400x
It’s like the
human brain
Exactly-once
Exactly-once

Why Flink (and not Spark/Storm/Samza...)
Because it’s
production-ready
streaming-first
low-latency
fault-tolerant
high-throughput
processing engine

Flink: what is it?
From Flink’s Documentation

Flink’s Runtime
From Flink’s Documentation

Flink’s DataFlow
From Flink’s Documentation
Written by the user through DataSet/DataStream API
Compiled and optimized in the client

Flink’s DataFlow
From Flink’s Documentation
The compiled job is translated to distributed tasks by
the master and executed by workers

Ready and awesome for parallel ML
Work in progress for distributed ML
ML on Flink

Flink for Model Evaluation Pipelines
Source
Data
Preparation
Evaluation Sink
Source
Post
process
-ing
Composable, modular Flink Operator

Evaluation with Flink-JPMML
Source
Operator
Flink -
JPMML
Operator
Sink
Operator
Source
Operator
model.pmml
Small library that implements basic model eval.
Data
Preparation

“I have seen people insisting on using Hadoop for
datasets that could easily fit on a flash drive and could
easily be processed on a laptop.”
- Yann LeCun
-
ML on Flink

FlinkML
What: Out-of-the-box workhorse algorithms (ALS,
SVM, LinReg, LogReg …)
Status: early phase, slow development

FlinkML
Pro: available out of the box, written with Flink API
Cons: reinvents the wheel, only a few algorithms,
no model persistence

Samsara
What: Linear algebra framework
Status: mature

Samsara
Pro: generic algorithms with platform-specific
bindings, skilled community
Cons: covers only a few use cases

SAMOA
What: Online learning algorithm framework (VHT,
AMR, …)
Status: early phase, complicated relationship with
the industry

SAMOA
Pro: many powerful generic online learning
algorithms, backed by academics (MOA, Weka)
Cons: not production ready, academic focus

ML on Flink: the future of the ecosystem

Apache Beam
Programming model for data processing pipelines
● Streaming first, batch as a bounded stream
● Layered API: What, Where, When, How
● Platform agnostic: same program, different
runners

Apache Beam - Runners
● Flink
● Spark (Partial)
● Google Cloud Dataflow
● Plain Java
● Gearpump (WIP)
● Apex (WIP)

FlinkML Roadmap
● More algorithms!
● Evaluation framework
● Persistence/export
● Online Learning Framework

Proteus
Online Learning Platform - based on Flink
Source: Proteus’ website

Contributions
● Cassandra Connector
● Scala API extensions
● FlinkML (Linear Algebra Framework, MinHash)
● Akka Connector

Our vision
Flink can become the ideal choice to build real-time decision-
heavy applications with high data-throughput
To achieve this:
● Ambitious applications (aim for real-time services)
● Reliable distributed online learning (Proteus?)
● A Pipelining Framework (experiment fast, increase testability and
modularity)

THANKS!
Simone Robutti
Mail: simone.robutti@radicalbit.io Medium: @simone.robutti
Twitter: @SimoneRobutti — @weareradicalbit

Mais conteúdo relacionado

Mais procurados

Apache Spark has rapidly become a key tool for data scientists to explore, understand and transform massive datasets and to build and train advanced machine learning models. The question then becomes, how do I deploy these model to a production environment? How do I embed what I have learned into customer facing data applications? In this webinar, we will discuss best practices from Databricks on how our customers productionize machine learning models do a deep dive with actual customer case studies, show live tutorials of a few example architectures and code in Python, Scala, Java and SQL.

Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models

Anyscale

Automated Machine Learning (AutoML) has received significant interest recently. We believe that the right automation would bring significant value and dramatically shorten time-to-value for data science teams. Databricks is automating the Data Science and Machine Learning process through a combination of product offerings, partnerships, and custom solutions. This talk will focus on how Databricks can help automate hyperparameter tuning. For both traditional Machine Learning and modern Deep Learning, tuning hyperparameters can dramatically increase model performance and improve training times. However, tuning can be a complex and expensive process. In this talk, we'll start with a brief survey of the most popular techniques for hyperparameter tuning (e.g., grid search, random search, and Bayesian optimization). We will then discuss open source tools that implement each of these techniques, helping to automate the search over hyperparameters. Finally, we will discuss and demo improvements we built for these tools in Databricks, including integration with MLflow: Apache PySpark MLlib integration with MLflow for automatically tracking tuning Hyperopt integration with Apache Spark to distribute tuning and with MLflow for automatic tracking Recording and notebooks will be provided after the webinar so that you can practice at your own pace. Presenters Joseph Bradley, Software Engineer, Databricks Joseph Bradley is a Software Engineer and Apache Spark PMC member working on Machine Learning at Databricks. Previously, he was a postdoc at UC Berkeley after receiving his Ph.D. in Machine Learning from Carnegie Mellon in 2013. Yifan Cao, Senior Product Manager, Databricks Yifan Cao is a Senior Product Manager at Databricks. His product area spans ML/DL algorithms and Databricks Runtime for Machine Learning. Prior to Databricks, Yifan worked on two Machine Learning products, applying NLP to find metadata and applying machine learning to predict equipment failures. He helped build the products from ground up to multi-million dollars in ARR. Yifan started his career as a researcher in quantum computing. Yifan received his B.S in UC Berkeley and Master from MIT.

Automated Hyperparameter Tuning, Scaling and Tracking

Databricks

Experimental Design for Distributed Machine Learning with Myles Baker

Databricks

Presented by Mostafa Madjipour., Senior Data Scientist at Time Inc. Next DSS NYC Event 👉 https://datascience.salon/newyork/ Next DSS LA Event 👉 https://datascience.salon/la/ Reducing the gap between R&D and production is still a challenge for data science/ machine learning engineering groups in many companies. Typically, data scientists develop the data-driven models in a research-oriented programming environment (such as R and python). Next, the data/machine learning engineers rewrite the code (typically in another programming language) in a way that is easy to integrate with production services. This process has some disadvantages: 1) It is time consuming; 2) slows the impact of data science team on business; 3) code rewriting is prone to errors. A possible solution to overcome the aforementioned disadvantages would be to implement a deployment strategy that easily embeds/transforms the model created by data scientists. Packages such as jPMML, MLeap, PFA, and PMML among others are developed for this purpose. In this talk we review some of the mentioned packages, motivated by a project at Time Inc. The project involves development of a near real-time recommender system, which includes a predictor engine, paired with a set of business rules.

Data Science Salon: A Journey of Deploying a Data Science Engine to Production

Formulatedby

Abstract: In recent years, the demand for machine learning experts has outpaced the supply, despite the surge of people entering the field. To address this gap, there have been big strides in the development of user-friendly machine learning software that can be used by non-experts. Although H2O and other tools have made it easier for practitioners to train and deploy machine learning models at scale, there is still a fair bit of knowledge and background in data science that is required to produce high-performing machine learning models. Deep Neural Networks in particular, are notoriously difficult for a non-expert to tune properly. In this presentation, we provide an overview of the the field of "Automatic Machine Learning" and introduce the new AutoML functionality in H2O. H2O's AutoML provides an easy-to-use interface which automates the process of training a large, comprehensive selection of candidate models and a stacked ensemble model which, in most cases, will be the top performing model in the AutoML Leaderboard. H2O AutoML is available in all the H2O interfaces including the h2o R package, Python module and the Flow web GUI. We will also provide simple code examples to get you started using AutoML. Erin’s Bio: Erin is a Statistician and Machine Learning Scientist at H2O.ai. She is the main author of H2O Ensemble. Before joining H2O, she was the Principal Data Scientist at Wise.io and Marvin Mobile Security (acquired by Veracode in 2012) and the founder of DataScientific, Inc. Erin received her Ph.D. in Biostatistics with a Designated Emphasis in Computational Science and Engineering from University of California, Berkeley. Her research focuses on ensemble machine learning, learning from imbalanced binary-outcome data, influence curve based variance estimation and statistical computing. She also holds a B.S. and M.A. in Mathematics.

Scalable Automatic Machine Learning in H2O

Sri Ambati

For a machine learning application to be successful, it is not enough to give highly accurate predictions: Customers also want to know why the model has made that prediction, so they can compare it against their intuition and (hopefully) gain trust in the model. However, there is a trade-off between model accuracy and explainability - for example, the more complex your feature transformations become, the harder it is to explain what the resulting features mean to the end customer. However, with the right system design this doesn't mean it has to be a binary choice between these two goals. It is possible to combine complex, even automatic, feature engineering with highly accurate models and explanations. We will describe how we are using lineage tracing to solve this issue at Salesforce Einstein, allowing good model explanations to coexist with automatic feature engineering and model selection. By building this into an open source AutoML library TransmogrifAI, an extension to SparkMlLib, it is easy to ensure a consistent level of transparency in all of our ML applications. As model explanations are provided out of the box, data scientists don't need to re-invent the wheel when model explanations need to be surfaced.

Balancing Automation and Explanation in Machine Learning

Databricks

MLLeap, or How to Productionize Data Science Workflows Using Spark by Mikha...

Spark Summit

Machine Learning With Spark

Shivaji Dutta

Using H2O AutoML for Kaggle Competitions

Sri Ambati

Machine Learning in Production The era of big data generation is upon us. Devices ranging from sensors to robots and sophisticated applications are generating increasing amounts of rich data (time series, text, images, sound, video, etc.). For such data to benefit a business’s bottom line, insights must be extracted, a process that increasingly requires machine learning (ML) and deep learning (DL) approaches deployed in production applications use cases. Production ML is complicated by several challenges, including the need for two very distinct skill sets (operations and data science) to collaborate, the inherent complexity and uniqueness of ML itself, when compared to other apps, and the varied array of analytic engines that need to be combined for a practical deployment, often across physically distributed infrastructure. Nisha Talagala shares solutions and techniques for effectively managing machine learning and deep learning in production with popular analytic engines such as Apache Spark, TensorFlow, and Apache Flink.

Strata parallel m-ml-ops_sept_2017

Nisha Talagala

Recommendations for Building Machine Learning Software: Building a real system that uses machine learning can be a difficult both in terms of the algorithmic and engineering challenges involved. In this talk, I will focus on the engineering side and discuss some of the practical lessons we’ve learned from years of developing the machine learning systems that power Netflix. I will go over what it takes to get machine learning working in a real-life feedback loop with our users and how that imposes different requirements and a different focus than doing machine learning only within a lab environment. This involves lessons around challenges such as where to place algorithmic components, how to handle distribution and parallelism, what kinds of modularity are useful, how to support both production experimentation, and how to test machine learning systems.

Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...

MLconf

Spark Summit EU talk by Reza Karimi

Spark Summit

Building on the "Best Practices for Hyperparameter Tuning with MLflow" talk, we will present advanced topics in HPO for deep learning, including early stopping, multi-metric optimization, and robust optimization. We will then discuss implementations using open source tools. Finally, we will discuss how we can leverage MLflow with these tools and techniques to analyze the performance of our models.

Advanced Hyperparameter Optimization for Deep Learning with MLflow

Databricks

Interested in learning how Showtime is leveraging the power of Spark to transform a traditional premium cable network into a data-savvy analytical competitor? The growth in our over-the-top (OTT) streaming subscription business has led to an abundance of user-level data not previously available. To capitalize on this opportunity, we have been building and evolving our unified platform which allows data scientists and business analysts to tap into this rich behavioral data to support our business goals. We will share how our small team of data scientists is creating meaningful features which capture the nuanced relationships between users and content; productionizing machine learning models; and leveraging MLflow to optimize the runtime of our pipelines, track the accuracy of our models, and log the quality of our data over time. From data wrangling and exploration to machine learning and automation, we are augmenting our data supply chain by constantly rolling out new capabilities and analytical products to help the organization better understand our subscribers, our content, and our path forward to a data-driven future. Authors: Josh McNutt, Keria Bermudez-Hernandez

Data-Driven Transformation: Leveraging Big Data at Showtime with Apache Spark

Databricks

Machine Learning In Production

Samir Bessalah

Spark Summit EU talk by Oscar Castaneda

Spark Summit

SparkApplicationDevMadeEasy_Spark_Summit_2015

Lance Co Ting Keh

Splice Machine is an ANSI-SQL Relational Database Management System (RDBMS) on Apache Spark. It has proven low-latency transactional processing (OLTP) as well as analytical processing (OLAP) at petabyte scale. It uses Spark for all analytical computations and leverages HBase for persistence. This talk highlights a new Native Spark Datasource - which enables seamless data movement between Spark Data Frames and Splice Machine tables without serialization and deserialization. This Spark Datasource makes machine learning libraries such as MLlib native to the Splice RDBMS . Splice Machine has now integrated MLflow into its data platform, creating a flexible Data Science Workbench with an RDBMS at its core. The transactional capabilities of Splice Machine integrated with the plethora of DataFrame-compatible libraries and MLflow capabilities manages a complete, real-time workflow of data-to-insights-to-action. In this presentation we will demonstrate Splice Machine's Data Science Workbench and how it leverages Spark and MLflow to create powerful, full-cycle machine learning capabilities on an integrated platform, from transactional updates to data wrangling, experimentation, and deployment, and back again.

Splice Machine's use of Apache Spark and MLflow

Databricks

This talk discusses the trajectory of MLlib, the Machine Learning (ML) library for Apache Spark. We will review the history of the project, including major trends and efforts leading up to today. These discussions will provide perspective as we delve into ongoing and future efforts within the community. This talk is geared towards both practitioners and developers and will provide a deeper understanding of priorities, directions and plans for MLlib. Since the original MLlib project was merged into Apache Spark, some of the most significant efforts have been in expanding algorithmic coverage, adding multiple language APIs, supporting ML Pipelines, improving DataFrame integration, and providing model persistence. At an even higher level, the project has evolved from building a standard ML library to supporting complex workflows and production requirements. This momentum continues. We will discuss some of the major ongoing and future efforts in Apache Spark based on discussions, planning and development amongst the MLlib community. We (the community) aim to provide pluggable and extensible APIs usable by both practitioners and ML library developers. To take advantage of Projects Tungsten and Catalyst, we are exploring DataFrame-based implementations of ML algorithms for better scaling and performance. Finally, we are making continuous improvements to core algorithms in performance, functionality, and robustness. We will augment this discussion with statistics from project activity.

Apache Spark's MLlib's Past Trajectory and new Directions

Databricks

Machine learning techniques are powerful, but building and deploying such models for production use require a lot of care and expertise. A lot of books, articles, and best practices have been written and discussed on machine learning techniques and feature engineering, but putting those techniques into use on a production environment is usually forgotten and under- estimated , the aim of this talk is to shed some lights on current machine learning deployment practices, and go into details on how to deploy sustainable machine learning pipelines.

Deploying Machine Learning Models to Production

Anass Bensrhir - Senior Data Scientist

Mais procurados (20)

Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models

Automated Hyperparameter Tuning, Scaling and Tracking

Experimental Design for Distributed Machine Learning with Myles Baker

Data Science Salon: A Journey of Deploying a Data Science Engine to Production

Scalable Automatic Machine Learning in H2O

Balancing Automation and Explanation in Machine Learning

MLLeap, or How to Productionize Data Science Workflows Using Spark by Mikha...

Machine Learning With Spark

Using H2O AutoML for Kaggle Competitions

Strata parallel m-ml-ops_sept_2017

Justin Basilico, Research/ Engineering Manager at Netflix at MLconf SF - 11/1...

Spark Summit EU talk by Reza Karimi

Advanced Hyperparameter Optimization for Deep Learning with MLflow

Data-Driven Transformation: Leveraging Big Data at Showtime with Apache Spark

Machine Learning In Production

Spark Summit EU talk by Oscar Castaneda

SparkApplicationDevMadeEasy_Spark_Summit_2015

Splice Machine's use of Apache Spark and MLflow

Apache Spark's MLlib's Past Trajectory and new Directions

Deploying Machine Learning Models to Production

Destaque

Pattern matching over event streams is increasingly being employed in many areas including financial services and click stream analysis. Flink, as a true stream processing engine, emerges as a natural candidate for these usecases. In this talk, we will present FlinkCEP, a library for Complex Event Processing (CEP) based on Flink. At the conceptual level, we will see the different patterns the library can support, we will present the main building blocks we implemented to support them, and we will discuss possible future additions that will further enhance the coverage of the library. At the practical level, we will show how the integration of FlinkCEP with Flink allows the former to take advantage of Flink's rich ecosystem (e.g. connectors) and its stream processing capabilities, such as support for event-time processing, exactly-once state semantics, fault-tolerance, savepoints and high throughput.

Kostas Kloudas - Complex Event Processing with Flink: the state of FlinkCEP

Ververica

Machine Learning with Apache Flink at Stockholm Machine Learning Group

Till Rohrmann

Flink Forward Berlin 2017: Kostas Kloudas - Complex Event Processing with Fli...

Flink Forward

This session will introduce a new open-source project - Flink TensorFlow - that enables Flink programs to operate on data using TensorFlow machine learning models. Applications include real-time image processing, NLP, and anomaly detection. The session will: - Introduce TensorFlow and describe its component model which allows for model reuse across environments - Demonstrate how to use TensorFlow models in Flink ML and Flink Streaming environments - Present a roadmap and provide opportunities to contribute

Flink Forward SF 2017: Eron Wright - Introducing Flink Tensorflow

Flink Forward

SK telecom shares our experience of using Flink in building a solution for Predictive Maintenance (PdM). Our PdM solution named metatron PdM consists of (1) a Deep Neural Network (DNN)-based prediction model for precise prediction, and (2) a Flink-based runtime system which applies the model to a sliding window on sensor data streams. Efficient handling of multi-sensor streaming data for real-time prediction of equipment condition is a critical component of our product. In this talk, we first show why we choose Flink as a core engine for our streaming use case in which we generate real-time predictions using DNNs trained with Keras on top of TensorFlow and Theano. In addition, we present a comparative study of methods to exploit learning models on JVM such as directly using Python libraries on CPython embedded in JVM, using TensorFlow Java API (including Flink TensorFlow), and making RPC calls to TensorFlow Serving. We then explain how we implement the runtime system using Flink DataStream API, especially with event time, various window mechanisms, timestamp and watermark, custom source and sink, and checkpointing. Lastly, we present how we use the official Flink Docker image for solution delivery and the Flink metric system for monitoring and management of our solution. We hope our use case sets a good example of building a DNN-based streaming solution using Flink.

Flink Forward Berlin 2017: Dongwon Kim - Predictive Maintenance with Apache F...

Flink Forward

Electricity price forecasting with Recurrent Neural Networks

Taegyun Jeon

Feature Engineering

HJ van Veen

Destaque (7)

Kostas Kloudas - Complex Event Processing with Flink: the state of FlinkCEP

Machine Learning with Apache Flink at Stockholm Machine Learning Group

Flink Forward Berlin 2017: Kostas Kloudas - Complex Event Processing with Fli...

Flink Forward SF 2017: Eron Wright - Introducing Flink Tensorflow

Flink Forward Berlin 2017: Dongwon Kim - Predictive Maintenance with Apache F...

Electricity price forecasting with Recurrent Neural Networks

Feature Engineering

Semelhante a Data Intensive Applications with Apache Flink

Apache Fink 1.0: A New Era for Real-World Streaming Analytics

Slim Baltagi

Apache Flink - Overview and Use cases of a Distributed Dataflow System (at pr...

Stephan Ewen

Apache Spark vs Apache Flink

AKASH SIHAG

Overview of Apache Flink: the 4G of Big Data Analytics Frameworks

DataWorks Summit/Hadoop Summit

Slides of my talk at the Hadoop Summit Europe in Dublin, Ireland on April 13th, 2016. The talk introduces Apache Flink as both a multi-purpose Big Data analytics framework and real-world streaming analytics framework. It is focusing on Flink's key differentiators and suitability for streaming analytics use cases. It also shows how Flink enables novel use cases such as distributed CEP (Complex Event Processing) and querying the state by behaving like a key value data store.

Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks

Slim Baltagi

Overview of Apache Fink: The 4G of Big Data Analytics Frameworks

Slim Baltagi

Portable Streaming Pipelines with Apache Beam

confluent

The world of big data involves an ever-changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the big data ecosystem together; it enables users to "run any data processing pipeline anywhere." This talk will briefly cover the capabilities of the Beam model for data processing and discuss its architecture, including the portability model. We’ll focus on the present state of the community and the current status of the Beam ecosystem. We’ll cover the state of the art in data processing and discuss where Beam is going next, including completion of the portability framework and the Streaming SQL. Finally, we’ll discuss areas of improvement and how anybody can join us on the path of creating the glue that interconnects the big data ecosystem. Speaker Davor Bonaci, Apache Software Foundation; Simbly, V.P. of Apache Beam; Founder/CEO at Operiant

Present and future of unified, portable, and efficient data processing with A...

DataWorks Summit

The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam (incubating) aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In this talk, I will: Cover briefly the capabilities of the Beam model for data processing and integration with IOs, as well as the current state of the Beam ecosystem. Discuss the benefits Beam provides regarding portability and ease-of-use. Demo the same Beam pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Flink on Google Cloud, Apache Spark on AWS, Apache Apex on-premise). Give a glimpse at some of the challenges Beam aims to address in the future.

Realizing the promise of portability with Apache Beam

J On The Beach

Apache Beam is a top-level Apache project which aims at providing a unified API for efficient and portable data processing pipeline. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, Apache Apex, ...) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, describe the main concepts of the programming model and talk about the current state of the project (new python support, first stable version). We'll illustrate the concepts with a use case running on several runners.

Portable batch and streaming pipelines with Apache Beam (Big Data Application...

Malo Denielou

Flink in action

Artem Semenenko

Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...

Provectus

Near real-time anomaly detection at Lyft

markgrover

Introduction to Apache Flink

datamantra

This talk was given at Capital One on September 15, 2015 at the launch of the Washington DC Area Apache Flink Meetup. Apache flink is positioned at the forefront of 2 major trends in Big Data Analytics: - Unification of Batch and Stream processing - Multi-purpose Big Data Analytics frameworks In these slides, we will also find answers to the burning question: Why Apache Flink? You will also learn more about how Apache Flink compares to Hadoop MapReduce, Apache Spark and Apache Storm.

Unified Batch and Real-Time Stream Processing Using Apache Flink

Slim Baltagi

Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019

Thomas Weise

LAMP is so yesterday, MEAN is so tomorrow! :)

Sascha Sambale

Apache Arrow at DataEngConf Barcelona 2018

Wes McKinney

Technology Stack Discussion

Zaiyang Li

Flink history, roadmap and vision

Stephan Ewen

Semelhante a Data Intensive Applications with Apache Flink (20)

Apache Fink 1.0: A New Era for Real-World Streaming Analytics

Apache Flink - Overview and Use cases of a Distributed Dataflow System (at pr...

Apache Spark vs Apache Flink

Overview of Apache Flink: the 4G of Big Data Analytics Frameworks

Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks

Overview of Apache Fink: The 4G of Big Data Analytics Frameworks

Portable Streaming Pipelines with Apache Beam

Present and future of unified, portable, and efficient data processing with A...

Realizing the promise of portability with Apache Beam

Portable batch and streaming pipelines with Apache Beam (Big Data Application...

Flink in action

Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...

Near real-time anomaly detection at Lyft

Introduction to Apache Flink

Unified Batch and Real-Time Stream Processing Using Apache Flink

Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019

LAMP is so yesterday, MEAN is so tomorrow! :)

Apache Arrow at DataEngConf Barcelona 2018

Technology Stack Discussion

Flink history, roadmap and vision

Último

At the recent Microsoft Ignite 2023 conference, Microsoft unveiled a groundbreaking strategy that will redefine enterprise work management. The plan involves integrating Microsoft’s key planning tools, Microsoft To Do, Microsoft Planner, and Microsoft Project for the web into a unified experience called “Microsoft Planner.” What does this new strategy from Microsoft mean for current users? Join us and learn how best to take advantage of this announcement while gaining a clear path on how to elevate the current state of Microsoft Planner from a basic task manager to a comprehensive tool for Enterprise Work Management using OnePlan. Learn how OnePlan’s integration with Microsoft Planner allows for strategic alignment with business goals through advanced features like strategic planning, portfolio management, resource management, financial management, and more!

Introducing Microsoft’s new Enterprise Work Management (EWM) Solution

OnePlan Solutions

(Vivek)Call Us, 8448380779,Call girls in Delhi NCr – We Offer best in class call girls. escort Service At Affordable Price At low Rate with Space Night 8000 We Are One Of The Oldest Escort and Call girls Agencies in Delhi. You Will Find That Our Female Escorts Are Full Of Fun, Sexy And They Would Love Enjoy Your Company. We Have A Fantastic Selection Of Escort Ladies Available For In-Calls As Well As Out-Calls. Our Escorts Are Not Only Beautiful But All Have Great Personalities Making Them The Perfect Companion For Any Occasion. In-Call:- You Can Come At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call:- You have To Come Pick The Girl From My Place We Are Also Provide Door Step Services (Delhi Ncr, Noida, Gurgaon, Faridabad, Ghaziabad Note:- Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic:- Full Ac room And Clean Rooms Available In Hotel 24 * 7 Hourly In Delhi NCR More Details, With WhatsApp Number, +91-8448380779

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️

Delhi Call girls

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

masabamasaba

MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...

Jittipong Loespradit

%in Bahrain+277-882-255-28 abortion pills for sale in Bahrain

masabamasaba

OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...

Shane Coughlan

Conference: Engage2024 in Antwerp Type: Workshop Speakers: Florian Vogler, Henning Kunz, Christoph Adler Title: Navigating the Future with The Hitchhiker's Guide to Notes and Domino 14 Abstract: Embark on an exhilarating journey with industry trailblazers Florian Vogler, Henning Kunz, and Christoph Adler in this not-to-be-missed workshop at the forefront of the tech universe. Get ready for a thrilling kick-off as we navigate the current state of the HCL universe, setting the stage for an exploration of the groundbreaking Notes and Domino 14. Discover the latest enhancements and revolutionary features that will redefine your experience. In this interactive session, unlock a treasure trove of tips and tricks to elevate your utilization of version 14, both with and without the game-changing panagenda MarvelClient. Brace yourself for also diving into Nomad, Nomad Web, and VoltMX, expanding your horizons in the expansive HCL landscape. Be a part of this exclusive opportunity to stay ahead in the ever-evolving world of HCL technologies. Your journey to mastering Notes and Domino 14 begins here. And remember, in the spirit of intergalactic exploration, don't forget to bring your towel!

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...

panagenda

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in Tembisa ● Abortion Pills For Sale in Tembisa ● Tembisa 🏥🚑!! Abortion Clinic Near Me Cost, Price, Women's Clinic Near Me, Abortion Clinic Near, Abortion Doctors Near me, Abortion Services Near Me, Abortion Pills Over The Counter, Abortion Pill Doctors' Offices, Abortion Clinics, Abortion Places Near Me, Cheap Abortion Places Near Me, Medical Abortion & Surgical Abortion, approved cyctotec pills and womb cleaning pills too plus all the instructions needed This Discrete women’s Termination Clinic offers same day services that are safe and pain free, we use approved pills and we clean the womb so that no side effects are present. Our main goal is that of preventing unintended pregnancies and unwanted births every day to enable more women to have children by choice, not chance. We offer Terminations by Pill and The Morning After Pill.” Our Private VIP Abortion Service offers the ultimate in privacy, efficiency and discretion. we do safe and same day termination and we do also womb cleaning as well its done from 1 week up to 28 weeks. We do delivery of our services world wide SAFE ABORTION CLINICS/PILLS ON SALE WE DO DELIVERY OF PILLS ALSO Abortion clinic at very low costs, 100% Guaranteed and it’s safe, pain free and a same day service. It Is A 45 Minutes Procedure, we use tested abortion pills and we do womb cleaning as well. Alternatively the medical abortion pill and womb cleansing !!!

Abortion Pill Prices Tembisa [(+27832195400*)] 🏥 Women's Abortion Clinic in T...

Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg

%in Harare+277-882-255-28 abortion pills for sale in Harare

masabamasaba

The title is not connected to what is inside

shinachiaurasa2

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️

Delhi Call girls

WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...

WSO2

WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity

WSO2

Craft an AI & Machine Learning Pitch with our Editable Professional PowerPoint Template. Ignite your AI & Machine Learning pitch with our cutting-edge PowerPoint template tailored for the industry. Perfect for AI conferences, investor presentations, sales pitches to tech-focused companies, training sessions, and educational programs. - 20+ editable slides: Get a variety of options to choose from for your presentation. - Time-saving solution: Download, replace text/images with a few clicks. - User-friendly customization: Easy to use and personalize. - Modern and attractive design: Captivating visuals, sleek layout. - Tailored to your requirements: Fully alterable for customization. - Well-organized slides: Complete control over content. - Thematic specificity: Reflects healthcare industry with relevant graphics. - Showcase your business idea: Communicate value proposition effectively.

AI & Machine Learning Presentation Template

Presentation.STUDIO

8257 interfacing 2 in microprocessor for btech students

HimanshiGarg82

tonesoftg

lanshi9

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal.

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

masabamasaba

Define the academic and professional writing..pdf

PearlKirahMaeRagusta1

Announcing Codolex 2.0 from GDK Software

Jim McKeeth

%in Soweto+277-882-255-28 abortion pills for sale in soweto

masabamasaba

Data Intensive Applications with Apache Flink

1. Milan – July 13 2016 Data Intensive Applications with Apache Flink Simone Robutti Machine Learning Engineer at Radicalbit @SimoneRobutti

2. Agenda 1. Brief Introduction to Apache Flink ○ Why ○ What ○ How 2. Machine Learning on Flink ○ Present landscape ○ Future of the Ecosystem 3. Closing notes on Radicalbit (shameless plug ahead)

3. 100% Buzzword-free guaranteed Big Data Machine Intelligence Web-scale 400x It’s like the human brain Exactly-once Exactly-once

4. Why Flink (and not Spark/Storm/Samza...) Because it’s production-ready streaming-first low-latency fault-tolerant high-throughput processing engine

5. Flink: what is it? From Flink’s Documentation

6. Connectors and integrations

7. Flink’s Runtime From Flink’s Documentation

8. Flink’s DataFlow From Flink’s Documentation Written by the user through DataSet/DataStream API Compiled and optimized in the client

9. Flink’s DataFlow From Flink’s Documentation The compiled job is translated to distributed tasks by the master and executed by workers

10. Machine Learning on Flink

11. Ready and awesome for parallel ML Work in progress for distributed ML ML on Flink

12. Flink for Model Evaluation Pipelines Source Data Preparation Evaluation Sink Source Post process -ing Composable, modular Flink Operator

13. Evaluation with Flink-JPMML Source Operator Flink - JPMML Operator Sink Operator Source Operator model.pmml Small library that implements basic model eval. Data Preparation

14. “I have seen people insisting on using Hadoop for datasets that could easily fit on a flash drive and could easily be processed on a laptop.” - Yann LeCun - ML on Flink

15.

16. FlinkML What: Out-of-the-box workhorse algorithms (ALS, SVM, LinReg, LogReg …) Status: early phase, slow development

17. FlinkML Pro: available out of the box, written with Flink API Cons: reinvents the wheel, only a few algorithms, no model persistence

18. Samsara What: Linear algebra framework Status: mature

19. Samsara Pro: generic algorithms with platform-specific bindings, skilled community Cons: covers only a few use cases

20. SAMOA What: Online learning algorithm framework (VHT, AMR, …) Status: early phase, complicated relationship with the industry

21. SAMOA Pro: many powerful generic online learning algorithms, backed by academics (MOA, Weka) Cons: not production ready, academic focus

22. ML on Flink: the future of the ecosystem

23. Apache Beam Programming model for data processing pipelines ● Streaming first, batch as a bounded stream ● Layered API: What, Where, When, How ● Platform agnostic: same program, different runners

24. Apache Beam - Runners ● Flink ● Spark (Partial) ● Google Cloud Dataflow ● Plain Java ● Gearpump (WIP) ● Apex (WIP)

25. BeamML: a runner-agnostic ML library

26. FlinkML Roadmap ● More algorithms! ● Evaluation framework ● Persistence/export ● Online Learning Framework

27. Proteus Online Learning Platform - based on Flink Source: Proteus’ website

28. The role of Radicalbit

29. Contributions ● Cassandra Connector ● Scala API extensions ● FlinkML (Linear Algebra Framework, MinHash) ● Akka Connector

30. Our vision Flink can become the ideal choice to build real-time decision- heavy applications with high data-throughput To achieve this: ● Ambitious applications (aim for real-time services) ● Reliable distributed online learning (Proteus?) ● A Pipelining Framework (experiment fast, increase testability and modularity)

31. Q&A

32. THANKS! Simone Robutti Mail: simone.robutti@radicalbit.io Medium: @simone.robutti Twitter: @SimoneRobutti — @weareradicalbit

Data Intensive Applications with Apache Flink

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (7)

Semelhante a Data Intensive Applications with Apache Flink

Semelhante a Data Intensive Applications with Apache Flink (20)

Último

Último (20)

Data Intensive Applications with Apache Flink