SparkR: Enabling Interactive Data Science at Scale

In this webcast, Patrick Wendell from Databricks will be speaking about Apache Spark's new 1.6 release. Spark 1.6 will include (but not limited to) a type-safe API called Dataset on top of DataFrames that leverages all the work in Project Tungsten to have more robust and efficient execution (including memory management, code generation, and query optimization) [SPARK-9999], adaptive query execution [SPARK-9850], and unified memory management by consolidating cache and execution memory [SPARK-10000].

"SparkR" presentation by Shivaram Venkataraman and Zongheng Yang

Software

SparkR: Enabling Interactive
Data Science at Scale
Shivaram Venkataraman
Zongheng Yang

Fast !
Scalable
Flexible
Statistics !
Plots
Packages

Outline
SparkR API
Live Demo
Design Details

RDD
Parallel Collection
Transformations
map
filter
groupBy
…
Actions
count
collect
saveAsTextFile
…

R + RDD =
RRDD
lapply
lapplyPartition
groupByKey
reduceByKey
sampleRDD
collect
cache
filter
…
broadcast
includePackage
textFile
parallelize

SparkR – R package for Spark
R
RRDD
Spark

$Example: word_count.R library(SparkR) lines <-‐ textFile(sc, “hdfs://my_text_file”) words <-‐ flatMap(lines, function(line) { strsplit(line, " ")[[1]] }) wordCount <-‐ lapply(words, function(word) { list(word, 1L) })$

A
b
|| Ax − b ||2 Minimize
x = (ATA)−1ATb

Dataflow
Local
R Spark
Context
Java
Spark
Context
JNI
Worker
Worker

Dataflow
Local Worker
R Spark
Context
Worker Java
Spark
Context
JNI
Spark
Executor exec R
Spark
Executor exec R

From http://obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/

Pipelined RDD
words
<-‐
flatMap(lines,…)
wordCount
<-‐
lapply(words,…)
Spark
Executor exec R Spark
Executor R exec

Pipelined RDD
Spark
Executor exec R Spark
Executor R exec
Spark
Executor exec R R Spark
Executor

Alpha developer release
One line install !
install_github("amplab-‐extras/SparkR-‐pkg",
subdir="pkg")

SparkR Implementation
Very similar to PySpark
Spark is easy to extend
329 lines of Scala code
2079 lines of R code
693 lines of test code in R

EC2 setup scripts
All Spark examples
MNIST demo
YARN, Windows support
Also on github

Developer Community
13 contributors
(10 from outside AMPLab)
Collaboration with Alteryx

On the Roadmap
High level DataFrame API
Integrating Spark’s MLLib from R
Merge with Apache Spark

SparkR
RDD à distributed lists
Run R on clusters
Re-use existing packages
Combine scalability & utility

SparkR
https://github.com/amplab-extras/SparkR-pkg
Shivaram Venkataraman shivaram@cs.berkeley.edu
Zongheng Yang zongheng.y@gmail.com
SparkR mailing list sparkr-dev@googlegroups.com

Mais conteúdo relacionado

Mais procurados

Apache® Spark™ 1.6 presented by Databricks co-founder Patrick Wendell

R is a favorite language of many data scientists. In addition to a language and runtime, R is a rich ecosystem of libraries for a wide range of use cases from statistical inference to data visualization. However, handling large datasets with R is challenging, especially when data scientists use R with frameworks or tools written in other languages. In this mode most of the friction is at the interface of R and the other systems. For example, when data is sampled by a big data platform, results need to be transferred to and imported in R as native data structures. In this talk we show how SparkR solves these problems to enable a much smoother experience. In this talk we will present an overview of the SparkR architecture, including how data and control is transferred between R and JVM. This knowledge will help data scientists make better decisions when using SparkR. We will demo and explain some of the existing and supported use cases with real large datasets inside a notebook environment. The demonstration will emphasize how Spark clusters, R and interactive notebook environments, such as Jupyter or Databricks, facilitate exploratory analysis of large data.

Enabling exploratory data science with Spark and R

New directions for Apache Spark in 2015

In this webcast, Patrick Wendell from Databricks will be speaking about Apache Spark's new 1.5 release. Spark 1.5 ships Spark's Project Tungsten initiative, a cross-cutting performance update that uses binary memory management and code generation to dramatically improve latency of most Spark jobs. This release also includes several updates to Spark's DataFrame API and SQL optimizer, along with new Machine Learning algorithms and feature transformers, and several new features in Spark's native streaming engine.

Apache® Spark™ 1.5 presented by Databricks co-founder Patrick Wendell

At Databricks, we have a unique view into over a hundred different companies trying out Spark for development and production use-cases, from their support tickets and forum posts. Having seen so many different workflows and applications, some discernible patterns emerge when looking at common performance and scalability issues that our users run into. This talk will discuss some of these common common issues from an engineering and operations perspective, describing solutions and clarifying misconceptions.

Spark Summit EU 2015: Lessons from 300+ production users

A Data Frame Abstraction Layer for SparkR-(Chris Freeman, Alteryx)

End-to-end Data Pipeline with Apache Spark

Big data tools are challenging to combine into a larger application: ironically, big data applications themselves do not tend to scale very well. These issues of integration and data management are only magnified by increasingly large volumes of data. Apache Spark provides strong building blocks for batch processes, streams and ad-hoc interactive analysis. However, users face challenges when putting together a single coherent pipeline that could involve hundreds of transformation steps, especially when confronted by the need of rapid iterations. This talk explores these issues through the lens of functional programming. It presents an experimental framework that provides full-pipeline guarantees by introducing more laziness to Apache Spark. This framework allows transformations to be seamlessly composed and alleviates common issues, thanks to whole program checks, auto-caching, and aggressive computation parallelization and reuse.

From Pipelines to Refineries: Scaling Big Data Applications

Performant data processing with PySpark, SparkR and DataFrame API

Ryuji Tamagawa

Building a modern Application with DataFrames

Spark Application Carousel: Highlights of Several Applications Built with Spark

These are the slides from the Jump Start into Apache Spark and Databricks webinar on February 10th, 2016. --- Spark is a fast, easy to use, and unified engine that allows you to solve many Data Sciences and Big Data (and many not-so-Big Data) scenarios easily. Spark comes packaged with higher-level libraries, including support for SQL queries, streaming data, machine learning, and graph processing. We will leverage Databricks to quickly and easily demonstrate, visualize, and debug our code samples; the notebooks will be available for you to download.

Jump Start into Apache® Spark™ and Databricks

R has evolved to become an ideal environment for exploratory data analysis. The language is highly flexible - there is an R package for almost any algorithm and the environment comes with integrated help and visualization. SparkR brings distributed computing and the ability to handle very large data to this list. SparkR is an R package distributed within Apache Spark. It exposes Spark DataFrames, which was inspired by R data.frames, to R. With Spark DataFrames, and Spark’s in-memory computing engine, R users can interactively analyze and explore terabyte size data sets. In this webinar, Hossein will introduce SparkR and how it integrates the two worlds of Spark and R. He will demonstrate one of the most important use cases of SparkR: the exploratory analysis of very large data. Specifically, he will show how Spark’s features and capabilities, such as caching distributed data and integrated SQL execution, complement R’s great tools such as visualization and diverse packages in a real world data analysis project with big data.

Enabling Exploratory Analysis of Large Data with Apache Spark and R

In the last year Spark has seen substantial growth in adoption as well as the pace and scope of development. This talk will look forward and discuss both technical initiatives and the evolution of the Spark community. On the technical side, I’ll discuss two key initiatives ahead for Spark. The first is a tighter integration of Spark’s libraries through shared primitives such as the data frame API. The second is across-the-board performance optimizations that exploit schema information embedded in Spark’s newer APIs. These initiatives are both designed to make Spark applications easier to write and faster to run. On the community side, this talk will focus on the growing ecosystem of extensions, tools, and integrations evolving around Spark. I’ll survey popular language bindings, data sources, notebooks, visualization libraries, statistics libraries, and other community projects. Extensions will be a major point of growth in the future, and this talk will discuss how we can position the upstream project to help encourage and foster this growth.

Up and running with pyspark

Krishna Sangeeth KS

Strata NYC 2015 - What's coming for the Spark community

Introduction to Spark (Intern Event Presentation)

In the world of Data Science, Python and R are very popular. Apache Spark is a highly scalable data platform. How could a Data Scientist integrate Spark into their existing Data Science toolset? How does Python work with Spark? How could one leverage the rich 10000+ packages on CRAN for R? We will start with PySpark, beginning with a quick walkthrough of data preparation practices and an introduction to Spark MLLib Pipeline Model. We will also discuss how to integrate native Python packages with Spark. Compare to PySpark, SparkR is a new language binding for Apache Spark and it is designed to be familiar to native R users. In this talk we will walkthrough many examples how several new features in Apache Spark 2.x will enable scalable machine learning on Big Data. In addition to talking about the R interface to the ML Pipeline model, we will explore how SparkR support running user code on large scale data in a distributed manner, and give examples on how that could be used to work with your favorite R packages. Python R Apache Spark ML DL

Scalable Data Science in Python and R on Apache Spark

felixcss

Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...

Apache spark linkedin

Yukti Kaura

New Developments in Spark

Mais procurados (20)

Apache® Spark™ 1.6 presented by Databricks co-founder Patrick Wendell

Enabling exploratory data science with Spark and R

New directions for Apache Spark in 2015

Apache® Spark™ 1.5 presented by Databricks co-founder Patrick Wendell

Spark Summit EU 2015: Lessons from 300+ production users

A Data Frame Abstraction Layer for SparkR-(Chris Freeman, Alteryx)

End-to-end Data Pipeline with Apache Spark

From Pipelines to Refineries: Scaling Big Data Applications

Performant data processing with PySpark, SparkR and DataFrame API

Building a modern Application with DataFrames

Spark Application Carousel: Highlights of Several Applications Built with Spark

Jump Start into Apache® Spark™ and Databricks

Enabling Exploratory Analysis of Large Data with Apache Spark and R

Up and running with pyspark

Strata NYC 2015 - What's coming for the Spark community

Introduction to Spark (Intern Event Presentation)

Scalable Data Science in Python and R on Apache Spark

Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...

Apache spark linkedin

New Developments in Spark

Semelhante a SparkR: Enabling Interactive Data Science at Scale

Parallelizing Existing R Packages

Craig Warman

Apache spark - Architecture , Overview & libraries

Walaa Hamdy Assy

Big Data Processing with .NET and Spark (SQLBits 2020)

Michael Rys

From DataFrames to Tungsten: A Peek into Spark's Future-(Reynold Xin, Databri...

R is the latest language added to Apache Spark, and the SparkR API is slightly different from PySpark. SparkR’s evolving interface to Apache Spark offers a wide range of APIs and capabilities to Data Scientists and Statisticians. With the release of Spark 2.0, and subsequent releases, the R API officially supports executing user code on distributed data. This is done primarily through a family of apply() functions. In this Data Science Central webinar, we will explore the following: ●Provide an overview of this new functionality in SparkR. ●Show how to use this API with some changes to regular code with dapply(). ●Focus on how to correctly use this API to parallelize existing R packages. ●Consider performance and examine correctness when using the apply family of functions in SparkR. Speaker: Hossein Falaki, Software Engineer -- Databricks Inc.

Parallelize R Code Using Apache Spark

We are a company driven by inquisitive data scientists, having developed a pragmatic and interdisciplinary approach, which has evolved over the decades working with over 100 clients across multiple industries. Combining several Data Science techniques from statistics, machine learning, deep learning, decision science, cognitive science, and business intelligence, with our ecosystem of technology platforms, we have produced unprecedented solutions. Welcome to the Data Science Analytics team that can do it all, from architecture to algorithms. Our practice delivers data driven solutions, including Descriptive Analytics, Diagnostic Analytics, Predictive Analytics, and Prescriptive Analytics. We employ a number of technologies in the area of Big Data and Advanced Analytics such as DataStax (Cassandra), Databricks (Spark), Cloudera, Hortonworks, MapR, R, SAS, Matlab, SPSS and Advanced Data Visualizations. This presentation is designed for Spark Enthusiasts to get started and details of the course are below. 1. Introduction to Apache Spark 2. Functional Programming + Scala 3. Spark Core 4. Spark SQL + Parquet 5. Advanced Libraries 6. Tips & Tricks 7. Where do I go from here?

Introduction to Spark - DataFactZ

DataFactZ

This presentation about Apache Spark covers all the basics that a beginner needs to know to get started with Spark. It covers the history of Apache Spark, what is Spark, the difference between Hadoop and Spark. You will learn the different components in Spark, and how Spark works with the help of architecture. You will understand the different cluster managers on which Spark can run. Finally, you will see the various applications of Spark and a use case on Conviva. Now, let's get started with what is Apache Spark. Below topics are explained in this Spark presentation: 1. History of Spark 2. What is Spark 3. Hadoop vs Spark 4. Components of Apache Spark 5. Spark architecture 6. Applications of Spark 7. Spark usecase What is this Big Data Hadoop training course about? The Big Data Hadoop and Spark developer course have been designed to impart an in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab. What are the course objectives? Simplilearn’s Apache Spark and Scala certification training are designed to: 1. Advance your expertise in the Big Data Hadoop Ecosystem 2. Help you master essential Apache and Spark skills, such as Spark Streaming, Spark SQL, machine learning programming, GraphX programming and Shell Scripting Spark 3. Help you land a Hadoop developer job requiring Apache Spark expertise by giving you a real-life industry project coupled with 30 demos What skills will you learn? By completing this Apache Spark and Scala course you will be able to: 1. Understand the limitations of MapReduce and the role of Spark in overcoming these limitations 2. Understand the fundamentals of the Scala programming language and its features 3. Explain and master the process of installing Spark as a standalone cluster 4. Develop expertise in using Resilient Distributed Datasets (RDD) for creating applications in Spark 5. Master Structured Query Language (SQL) using SparkSQL 6. Gain a thorough understanding of Spark streaming features 7. Master and describe the features of Spark ML programming and GraphX programming Who should take this Scala course? 1. Professionals aspiring for a career in the field of real-time big data analytics 2. Analytics professionals 3. Research professionals 4. IT developers and testers 5. Data scientists 6. BI and reporting professionals 7. Students who wish to gain a thorough understanding of Apache Spark Learn more at https://www.simplilearn.com/big-data-and-analytics/apache-spark-scala-certification-training

What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...

Simplilearn

Apache spark-melbourne-april-2015-meetup

Ned Shawa

R is the latest language added to Apache Spark, and the SparkR API is slightly different from PySpark. With the release of Spark 2.0, the R API officially supports executing user code on distributed data. This is done through a family of apply() functions. In this talk, Hossein Falaki gives an overview of this new functionality in SparkR. Using this API requires some changes to regular code with dapply(). This talk will focus on how to correctly use this API to parallelize existing R packages. Most important topics of consideration will be performance and correctness when using the apply family of functions in SparkR. Speaker: Hossein Falaki This talk was originally presented at Spark Summit East 2017.

Parallelizing Existing R Packages with SparkR

In this talk you will learn how to easily configure Apache Arrow with R on Apache Spark, which will allow you to gain speed improvements and expand the scope of your data science workflows; for instance, by enabling data to be efficiently transferred between your local environment and Apache Spark. This talk will present use cases for running R at scale on Apache Spark. It will also introduce the Apache Arrow project and recent developments that enable running R with Apache Arrow on Apache Spark to significantly improve performance and efficiency. We will end this talk by discussing performance and recent development in this space. Author: Javier Luraschi

Apache spark basics

sparrowAnalytics.com

Apache Spark Introduction.pdf

MaheshPandit16

TDWI Accelerate, Seattle, Oct 16, 2017: Distributed and In-Database Analytics...

Debraj GuhaThakurta

TWDI Accelerate Seattle, Oct 16, 2017: Distributed and In-Database Analytics ...

Debraj GuhaThakurta

Apache Spark Fundamentals Training

Eren Avşaroğulları

Running R at Scale with Apache Arrow on Spark

R is the favorite language of many data scientists. In addition to a language and runtime, R is a rich ecosystem of libraries for a wide range of use cases from statistical inference to data visualization. However, handling large or distributed data with R is challenging. Hence R is used along with other frameworks and languages by most data scientist. In this mode most of the friction is at the interface of R and the other systems. For example, when data is sampled by a big data platform, results need to be transferred to and imported in R as native data structures. In this talk we show an alternative, and complimentary, approach to SparkR for integrating Spark and R. Since SparkR was released in version 1.4 of Apache Spark distributed data remains inside the JVM instead of individual R processes running on workers. This approach is more convenient when dealing with external data sources such as Cassandra, Hive, and Spark’s own distributed DataFrames. We show two specific techniques to remove the data transfer friction between R and JVM: collecting Spark DataFrames as R data frames and user space filesystems. We think this model complements and improves the day-to-day workload of many data scientists who use R. Spark’s interactive query processing, especially with in-memory datasets, closely matches the R interactive session model. When integrated together Spark and R can provide state of the art tools for the entire end-to-end data science pipeline. We will show how such a pipeline works in real world use cases in a live demo at the end of the talk.

Big data analysis using spark r published

Dipendra Kusi

Apache spark sneha challa- google pittsburgh-aug 25th

Sneha Challa

Strata NYC 2015 - Supercharging R with Apache Spark

Spark Summit EU 2015: Reynold Xin Keynote

Apache Spark An Overview

Mohit Jain

Semelhante a SparkR: Enabling Interactive Data Science at Scale (20)

Parallelizing Existing R Packages

Apache spark - Architecture , Overview & libraries

Big Data Processing with .NET and Spark (SQLBits 2020)

From DataFrames to Tungsten: A Peek into Spark's Future-(Reynold Xin, Databri...

Parallelize R Code Using Apache Spark

Introduction to Spark - DataFactZ

What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...

Apache spark-melbourne-april-2015-meetup

Parallelizing Existing R Packages with SparkR

Apache spark basics

Apache Spark Introduction.pdf

TDWI Accelerate, Seattle, Oct 16, 2017: Distributed and In-Database Analytics...

TWDI Accelerate Seattle, Oct 16, 2017: Distributed and In-Database Analytics ...

Apache Spark Fundamentals Training

Running R at Scale with Apache Arrow on Spark

Big data analysis using spark r published

Apache spark sneha challa- google pittsburgh-aug 25th

Strata NYC 2015 - Supercharging R with Apache Spark

Spark Summit EU 2015: Reynold Xin Keynote

Apache Spark An Overview

Mais de jeykottalam

AMP Camp 5 Intro

Intro to Spark and Spark SQL

Concurrency Control for Parallel Machine Learning

MLlib: Spark's Machine Learning Library

SampleClean: Bringing Data Cleaning into the BDAS Stack

Machine Learning Pipelines

COCOA: Communication-Efficient Coordinate Ascent

The BDAS Open Source Community