Unlocking Value in Device Data Using Spark: Spark Summit East talk by John Landry

•

2 gostaram•1,315 visualizações

HP ships millions of PCs, Printers, and other devices every year to customers in all market segments. More customers are seeking services provided with our products enabling new opportunities for HP to create services from the data we can collect from our devices. Every device we ship is an IoT endpoint with powerful CPU to capture rich data. Insights from this data are used internally to improve our products and focus on customer needs. In this presentation, John will focus on HP’s journey to enabling Big Data analytics from within a large enterprise environment. He will review the challenges and how HP decided on AWS, Apache Spark and Databricks as the foundation for their entry into Big Data Analytics. John will also review how HP uses Spark to build analytic services from the data they generate from their devices.

Dados e análise

HP Big Data
Unlocking Value in Device Data using Spark
John Landry - February, 2017
1

Agenda
2
HP Inc. History1
HP Big Data Opportunities and
Pain2
HP Big Data Platform3
HP Big Data enabled Services4
HP and Big Data 20175

History of HP Inc.
3
1939 2002 2015
HP Inc.

Business Opportunities
4
Product
Improvement
Cost
Avoidance
User
Profiling
Business
Optimization
Customer
Services
Big Data
HP Touchpoint Manager
HP Workspace
HP DAAS
HP Jet Admin
HP Instant Ink
HP 3D Print

Big Data Pain Points for HP Business Groups
HP 2014 - 2016 Flexibility
Repeatability
Scalability
Accessibility
Velocity
Performance
Latency
Geolocation
New Data Metadata
Automation Security

Biggest Challenge: COST $$$$
HP company split – November 2015

“What we have is a data glut.”
Vernor Vinge, professor of mathematics,computer scientist,and science fiction
author
7

HP Big Data Platform
HP Big Data Platform
HP Global Function Data
HP Security and Privacy Governance
Redshift
Athena
Kinesis
Quick sights
Tableau
Qlik
Cost Avoidance
Improving Products
Customer Profiling
Optimize Business
New Service Offerings
Personal Systems
Print
3D
Designed for Analytics
Accumulate Analyze Act

Why …
9
Adhoc Analytics
Business Analytic Requests
Analytics POC
Crowd Sourcing
S3
Secure Data Access
NotebooksCollaboration
Scala, Python, R, SQL
Interactive
Standard Container
GitHub Integration
Secure Usage
Dev OPs
Development ProductMonitoring
- Device Management
- Automated Support
- Device As A Service
- Business Optimization
Services
ProductMonitoring
- Quality Alerts
- Usage Profiling
- User Feedback
- Business Optimization

Example HP Big Data Instance
10
Ingestion
(Kinesis)
Services
DB
(Redshift)
Printers
Data
Repository
(S3)
Other Data
HP Data
PC Devices
Ingest
Parquet
AnalyticsServices
Streaming
Analytics
MessagingQ
Events
HP Services
Managed Services, Customers, Partners, Internal
Data
Science
Spark
DEV
DEV SQL
HPAPI Host
Redshift, Athena, QuickSights

HP Big Data enabled Services
11
Smart Device Service
www.HP.com
Instant Ink
Workspace

HP Big Data Platform – Key points
12
MillionsofDevices
Internet ofThings
New Data Sources
HP Operational
Geolocation
Secure
Unlimited
Accessible
LeadingTechnology
Common Platform
Scalable
Elastic
Affordable
Open Source
Built on Services
Growth
Prescriptive
Predictive
Descriptive
Automated
Services
Flexible
Fast
Accumulate ActAnalyze

HP Big Data 2017
13
HP is invested into Big Dataanalyticsto better serve our
customerswith betterproducts and new services
HP has strategicallyselected Spark with Databricks as
our Big Data platform
HP has partnered with Databricks to reduce the timeto
market anddevelopmentcosts of delivering a Spark
solution for Big Data

Mais conteúdo relacionado

Mais procurados

MZ is re-inventing how the entire world experiences data via our mobile games division MZ Games Studios, our digital marketing division Cognant, and our live data platform division Satori. Growing need of data science capabilities across the organization requires an architecture that can democratize building these applications and disseminating insight from the outcome of data science applications to the wider organization. Attend this session to learn about how we built a platform for data science using spark, hive, and druid specifically for our performance marketing division cognant.This platform powers several data science application like fraud detection and bid optimization at large scale. We will be sharing lessons learned over past 3 years in building this platform by also walking through some of the actual data science applications built on top of this platform. Attendees from ML engineering and data science background can gain deep insight from our experience of building this platform. Speakers Pushkar Priyadarshi, Director of Engineer, Michaine Zone Inc Igor Yurinok, Staff Software Engineer, MZ

Democratizing data science Using spark, hive and druid

DataWorks Summit

Spark and Hadoop at Production Scale-(Anil Gadre, MapR)

Spark Summit

Spark Summit Keynote by Suren Nathan

Spark Summit

"In the oil and gas industry, utilizing vast amounts of data has long been identified as an important indicator of operational performance. The measurement of key performance indicators is a routine practice in well construction, but a systematic way of statistically analyzing performance against a large data bank of offset wells is not a common practice. The performance of statistical analysis in real-time is even less common. With the adoption of distributed computing platforms, like Apache Spark, new analysis opportunities become available to leverage large-scale time-series data sets to optimize performance. Two case studies are presented in this talk: the rate of penetration (ROP) and the amount of vibration per run. By collecting real-time, telemetry data and comparing it with historic sample datasets within the Databricks Unified Analytics Platform, the optimization team was able to quickly determine whether the performance being delivered matched or exceeded past performance with statistical certainty. This is extremely important while trying new techniques with data that is highly variable. By substituting anecdotal evidence with statistical analysis, decision making is more precise and better informed. In this talk we'll share how we accomplished this and the lessons learned along the way."

Unifying Streaming and Historical Telemetry Data For Real-time Performance Re...

Databricks

Spark Summit East Keynote by Anjul Bhambhri

Jen Aman

2015 was a year of continued growth for Spark, with numerous additions to the core project and very fast growth of use cases across the industry. In this talk, I’ll look back at how the Spark community is has grown and changed in 2015, based on a large Apache Spark user survey conducted by Databricks. We see some interesting trends in the diversity of runtime environments (which are increasingly not just Hadoop); the types of applications run on Spark; and the types of users, now that features like R support and DataFrames are available in Spark. I’ll also cover the ongoing work in the upcoming releases of Spark to support new use cases.

Spark Summit EU 2015: Matei Zaharia keynote

Databricks

Zalando SE is Europe’s leading online fashion platform and connects customers, brands and partners. With millions of visitors each month, we have petabytes of purchase, click-stream, product and other data in our data lake. This data is crucial to powering insights on shopper behavior and driving an AI-first strategy to improve site engagement. Over 7 months ago, Zalando adopted Apache Spark, Delta Lake and Databricks as its de-facto computation platform for analytics and machine learning. During this period, we onboarded well over 50 internal teams ranging from BI teams, with no knowledge of Spark or big data running ETL pipelines to AI/ML teams already using EMR and Spark for heavy model training. Provided the spectrum of varied business problems they were trying to solve, we worked with each team individually, understanding their use cases, helping them validate assumptions, developing working code and taking them to production. In this talk we will share best practices for building a unified data and analytics architecture on Databricks, lessons learned rolling it out across the organization and provide a deep dive on AI & Analytics use cases in the fashion ecommerce space.

Building an AI-Powered Retail Experience with Delta Lake, Spark, and Databricks

Databricks

Spark Summit EU talk by Pat Patterson

Spark Summit

Building a Just in Time Data Warehouse by Dan Morris and Jason Pohl

Spark Summit

"Most data practitioners grapple with data quality issues and data pipeline complexities—it's the bane of their existence. Data engineers, in particular, strive to design and deploy robust data pipelines that serve reliable data in a performant manner so that their organizations can make the most of their valuable corporate data assets. Databricks Delta, part of Databricks Runtime, is a next-generation unified analytics engine built on top of Apache Spark. Built on open standards, Delta employs co-designed compute and storage and is compatible with Spark API’s. It powers high data reliability and query performance to support big data use cases, from batch and streaming ingests, fast interactive queries to machine learning. In this tutorial we will discuss the requirements of modern data pipelines, the challenges data engineers face when it comes to data reliability and performance and how Delta can help. Through presentation, code examples and notebooks, we will explain pipeline challenges and the use of Delta to address them. You will walk away with an understanding of how you can apply this innovation to your data architecture and the benefits you can gain. This tutorial will be both instructor-led and hands-on interactive session. Instructions in how to get tutorial materials will be covered in class. WHAT YOU’LL LEARN: – Understand the key data reliability and performance data pipelines challenges – How Databricks Delta helps build robust pipelines at scale – Understand how Delta fits within an Apache Spark™ environment – How to use Delta to realize data reliability improvements – How to deliver performance gains using Delta PREREQUISITES: – A fully-charged laptop (8-16GB memory) with Chrome or Firefox – Pre-register for Databricks Community Edition" Speakers: Steven Yu, Burak Yavuz

Building Robust Production Data Pipelines with Databricks Delta

Databricks

GE Digital is undertaking a journey to optimize the reliability, availability, and efficiency of assets in the industrial sector and converge IT and OT. To do so, GE Digital is building cloud-based products that enable customers to analyze the asset data, detect anomalies, and provide recommendations for operating plants efficiently while increasing productivity. In a energy sector such as oil and gas, power, or renewables, a single plant comprises multiple complex assets, such as steam turbines, gas turbines, and compressors, to generate power. Each system contains various sensors to detect the operating conditions of the assets, generating large volumes of variety of data. A highly scalable distributed environment is required to analyze such a large volume of data and provide operating insights in near real time. In this session I will share the challenges encountered when analyzing the large volumes of data, in-stream data analysis and how we standardized the industrial data based on data frames, and performance tuning.

Optimizing industrial operations using the big data ecosystem

DataWorks Summit

Lambda architecture for real time big data

Trieu Nguyen

Building Data Lakes with Apache Airflow

Gary Stafford

Workday Prism Analytics enables data discovery and interactive Business Intelligence analysis for Workday customers. Workday is a “pure SaaS” company, providing a suite of Financial and HCM (Human Capital Management) apps to about 2000 companies around the world, including more than 30% from Fortune-500 list. There are significant business and technical challenges to support millions of concurrent users and hundreds of millions daily transactions. Using memory-centric graph-based architecture allowed to overcome most of these problems. As Workday grew, data transactions from existing and new customers generated vast amounts of valuable and highly sensitive data. The next big challenge was to provide in-app analytics platform, which for the multiple types of accumulated data, and also would allow using blend in external datasets. Workday users wanted it to be super-fast, but also intuitive and easy-to-use both for the financial and HR analysts and for regular, less technical users. Existing backend technologies were not a good fit, so we turned to Apache Spark. In this presentation, we will share the lessons we learned when building highly scalable multi-tenant analytics service for transactional data. We will start with the big picture and business requirements. Then describe the architecture with batch and interactive modules for data preparation, publishing, and query engine, noting the relevant Spark technologies. Then we will dive into the internals of Prism’s Query Engine, focusing on Spark SQL, DataFrames and Catalyst compiler features used. We will describe the issues we encountered while compiling and executing complex pipelines and queries, and how we use caching, sampling, and query compilation techniques to support interactive user experience. Finally, we will share the future challenges for 2018 and beyond.

Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...

Databricks

Upwork has the biggest closed-loop online dataset of jobs and job seekers in labor history (>10M Profiles, >100M Job Posts, Job Proposals and Hiring Decisions, >10B of Messages, Transaction and Feedback Data). Besides sheer quantity, our data is also contextually very rich. We have client and contractor data for the entire job-funnel – from finding jobs to getting the job done. For various machine learning applications including search and recommendations and labor marketplace optimization (rate, supply and demand), we heavily relied on a Greenplum-based data warehouse solution for data processing and ad-hoc ML pipelines (weka, scikit-learn, R) for offline model development and online model scoring. In this talk, we present our modernization efforts in moving towards a 1) holistic data processing infrastructure for batch and stream data processing using S3, Kinesis, Spark and Spark Structured Streaming 2) model development using Spark MLlib and other ML libraries for Spark 3) model serving using Databricks Model Scoring, Scoring over Structured Streams and microservices and 3) how we orchestrate and streamline all these processes using Apache Airflow and a CI/CD workflow customized to our Data Science product engineering needs. The focus of this talk is on how we were able to leverage the Databricks service offering to reduce DevOps overhead and costs, complete the entire modernization with moderate efforts and adopt a collaborative notebook-based solution for all our data scientists to develop model, reuse features and share results. We will shared the core lessons learned and pitfalls we encountered during this journey.

How to Rebuild an End-to-End ML Pipeline with Databricks and Upwork with Than...

Databricks

Operationalizing Machine Learning at Scale at Starbucks

Databricks

Life occurs in real-time, and not surprisingly, more solutions are being built using streaming technologies. Event-based architectures are becoming the norm, and customers are expecting immediate access to their data. This new world offers many exciting opportunities, but also some new challenges. What do you do when your streaming data is not complete? What if it relies on another data source? Does the dependent data exist yet, and does it come from a 3rd party? How do we merge a complete picture of data when data is sourcing from multiple places at the same time? A new norm in the world of distributed services. Join us as we dive deep into the technical details around these scenarios and more. Expect to learn about stream-stream joins, enriching stream data using local or remote data, and ways to anticipate and correct errors within the stream. Leave with a better understanding of managing data dependencies within a Spark Structured Streaming application.

Life is but a Stream

Databricks

How did eBay move their ETL computation from conventional RDBMS environment over to Spark? What did it take to go from a strategic vision to a viable solution? This paper will take you through a journey which lead to an implementation of a 1000+ node Spark Cluster running 10,000+ ETL jobs daily, all done in a span of less than 6 months, by a team with limited Spark experience. We will share the vision, technical architecture, critical Management decisions, Challenges and Road ahead. This will be a unique opportunity to look into this awesome Spark success story at eBay!

Moving eBay’s Data Warehouse Over to Apache Spark – Spark as Core ETL Platfor...

Databricks

Cloud Data Services: A Brand New Ballgame for Business

IBM Cloud Data Services

Feeling the need to contribute something to Apache Cassandra? Maybe you want to help guide the future of your favorite database? Get off the sidelines and get in the game! It's easy to say but how do you even get started? I will outline some of the ways you can help contribute to Apache Cassandra from minor to major. If you don't have the time or ability to submit code, there are a lot of ways you can participate. What if you do want to write some code? I can walk you through the process of creating a patch and submitting for final approval. Got a great idea? I'll show you propose that to the community at large. Take it from me, participating is so much more fun than just watching the project from a distance. Time to jump in! About the Speaker Patrick McFadin Chief Evangelist, DataStax Patrick McFadin is one of the leading experts of Apache Cassandra and data modeling techniques. As the Chief Evangelist for Apache Cassandra and consultant for DataStax, he has helped build some of the largest and exciting deployments in production. Previous to DataStax, he was Chief Architect at Hobsons and an Oracle DBA/Developer for over 15 years.

Spectator to Participant. Contributing to Cassandra (Patrick McFadin, DataSta...

DataStax

Mais procurados (20)

Democratizing data science Using spark, hive and druid

Spark and Hadoop at Production Scale-(Anil Gadre, MapR)

Spark Summit Keynote by Suren Nathan

Unifying Streaming and Historical Telemetry Data For Real-time Performance Re...

Spark Summit East Keynote by Anjul Bhambhri

Spark Summit EU 2015: Matei Zaharia keynote

Building an AI-Powered Retail Experience with Delta Lake, Spark, and Databricks

Spark Summit EU talk by Pat Patterson

Building a Just in Time Data Warehouse by Dan Morris and Jason Pohl

Building Robust Production Data Pipelines with Databricks Delta

Optimizing industrial operations using the big data ecosystem

Lambda architecture for real time big data

Building Data Lakes with Apache Airflow

Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...

How to Rebuild an End-to-End ML Pipeline with Databricks and Upwork with Than...

Operationalizing Machine Learning at Scale at Starbucks

Life is but a Stream

Moving eBay’s Data Warehouse Over to Apache Spark – Spark as Core ETL Platfor...

Cloud Data Services: A Brand New Ballgame for Business

Spectator to Participant. Contributing to Cassandra (Patrick McFadin, DataSta...

Destaque

In 2017, 60% of the US population will be a digital banking user. The challenges to meet the demands of a more engaged customer is increasing as the banking experience becomes less formal and easier to deliver. Building a better customer experience starts with how you handle your data to leverage accountable, actionable analytics. To take this on, FIS had to overcome the challenge posed by increasing data volumes and data velocity, enterprise level operational complexity and requirements, and work with antiquated traditional techniques. Within a year we transformed to Apache Spark and Databricks, providing thousands of financial institutions the ability to build better relationships with their customers by understanding the behaviors and interactions in digital banking.

FIS: Accelerating Digital Intelligence in FinTech: Spark Summit East talk by...

Spark Summit

Spark 2.0 provided strong performance enhancements to the Spark core while advancing Spark ML usability to use data frames. But what happens when you run Spark 2.0 machine learning algorithms on a large cluster with a very large data set? Do you even get any benefit from using a very large data set? It depends.  How do new hardware advances affect the topology of high performance Spark clusters. In this talk we will explore Spark 2.0 Machine Learning at scale and share our findings with the community. As our test platform we will be using a new cluster design, different from typical Hadoop clusters, with more cores, more RAM and latest generation NVMe SSD’s and a 100GbE network with a goal of more performance, in a more space and energy efficient footprint.

Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...

Spark Summit

Scalable Data Science with SparkR: Spark Summit East talk by Felix Cheung

Spark Summit

Moving at the speed of a startup often means rapid iterative development, which can lead to a patchwork of systems and processes. In the early days at Kik (one of the most popular chat apps among U.S. teens), the data team was able to move extremely quickly but often at the expense of scalable data engineering. In this session, Kik’s head of data will share the eight things they did to save time and money. The team took their data stack from a complex combination of systems and processes to a scalable, simple, and robust platform leveraging Apache Spark and Databricks to make data super easy for everyone in the company to use.

Scaling Through Simplicity—How a 300 million User Chat App Reduced Data Engin...

Spark Summit

This talk will cover the tools we used, the hurdles we faced and the work arounds we developed with the help from Databricks support in our attempt to build a custom machine learning model and use it to predict the TV ratings for different networks and demographics. The Apache Spark machine learning and dataframe APIs make it incredibly easy to produce a machine learning pipeline to solve an archetypal supervised learning problem. In our applications at Cadent, we face a challenge with high dimensional labels and relatively low dimensional features; at first pass such a problem is all but intractable but thanks to a large number of historical records and the tools available in Apache Spark, we were able to construct a multi-stage model capable of forecasting with sufficient accuracy to drive the business application. Over the course of our work we have come across many tools that made our lives easier, and others that forced work around. In this talk we will review our custom multi-stage methodology, review the challenges we faced and walk through the key steps that made our project successful.

Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...

Spark Summit

Apache Spark MLlib provides scalable implementation of popular machine learning algorithms, which lets users train models from big dataset and iterate fast. The existing implementations assume that the number of parameters is small enough to fit in the memory of a single machine. However, many applications require solving problems with billions of parameters on a huge amount of data such as Ads CTR prediction and deep neural network. This requirement far exceeds the capacity of exisiting MLlib algorithms many of who use L-BFGS as the underlying solver. In order to fill this gap, we developed Vector-free L-BFGS for MLlib. It can solve optimization problems with billions of parameters in the Spark SQL framework where the training data are often generated. The algorithm scales very well and enables a variety of MLlib algorithms to handle a massive number of parameters over large datasets. In this talk, we will illustrate the power of Vector-free L-BFGS via logistic regression with real-world dataset and requirement. We will also discuss how this approach could be applied to other ML algorithms.

Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...

Spark Summit

The demand for stream processing is increasing a lot these days. Immense amounts of data have to be processed fast from a rapidly growing set of disparate data sources. This pushes the limits of traditional data processing infrastructures. These stream-based applications include trading, social networks, Internet of things, system monitoring, and many other examples. A number of powerful, easy-to-use open source platforms have emerged to address this. But the same problem can be solved differently, various but sometimes overlapping use-cases can be targeted or different vocabularies for similar concepts can be used. This may lead to confusion, longer development time or costly wrong decisions.

Distributed Real-Time Stream Processing: Why and How: Spark Summit East talk ...

Spark Summit

Processing real-time analytics of big data streams from sensor data will continue to be an important task as embedded technology increases and we continue to generate new types and ways of data analysis, particularly in regard to the Internet of Things (IoT). Robotics models many of these key challenges well and incorporates the possibility of high- throughput streams as well as complex online machine learning and analytics algorithms. These challenges make it an almost ideal candidate for in depth analysis of real-time streaming analytics. We look at a simultaneous localization and mapping (SLAM) problem, an ongoing research area in robotics for autonomous vehicles, and well recognized as a non-trivial problem space in both industry and research. We will use a new integrated framework on Kafka and Spark Streaming to explore a constrained SLAM problem using online algorithms to navigate and map a space in real time. We present benchmarks of our open-source robot’s integration with Kafka and Spark Streaming for performance against other SLAM algorithms currently in use, explore some of the challenges we faced in our implementation, and make recommendations for improvement of performance and optimization on our framework. Finally, new to this talk, we demo real-time usage of our implementation with the Turtlebot II and explore relevant benchmarks and their implications on the future of autonomous vehicles in the IoT and cloud analytics space.

IoT and the Autonomous Vehicle in the Clouds: Simultaneous Localization and M...

Spark Summit

Reinsurance company’s core competencies include the quantification of risk associated with catastrophes, such as hurricanes and earthquakes. Various so-called catastrophe models are available publicly, some commercial and some open-source. The volume of data processed by such “cat models” requires Big Data and High Performance capabilities. This is clearly reflected in the landscape of public models. And the observed trend is towards more and more detailed inputs, as well as outputs. This makes scalability an important concern. Companies that deal with catastrophe risk commonly use one or several public cat models. If they wish to differentiate themselves from the market, they may build internal proprietary models, in particular in areas that are not covered by existing models. The result is a deeper understanding and an independent quantification of risk, both of which can lead to a competitive edge.

Modeling Catastrophic Events in Spark: Spark Summit East Talk by Georg Hofman...

Spark Summit

Algorithms for sketching probability distributions from large data sets are a fundamental building block of modern data science. Sketching plays a role in diverse applications ranging from visualization, optimizing data encodings, estimating quantiles, data synthesis and imputation. The T-Digest is a versatile sketching data structure. It operates on any numeric data, models tricky distribution tails with high fidelity, and most crucially it works smoothly with aggregators and map-reduce. T-Digest is a perfect fit for Apache Spark; it is single-pass and intermediate results can be aggregated across partitions in batch jobs or aggregated across windows in streaming jobs. In this talk I will describe a native Scala implementation of the T-Digest sketching algorithm and demonstrate its use in Spark applications for visualization, quantile estimations and data synthesis. Attendees of this talk will leave with an understanding of data sketching with T-Digest sketches, and insights about how to apply T-Digest to their own data analysis applications.

Sketching Data with T-Digest In Apache Spark: Spark Summit East talk by Erik ...

Spark Summit

R is a hugely popular platform for Data Scientists to create analytic models in many different domains. But when these applications should move from the science lab to the production environment of large enterprises a new set of challenges arises. Independently of R, Spark has been very successful as a powerful general-purpose computing platform. With the introduction of SparkR an exciting new option to productionize Data Science applications has been made available. This talk will give insight into two real-life projects at major enterprises where Data Science applications in R have been migrated to SparkR. • Dealing with platform challenges: R was not installed on the cluster. We show how to execute SparkR on a Yarn cluster with a dynamic deployment of R. • Integrating Data Engineering and Data Science: we highlight the technical and cultural challenges that arise from closely integrating these two different areas. • Separation of concerns: we describe how to disentangle ETL and data preparation from analytic computing and statistical methods. • Scaling R with SparkR: we present what options SparkR offers to scale R applications and how we applied them to different areas such as time series forecasting and web analytics. • Performance Improvements: we will show benchmarks for an R applications that took over 20 hours on a single server/single-threaded setup. With moderate effort we have been able to reduce that number to 15 minutes with SparkR. And we will show how we plan to further reduces this to less than a minute in the future. • Mixing SparkR, SparkSQL and MLlib: we show how we combined the three different libraries to maximize efficiency. • Summary and Outlook: we describe what we have learnt so far, what the biggest gaps currently are and what challenges we expect to solve in the short- to mid-term.

Using SparkR to Scale Data Science Applications in Production. Lessons from t...

Spark Summit

The central premise of DataXu is to apply data science to better marketing. At its core, is the Real Time Bidding Platform that processes 2 Petabytes of data per day and responds to ad auctions at a rate of 2.1 million requests per second across 5 different continents. Serving on top of this platform is Dataxu’s analytics engine that gives their clients insightful analytics reports addressed towards client marketing business questions. Some common requirements for both these platforms are the ability to do real-time processing, scalable machine learning, and ad-hoc analytics. This talk will showcase DataXu’s successful use-cases of using the Apache Spark framework and Databricks to address all of the above challenges while maintaining its agility and rapid prototyping strengths to take a product from initial R&D phase to full production. The team will share their best practices and highlight the steps of large scale Spark ETL processing, model testing, all the way through to interactive analytics.

R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk ...

Spark Summit

Spark SQL and Mllib are optimized for running feature extraction and machine learning algorithms on row based columnar datasets through full scan but does not provide constructs for column indexing and time series analysis. For dealing with document datasets with timestamps where the features are represented as variable number of columns in each document and use-cases demand searching over columns and time to retrieve documents to generate learning models in realtime, a close integration within Spark and Lucene was needed. We introduced LuceneDAO in Spark Summit Europe 2016 to build distributed lucene shards from data frame but the time series attributes were not part of the data model. In this talk we present our extension to LuceneDAO to maintain time stamps with document-term view for search and allow time filters. Lucene shards maintain the time aware document-term view for search and vector space representation for machine learning pipelines. We used Spark as our distributed query processing engine where each query is represented as boolean combination over terms with filters on time. LuceneDAO is used to load the shards to Spark executors and power sub-second distributed document retrieval for the queries. Our synchronous API uses Spark-as-a-Service to power analytical queries while our asynchronous API uses kafka, spark streaming and HBase to power time series prediction algorithms. In this talk we will demonstrate LuceneDAO write and read performance on millions of documents with 1M+ terms and configurable time stamp aggregate columns. We will demonstrate the latency of APIs on a suite of queries generated from terms. Key takeaways from the talk will be a thorough understanding of how to make Lucene powered time aware search a first class citizen in Spark to build interactive analytical query processing and time series prediction algorithms.

Realtime Analytical Query Processing and Predictive Model Building on High Di...

Spark Summit

Powering Predictive Mapping at Scale with Spark, Kafka, and Elastic Search: S...

Spark Summit

In this talk we will introduce the business use case of how we create a real-time platform for our Second Look project using Spark and Kafka. Second Look is a feature created by Capital One to detect and notify cardholders of these potential mistakes and unexpected charges. We bring them to the attention of the customers automatically through email alerts and push notifications to ensure customers can take timely action. The situations can be resolved through a conversation with the merchant, or a dispute on your charge directly to Capital One. We help to guide the user through this resolution path through our user experiences. We use Spark extensively to build the infrastructure for this project. Before we use Spark and Kafka, the alerts were not sent in real-time and there were delays in days between when the customers transact and when customers receive the alerts. With the power of Spark and Kafka, we are able to send the alert in a more timely manner. We will share how we connect each parts together from data ingestion to processing, alert generation, and alert delivery. We will demonstrate how Spark plays critical role in the whole infrastructure. What’s next? We will leverage more power of machine learning using Spark to generate various types of alerts.

Real-time Platform for Second Look Business Use Case Using Spark and Kafka: S...

Spark Summit

A web crawler is a bot program that fetches resources from the web for the sake of building applications like search engines, knowledge bases, etc. In this talk, Karanjeet Singh and Thamme Gowda will describe a new crawler called Sparkler (contraction of Spark-Crawler) that makes use of recent advancements in distributed computing and information retrieval domains by conglomerating various Apache projects like Spark, Kafka, Lucene/Solr, Tika, and Felix. Sparkler is extensible, highly scalable, and high-performance web crawler that is an evolution of Apache Nutch and runs on Apache Spark Cluster. https://github.com/USCDataScience/sparkler

Sparkler—Crawler on Apache Spark: Spark Summit East talk by Karanjeet Singh a...

Spark Summit

One of the key challenges in working with real-time and streaming data is that the data format for capturing data is not necessarily the optimal format for ad hoc analytic queries. For example, Avro is a convenient and popular serialization service that is great for initially bringing data into HDFS. Avro has native integration with Flume and other tools that make it a good choice for landing data in Hadoop. But columnar file formats, such as Parquet and ORC, are much better optimized for ad hoc queries that aggregate over large number of similar rows.

Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...

Spark Summit

What if you could get the simplicity, convenience, interoperability, and storage niceties of an old-fashioned CSV with the speed of a NoSQL database and the storage requirements of a gzipped file? Enter Parquet. At The Weather Company, Parquet files are a quietly awesome and deeply integral part of our Spark-driven analytics workflow. Using Spark + Parquet, we’ve built a blazing fast, storage-efficient, query-efficient data lake and a suite of tools to accompany it. We will give a technical overview of how Parquet works and how recent improvements from Tungsten enable SparkSQL to take advantage of this design to provide fast queries by overcoming two major bottlenecks of distributed analytics: communication costs (IO bound) and data decoding (CPU bound).

Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...

Spark Summit

As advanced sensor technologies are becoming widely deployed in the energy industry, the availability of higher-frequency data results in both analytical benefits and computational costs. To an energy forecaster or data scientist, some of these benefits might include enhanced predictive performance from forecasting models as well as improved pattern recognition in energy consumption across building types, economic sectors, and geographies. To a utility or electricity service provider, these benefits might include significantly deeper insights into their diverse customer base. However, these advantages can come with a high computational price tag. With Spark 2.0, User-Defined Functions can be applied across grouped SparkDataFrames in the SparkR API to solve the multivariate optimization and model selection problems typically required for fitting site-level models. This recently added feature of Spark 2.0 on Databricks has allowed DNV GL to efficiently fit predictive models that relate weather, electricity, water, and gas consumption across virtually any number of buildings.

High Resolution Energy Modeling that Scales with Apache Spark 2.0 Spark Summi...

Spark Summit

Come explore a feature we’ve created that is not supported out-of-the-box: the ability to add or remove nodes to always-on real time Spark Streaming jobs. Elastic Spark Streaming jobs can automatically adjust to the demands of traffic or volume. Using a set of configurable utility classes, these jobs scale down when lulls are detected and scale up when load is too high. We process multiple TB’s per day with billions of events. Our traffic pattern experiences natural peaks and valleys with the occasional sustained unexpected spike. Elastic jobs has freed us from manual intervention, given back developer time, and has made a large financial impact through maximized resource utilization.

Auto Scaling Systems With Elastic Spark Streaming: Spark Summit East talk by ...

Spark Summit

Destaque (20)