Yahoo spark

•

3 likes•1,200 views

Spark and Hadoop at Yahoo:
Brought to you by YARN

Andy Feng
Yahoo! Hadoop
(afeng@yahoo-inc.com)

Personalized Web

Big-Data in Yahoo!

3

9/10/13

Hadoop + Spark:
Empowered by YARN

30k+ Yahoo! production nodes on YARN since Q1 2013

Shark Pilot: Advertising Data Analytics
§  Business questions
› 

Are two sets of audience cohorts similar to each other?

› 

What audience segment is most likely to be interested in this ad
campaign?

› 

In what way was the new front page rollout different than the
previous front page as far as audience engagement goes?

› 

What are the right metrics to define user engagement?

§  Shark pilot
› 

50 nodes, each w/ 96GB RAM
•  Currently loaded w/ 3.2 TB sample data in memory

› 

Homegrown BI tools for ad-hoc queries
•  Using Shark Server (contributed to community by Yahoo!)

Shark Perf: TCP-H Benchmark
Average
Seconds
600
500
400
300
200
100
0

Spark Pilot: Model Training Pipeline
§  A DAG of M/R jobs in Hadoop Streaming
› 

Feature extraction

› 

Train models

› 

Score and analyze models

§  Initial Spark prototype
› 

3x speedup on feature extraction

§  Production launch
› 

Apply Spark against complete pipeline

› 

Spark on 80 node cluster
•  Thanks to the enhanced UI and metrics in Spark 0.8

7

9/10/13

Use Case: Ad Targeting

Spark

M/R and Storm

8

9/10/13

Use Case: Content Recommendation
w/ Collaborative Filtering
Input

CF Learning

Ranking

Spark

Spark

9

9/10/13

Output

Spark-YARN: Deployment Simplified
run spark.deploy.yarn.Client --jar … --class … --args …
--queue …--num-workers … --worker-memory …

Spark-YARN (contributed by Yahoo!) is being adopted by
community (ex. Taobao) for production use. You should try it
on your Hadoop cluster.
10

9/10/13

Acknowledgement
§  AMPLab team
› 

Outstanding collaboration: Ion, Matei, Reynold, Patrick, Matt, …

§  Yahoo! Hadoop team
› 

Thomas, Bobby, Paul, Rajiv, Mithun, …

§  Yahoo! Lab.
› 

Mridul, Nathan, …

§  Yahoo! data analytics
› 

Supreeth, Ram, Tim, …

§  Yahoo! spark users
› 

Gavin, Jay, Hirakendu, …

11

9/10/13

We Are Hiring!
http://careers.yahoo.com/

More Related Content

What's hot

This presentation discusses the support guidelines for using Oracle Real Application Clusters (RAC) in virtualized environments, for which general Oracle Database support guidelines are discussed shortly first. First presented during DOAG 2021 User Conference, this presentation replaces its predecessor from 2016: https://www.slideshare.net/MarkusMichalewicz/how-to-use-oracle-rac-in-a-cloud-a-support-question

Oracle RAC Virtualized - In VMs, in Containers, On-premises, and in the Cloud

Oracle RAC Virtualized - In VMs, in Containers, On-premises, and in the Cloud

Oracle RAC Virtualized - In VMs, in Containers, On-premises, and in the Cloud

Markus Michalewicz

Nowadays, people are creating, sharing and storing data at a faster pace than ever before, effective data compression / decompression could significantly reduce the cost of data usage. Apache Spark is a general distributed computing engine for big data analytics, and it has large amount of data storing and shuffling across cluster in runtime, the data compression/decompression codecs can impact the end to end application performance in many ways. However, there’s a trade-off between the storage size and compression/decompression throughput (CPU computation). Balancing the data compress speed and ratio is a very interesting topic, particularly while both software algorithms and the CPU instruction set keep evolving. Apache Spark provides a very flexible compression codecs interface with default implementations like GZip, Snappy, LZ4, ZSTD etc. and Intel Big Data Technologies team also implemented more codecs based on latest Intel platform like ISA-L(igzip), LZ4-IPP, Zlib-IPP and ZSTD for Apache Spark; in this session, we’d like to compare the characteristics of those algorithms and implementations, by running different micro workloads as well as end to end workloads, based on different generations of Intel x86 platform and disk. It’s supposedly to be the best practice for big data software engineers to choose the proper compression/decompression codecs for their applications, and we also will present the methodologies of measuring and tuning the performance bottlenecks for typical Apache Spark workloads.

Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...

Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...

Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...

Rds data lake @ Robinhood

Rds data lake @ Robinhood

Rds data lake @ Robinhood

BalajiVaradarajan13

Bucketing is a partitioning technique that can improve performance in certain data transformations by avoiding data shuffling and sorting. The general idea of bucketing is to partition, and optionally sort, the data based on a subset of columns while it is written out (a one-time cost), while making successive reads of the data more performant for downstream jobs if the SQL operators can make use of this property. Bucketing can enable faster joins (i.e. single stage sort merge join), the ability to short circuit in FILTER operation if the file is pre-sorted over the column in a filter predicate, and it supports quick data sampling. In this session, you’ll learn how bucketing is implemented in both Hive and Spark. In particular, Patil will describe the changes in the Catalyst optimizer that enable these optimizations in Spark for various bucketing scenarios. Facebook’s performance tests have shown bucketing to improve Spark performance from 3-5x faster when the optimization is enabled. Many tables at Facebook are sorted and bucketed, and migrating these workloads to Spark have resulted in a 2-3x savings when compared to Hive. You’ll also hear about real-world applications of bucketing, like loading of cumulative tables with daily delta, and the characteristics that can help identify suitable candidate jobs that can benefit from bucketing.

Hive Bucketing in Apache Spark with Tejas Patil

Hive Bucketing in Apache Spark with Tejas Patil

Hive Bucketing in Apache Spark with Tejas Patil

Intro To Hadoop

Intro To Hadoop

Intro To Hadoop

Slides for my talk in Spark Summit 2017 SF : https://spark-summit.org/2017/events/hive-bucketing-in-apache-spark/ Bucketing is a partitioning technique that can improve performance in certain data transformations by avoiding data shuffling and sorting. The general idea of bucketing is to partition, and optionally sort, the data based on a subset of columns while it is written out (a one-time cost), while making successive reads of the data more performant for downstream jobs if the SQL operators can make use of this property. Bucketing can enable faster joins (i.e. single stage sort merge join), the ability to short circuit in FILTER operation if the file is pre-sorted over the column in a filter predicate, and it supports quick data sampling. In this session, you’ll learn how bucketing is implemented in both Hive and Spark. In particular, Patil will describe the changes in the Catalyst optimizer that enable these optimizations in Spark for various bucketing scenarios. Facebook’s performance tests have shown bucketing to improve Spark performance from 3-5x faster when the optimization is enabled. Many tables at Facebook are sorted and bucketed, and migrating these workloads to Spark have resulted in a 2-3x savings when compared to Hive. You’ll also hear about real-world applications of bucketing, like loading of cumulative tables with daily delta, and the characteristics that can help identify suitable candidate jobs that can benefit from bucketing.

Hive Bucketing in Apache Spark

Hive Bucketing in Apache Spark

Hive Bucketing in Apache Spark

We will give a detailed introduction to Apache Spark and why and how Spark can change the analytics world. Apache Spark's memory abstraction is RDD (Resilient Distributed DataSet). One of the key reason why Apache Spark is so different is because of the introduction of RDD. You cannot do anything in Apache Spark without knowing about RDDs. We will give a high level introduction to RDD and in the second half we will have a deep dive into RDDs.

Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

Sachin Aggarwal

Dm from databases perspective u 1

Dm from databases perspective u 1

Dm from databases perspective u 1

Database Basics Theory

Database Basics Theory

Database Basics Theory

sunmitraeducation

Top 5 Considerations for a Big Data Solution

Top 5 Considerations for a Big Data Solution

Top 5 Considerations for a Big Data Solution

Netflix’s playback data records every user interaction with video on the service, from trailers on the home page to full-length movies. This is a critical dataset with high volume that is used broadly across Netflix, powering product experiences, AB test metrics, and offline insights. In processing playback data, we depend heavily on event-time partitioning to handle a long tail of late arriving events. In this talk, I’ll provide an overview of our recent implementation of generic event-time partitioning on high volume streams using Apache Flink and Apache Iceberg (Incubating). Built as configurable Flink components that leverage Iceberg as a new output table format, we are now able to write playback data and other large scale datasets directly from a stream into a table partitioned on event time, replacing the common pattern of relying on a post-processing batch job that “puts the data in the right place”. We’ll talk through what it took to apply this to our playback data in practice, as well as challenges we hit along the way and tradeoffs with a streaming approach to event-time partitioning.

Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...

Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...

Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...

Introduction to Big Data and hadoop

Introduction to Big Data and hadoop

Introduction to Big Data and hadoop

Hadoop cluster configuration

Hadoop cluster configuration

Hadoop cluster configuration

prabakaranbrick

Big Data Analytics with Hadoop

Big Data Analytics with Hadoop

Big Data Analytics with Hadoop

Apache Spark 101

Apache Spark 101

Apache Spark 101

Abdullah Çetin ÇAVDAR

Hadoop Installation presentation

Hadoop Installation presentation

Hadoop Installation presentation

At Spark Summit 2017, we described our framework to migrate production Hive workload to Spark with minimal user intervention. After a year of migration, Spark now powers an important part of our batch processing workload. The migration framework supports syntax compatibility analysis, offline/online shadowing, and data validation. In this session, we first introduce new features and improvements in the migration framework to support bucketed tables and increase automation. Next, we will deep dive into the top technical challenges we encountered and how we addressed them. We improved the the syntax compatibility between Hive and Spark from around 51% to 85% by identifying/developing top missing features, fixing incompatible UDFs, and implementing a UDF testing framework. In addition, we developed reliable join operators to improve Spark stability in production when leveraging optimizations such as ShuffledHashJoin. Finally, we will share an update on our overall migration effort and examples of migrations wins. For example, we were able to migrate one of the most complicated workloads in Facebook from Hive to Spark with more than 2.5X performance gain.

Migrating Apache Hive Workload to Apache Spark: Bridge the Gap with Zhan Zhan...

Migrating Apache Hive Workload to Apache Spark: Bridge the Gap with Zhan Zhan...

Migrating Apache Hive Workload to Apache Spark: Bridge the Gap with Zhan Zhan...

03 spark rdd operations

03 spark rdd operations

03 spark rdd operations

New in Oracle Universal Installer (OUI)

New in Oracle Universal Installer (OUI)

New in Oracle Universal Installer (OUI)

Markus Michalewicz

At Databricks, we have a unique view into over a hundred different companies trying out Spark for development and production use-cases, from their support tickets and forum posts. Having seen so many different workflows and applications, some discernible patterns emerge when looking at common performance and scalability issues that our users run into. This talk will discuss some of these common common issues from an engineering and operations perspective, describing solutions and clarifying misconceptions.

Spark Summit EU 2015: Lessons from 300+ production users

Spark Summit EU 2015: Lessons from 300+ production users

Spark Summit EU 2015: Lessons from 300+ production users

What's hot (20)

Oracle RAC Virtualized - In VMs, in Containers, On-premises, and in the Cloud

Oracle RAC Virtualized - In VMs, in Containers, On-premises, and in the Cloud

Oracle RAC Virtualized - In VMs, in Containers, On-premises, and in the Cloud

Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...

Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...

Best Practice of Compression/Decompression Codes in Apache Spark with Sophia...

Rds data lake @ Robinhood

Rds data lake @ Robinhood

Rds data lake @ Robinhood

Hive Bucketing in Apache Spark with Tejas Patil

Hive Bucketing in Apache Spark with Tejas Patil

Hive Bucketing in Apache Spark with Tejas Patil

Intro To Hadoop

Intro To Hadoop

Intro To Hadoop

Hive Bucketing in Apache Spark

Hive Bucketing in Apache Spark

Hive Bucketing in Apache Spark

Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

Dm from databases perspective u 1

Dm from databases perspective u 1

Dm from databases perspective u 1

Database Basics Theory

Database Basics Theory

Database Basics Theory

Top 5 Considerations for a Big Data Solution

Top 5 Considerations for a Big Data Solution

Top 5 Considerations for a Big Data Solution

Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...

Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...

Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...

Introduction to Big Data and hadoop

Introduction to Big Data and hadoop

Introduction to Big Data and hadoop

Hadoop cluster configuration

Hadoop cluster configuration

Hadoop cluster configuration

Big Data Analytics with Hadoop

Big Data Analytics with Hadoop

Big Data Analytics with Hadoop

Apache Spark 101

Apache Spark 101

Apache Spark 101

Hadoop Installation presentation

Hadoop Installation presentation

Hadoop Installation presentation

Migrating Apache Hive Workload to Apache Spark: Bridge the Gap with Zhan Zhan...

Migrating Apache Hive Workload to Apache Spark: Bridge the Gap with Zhan Zhan...

Migrating Apache Hive Workload to Apache Spark: Bridge the Gap with Zhan Zhan...

03 spark rdd operations

03 spark rdd operations

03 spark rdd operations

New in Oracle Universal Installer (OUI)

New in Oracle Universal Installer (OUI)

New in Oracle Universal Installer (OUI)

Spark Summit EU 2015: Lessons from 300+ production users

Spark Summit EU 2015: Lessons from 300+ production users

Spark Summit EU 2015: Lessons from 300+ production users

Similar to Yahoo spark

Fikrimuhal TRHUG 2016 Machine Learning

Fikrimuhal TRHUG 2016 Machine Learning

Fikrimuhal TRHUG 2016 Machine Learning

Clickstream & Social Media Analysis using Apache Spark

Clickstream & Social Media Analysis using Apache Spark

Clickstream & Social Media Analysis using Apache Spark

TUMRA | Big Data Science - Gain a competitive advantage through Big Data & Data Science

Hadoop @ Yahoo! - Internet Scale Data Processing

Hadoop @ Yahoo! - Internet Scale Data Processing

Hadoop @ Yahoo! - Internet Scale Data Processing

Yahoo Developer Network

These are the slides to support the Apache® Spark™ MLlib: From Quick Start to Scikit-Learn webinar. In this webcast, Joseph Bradley from Databricks will be speaking about Apache Spark’s distributed Machine Learning Library - MLlib. We will start off with a quick primer on machine learning, Spark MLlib, and a quick overview of some Spark machine learning use cases. We will continue with multiple Spark MLlib quick start demos. Afterwards, the talk will transition toward the integration of common data science tools like Python pandas, scikit-learn, and R with MLlib

Apache® Spark™ MLlib: From Quick Start to Scikit-Learn

Apache® Spark™ MLlib: From Quick Start to Scikit-Learn

Apache® Spark™ MLlib: From Quick Start to Scikit-Learn

Ncku csie talk about Spark

Ncku csie talk about Spark

Ncku csie talk about Spark

The Hadoop project is an integral part of Yahoo!'s cloud infrastructure and is at the heart of many of Yahoo!'s important business processes. Sumeet Singh, the Head of Products for Cloud Services and Hadoop at Yahoo!, explains how Yahoo! leverages Hadoop and Cloud Platforms to process and serve Internet- scale data. Yahoo! operates one of the world's largest private cloud infrastructures. Learn how technologies scale out for building enterprise-wide trusted platforms with tight SLAs. URL: http://www.saptechnologyservice.com/track1.html

SAP Technology Services Conference 2013: Big Data and The Cloud at Yahoo!

SAP Technology Services Conference 2013: Big Data and The Cloud at Yahoo!

SAP Technology Services Conference 2013: Big Data and The Cloud at Yahoo!

Hadoop at Yahoo! -- University Talks

Hadoop at Yahoo! -- University Talks

Hadoop at Yahoo! -- University Talks

Big Data Trend and Open Data

Big Data Trend and Open Data

Big Data Trend and Open Data

Capstone Project Slides- Yelper

Capstone Project Slides- Yelper

Capstone Project Slides- Yelper

Databricks Meetup @ Los Angeles Apache Spark User Group

Databricks Meetup @ Los Angeles Apache Spark User Group

Databricks Meetup @ Los Angeles Apache Spark User Group

SDSC18 and DSATL Meetup March 2018

SDSC18 and DSATL Meetup March 2018

SDSC18 and DSATL Meetup March 2018

CareerBuilder.com

Hw09 Hadoop Applications At Yahoo!

Hw09 Hadoop Applications At Yahoo!

Hw09 Hadoop Applications At Yahoo!

Hadoop at Yahoo! -- Hadoop World NY 2009

Hadoop at Yahoo! -- Hadoop World NY 2009

Hadoop at Yahoo! -- Hadoop World NY 2009

In this one-hour webinar, you will be introduced to Spark, the data engineering that supports it, and the data science advances that it has spurned. You’ll discover the interesting story of its academic origins and then get an overview of the organizations who are using the technology. After being briefed on some impressive Spark case studies, you’ll come to know of the next-generation Spark 2.0 (to be released in just a few months). We will also tell you about the tremendous impact that learning Spark can have upon your current salary, and the best ways to get trained in this ground-breaking new technology.

Big Data 2.0 - How Spark technologies are reshaping the world of big data ana...

Big Data 2.0 - How Spark technologies are reshaping the world of big data ana...

Big Data 2.0 - How Spark technologies are reshaping the world of big data ana...

Lillian Pierson

Bobby Evans and Tom Graves, the engineering leads for Spark and Storm development at Yahoo will talk about how these technologies are used on Yahoo's grids and reasons why to use one or the other. Bobby Evans is the low latency data processing architect at Yahoo. He is a PMC member on many Apache projects including Storm, Hadoop, Spark, and Tez. His team is responsible for delivering Storm as a service to all of Yahoo and maintaining Spark on Yarn for Yahoo (Although Tom really does most of that work). Tom Graves a Senior Software Engineer on the Platform team at Yahoo. He is an Apache PMC member on Hadoop, Spark, and Tez. His team is responsible for delivering and maintaining Spark on Yarn for Yahoo.

Yahoo compares Storm and Spark

Yahoo compares Storm and Spark

Yahoo compares Storm and Spark

Chicago Hadoop Users Group

Introduction To Big Data and Use Cases on Hadoop

Introduction To Big Data and Use Cases on Hadoop

Introduction To Big Data and Use Cases on Hadoop

Power Platform Leeds - November 2019 - Microsoft Ignite Announcements

Power Platform Leeds - November 2019 - Microsoft Ignite Announcements

Power Platform Leeds - November 2019 - Microsoft Ignite Announcements

The Future Of SEO/Content Marketing

The Future Of SEO/Content Marketing

The Future Of SEO/Content Marketing

Presenter: Ofer Mendelevitch of Hortonworks > Learn the benefits of big data for data scientists, and how Hadoop and HDInsight fit into the modern data architecture and enable data-driven products. You'll learn: * What data science actually means * The term "data products" * The benefits of using big data for data scientists * How Hadoop helps data scientists work with big data * About HDInsight, the big data platform from Microsoft and Hortonworks

Hortonworks Big Data & Hadoop

Hortonworks Big Data & Hadoop

Hortonworks Big Data & Hadoop

Mark Ginnebaugh

London Spark Meetup 2014-11-11 @Skimlinks http://www.meetup.com/Spark-London/events/217362972/ To paraphrase the immortal crooner Don Ho: "Tiny Batches, in the wine, make me happy, make me feel fine." http://youtu.be/mlCiDEXuxxA Apache Spark provides support for streaming use cases, such as real-time analytics on log files, by leveraging a model called discretized streams (D-Streams). These "micro batch" computations operated on small time intervals, generally from 500 milliseconds up. One major innovation of Spark Streaming is that it leverages a unified engine. In other words, the same business logic can be used across multiple uses cases: streaming, but also interactive, iterative, machine learning, etc. This talk will compare case studies for production deployments of Spark Streaming, emerging design patterns for integration with popular complementary OSS frameworks, plus some of the more advanced features such as approximation algorithms, and take a look at what's ahead — including the new Python support for Spark Streaming that will be in the upcoming 1.2 release. Also, let's chat a bit about the new Databricks + O'Reilly developer certification for Apache Spark…

Tiny Batches, in the wine: Shiny New Bits in Spark Streaming

Tiny Batches, in the wine: Shiny New Bits in Spark Streaming

Tiny Batches, in the wine: Shiny New Bits in Spark Streaming

Similar to Yahoo spark (20)

Fikrimuhal TRHUG 2016 Machine Learning

Fikrimuhal TRHUG 2016 Machine Learning

Fikrimuhal TRHUG 2016 Machine Learning

Clickstream & Social Media Analysis using Apache Spark

Clickstream & Social Media Analysis using Apache Spark

Clickstream & Social Media Analysis using Apache Spark

Hadoop @ Yahoo! - Internet Scale Data Processing

Hadoop @ Yahoo! - Internet Scale Data Processing

Hadoop @ Yahoo! - Internet Scale Data Processing

Apache® Spark™ MLlib: From Quick Start to Scikit-Learn

Apache® Spark™ MLlib: From Quick Start to Scikit-Learn

Apache® Spark™ MLlib: From Quick Start to Scikit-Learn

Ncku csie talk about Spark

Ncku csie talk about Spark

Ncku csie talk about Spark

SAP Technology Services Conference 2013: Big Data and The Cloud at Yahoo!

SAP Technology Services Conference 2013: Big Data and The Cloud at Yahoo!

SAP Technology Services Conference 2013: Big Data and The Cloud at Yahoo!

Hadoop at Yahoo! -- University Talks

Hadoop at Yahoo! -- University Talks

Hadoop at Yahoo! -- University Talks

Big Data Trend and Open Data

Big Data Trend and Open Data

Big Data Trend and Open Data

Capstone Project Slides- Yelper

Capstone Project Slides- Yelper

Capstone Project Slides- Yelper

Databricks Meetup @ Los Angeles Apache Spark User Group

Databricks Meetup @ Los Angeles Apache Spark User Group

Databricks Meetup @ Los Angeles Apache Spark User Group

SDSC18 and DSATL Meetup March 2018

SDSC18 and DSATL Meetup March 2018

SDSC18 and DSATL Meetup March 2018

Hw09 Hadoop Applications At Yahoo!

Hw09 Hadoop Applications At Yahoo!

Hw09 Hadoop Applications At Yahoo!

Hadoop at Yahoo! -- Hadoop World NY 2009

Hadoop at Yahoo! -- Hadoop World NY 2009

Hadoop at Yahoo! -- Hadoop World NY 2009

Big Data 2.0 - How Spark technologies are reshaping the world of big data ana...

Big Data 2.0 - How Spark technologies are reshaping the world of big data ana...

Big Data 2.0 - How Spark technologies are reshaping the world of big data ana...

Yahoo compares Storm and Spark

Yahoo compares Storm and Spark

Yahoo compares Storm and Spark

Introduction To Big Data and Use Cases on Hadoop

Introduction To Big Data and Use Cases on Hadoop

Introduction To Big Data and Use Cases on Hadoop

Power Platform Leeds - November 2019 - Microsoft Ignite Announcements

Power Platform Leeds - November 2019 - Microsoft Ignite Announcements

Power Platform Leeds - November 2019 - Microsoft Ignite Announcements

The Future Of SEO/Content Marketing

The Future Of SEO/Content Marketing

The Future Of SEO/Content Marketing

Hortonworks Big Data & Hadoop

Hortonworks Big Data & Hadoop

Hortonworks Big Data & Hadoop

Tiny Batches, in the wine: Shiny New Bits in Spark Streaming

Tiny Batches, in the wine: Shiny New Bits in Spark Streaming

Tiny Batches, in the wine: Shiny New Bits in Spark Streaming

Recently uploaded

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Manulife - Insurer Transformation Award 2024

Manulife - Insurer Transformation Award 2024

Manulife - Insurer Transformation Award 2024

The Digital Insurer

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

In the thrilling conclusion to 2023, ransomware groups had a banner year, really outdoing themselves in the "make everyone's life miserable" department. LockBit 3.0 took gold in the hacking olympics, followed by the plucky upstarts Clop and ALPHV/BlackCat. Apparently, 48% of organizations were feeling left out and decided to get in on the cyber attack action. Business services won the "most likely to get digitally mugged" award, with education and retail nipping at their heels. Hackers expanded their repertoire beyond boring old encryption to the much more exciting world of extortion. The US, UK and Canada took top honors in the "countries most likely to pay up" category. Bitcoins were the currency of choice for discerning hackers, because who doesn't love untraceable money?

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Overkill Security

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Architecting Cloud Native Applications

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

Rustici Software

ICT role in 21st century education and its challenges

ICT role in 21st century education and its challenges

ICT role in 21st century education and its challenges

rafiqahmad00786416

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Recently uploaded (20)

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Manulife - Insurer Transformation Award 2024

Manulife - Insurer Transformation Award 2024

Manulife - Insurer Transformation Award 2024

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Ransomware_Q4_2023. The report. [EN].pdf

Architecting Cloud Native Applications

Architecting Cloud Native Applications

Architecting Cloud Native Applications

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

ICT role in 21st century education and its challenges

ICT role in 21st century education and its challenges

ICT role in 21st century education and its challenges

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

Real Time Object Detection Using Open CV

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Artificial Intelligence Chap.5 : Uncertainty

Yahoo spark

1. Spark and Hadoop at Yahoo: Brought to you by YARN Andy Feng Yahoo! Hadoop (afeng@yahoo-inc.com)

2. Personalized Web

3. Big-Data in Yahoo! 3 9/10/13

4. Hadoop + Spark: Empowered by YARN 30k+ Yahoo! production nodes on YARN since Q1 2013

5. Shark Pilot: Advertising Data Analytics §  Business questions ›  Are two sets of audience cohorts similar to each other? ›  What audience segment is most likely to be interested in this ad campaign? ›  In what way was the new front page rollout different than the previous front page as far as audience engagement goes? ›  What are the right metrics to define user engagement? §  Shark pilot ›  50 nodes, each w/ 96GB RAM •  Currently loaded w/ 3.2 TB sample data in memory ›  Homegrown BI tools for ad-hoc queries •  Using Shark Server (contributed to community by Yahoo!)

6. Shark Perf: TCP-H Benchmark Average Seconds 600 500 400 300 200 100 0

7. Spark Pilot: Model Training Pipeline §  A DAG of M/R jobs in Hadoop Streaming ›  Feature extraction ›  Train models ›  Score and analyze models §  Initial Spark prototype ›  3x speedup on feature extraction §  Production launch ›  Apply Spark against complete pipeline ›  Spark on 80 node cluster •  Thanks to the enhanced UI and metrics in Spark 0.8 7 9/10/13

8. Use Case: Ad Targeting Spark M/R and Storm 8 9/10/13

9. Use Case: Content Recommendation w/ Collaborative Filtering Input CF Learning Ranking Spark Spark 9 9/10/13 Output

10. Spark-YARN: Deployment Simplified run spark.deploy.yarn.Client --jar … --class … --args … --queue …--num-workers … --worker-memory … Spark-YARN (contributed by Yahoo!) is being adopted by community (ex. Taobao) for production use. You should try it on your Hadoop cluster. 10 9/10/13

11. Acknowledgement §  AMPLab team ›  Outstanding collaboration: Ion, Matei, Reynold, Patrick, Matt, … §  Yahoo! Hadoop team ›  Thomas, Bobby, Paul, Rajiv, Mithun, … §  Yahoo! Lab. ›  Mridul, Nathan, … §  Yahoo! data analytics ›  Supreeth, Ram, Tim, … §  Yahoo! spark users ›  Gavin, Jay, Hirakendu, … 11 9/10/13

12. We Are Hiring! http://careers.yahoo.com/