Today’s reality Hadoop with Spark- How to select the best Data Science approach when using Big Data Platforms and Technologies?

•

0 gostou•767 visualizações

Martin Oberhuber and Eliano Marques, Senior Data Scientists @Think Big International In this talk Think Big International Lead Data Scientists will discuss the options that exist today for engineering and data science teams aiming to use big data patterns to solve new business problems. With the enterprise adoption of the Hadoop ecosystem and the emerging momentum of open source projects like Spark it is becoming mandatory to have an approach that solves for business results but remains flexible to adapt and change with the open source market.

Tecnologia

DATA SCIENCE
DATA ENGINEERS
DATA SOLUTIONS
Think&Big&Start&Smart&Scale&Fast
Eliano Marques-– Senior-Data-Scientist
Martin-Oberhuber-– Senior-Data-Scientist

CONFIDENTIAL+ +++++++| 2© 2015 Think Big, a Teradata Company
Think+Big+History
1st
SI+Solution+Provider+with+100%+focus+on+open+source+
and+Big+Data+Hadoop ecosystem
• 100++Successful+Programs
• 70++Clients
• Global+Delivery+Capabilities
• We-are-hiring

CONFIDENTIAL+ +++++++| 3© 2015 Think Big, a Teradata Company
Think-Big-Clients
Trusted&Analytics&Services&Provider&to&the&Fortune&1000
eCommerce
2+of+Global+Top+5
Internet-Transaction-Security
Global #1
Retail
2+of+Global+Top+5
Brokerage &-Mutual-Funds
2+of+Global+Top+5
Social-Networking
Global #1
Asset-Management
Global #1
Credit-Issuer
2+of Global+Top+5
Semiconductor
2+of+Global Top+5
Banking
4+of+Global Top+10
Data Storage-Devices
3+of+Global Top+5
Financial Data-Services
2+of+Global+Top+5
Disk Manufacturing
Global+#1
Financial-Exchanges
Global #2
Telecommunications
2+of+Global Top+5
Media-& Advertising
2+of+Global+Top+5

CONFIDENTIAL+ +++++++| 4© 2015 Think Big, a Teradata Company
Think+Big+VELOCITY Methodology
Big+Data
Strategy
Think+Big
Academy
Big+Data
Program+Mgt
Business
Analytics
Managed+
Services
Data+
Engineering
Big+Data+Lab
Think+Big+engages+with+it’s+client’s+business,+technical,+analyst+and+support+teams+in+
an+agile+inspired+VELOCITY+Methodology+to+continuously+develop+Big+Data+solutions+

CONFIDENTIAL+ +++++++| 5© 2015 Think Big, a Teradata Company
What+is+Apache+Spark?+
• Open+source+Apache+project
− Parallel+middleware+for+server+
clusters
− Spark.apache.org+(2014)
• Developed+by+UC+Berkeley’s+
AMPLab
− Supported+by+Databricks
• Top+use+cases
− SQLaonaHadoop
− Machine+learning
− Streaming+data+miniabatches

CONFIDENTIAL+ +++++++| 6© 2015 Think Big, a Teradata Company
Apache-Spark-Core-Engine
Spark-SQL
Spark-
Streaming
MLib
(Machine-learning)
GraphX
(Graph)
Scala,-R-(SparkR),-Python-(PySpark)
What+is+Apache+Spark?+

CONFIDENTIAL+ +++++++| 7© 2015 Think Big, a Teradata Company
Data+Science+Approaches
7
Single-Workstation
- Small+data+sets
- No+distributed+analytics+
across+multiple+nodes
- Powerful+tools+are+R+or+
Python
- Data+Scientist+can+focus+on+
business+problem
Mixed
Single/Workstation/+/Cluster
- Small+or+large+data+sets
- Data+wrangling+and+feature+
engineering+is+performed+on+
cluster
- Predictive+analysis+and+
modeling+can+be+performed+on+
single+workstation
- Powerful+tools+are+Hadoop
Streaming+and+Spark
combined+with+R+and+Python
- Data+Scientist+now+have+to+
worry+about+parallelisation of+
some+data+mining+tasks+
(ususally the+ones+that+are+
embarrassingly+parallel)
Cluster
- Large+data+sets
- Both+data+wrangling+and+
modeling+is+performed+on+
cluster
- Spark+is+one+of+the+few+tools+
that+support+efficient+parallel+
machine+learning
- Parallelising machine+learning+
algorithms+is+challenging

CONFIDENTIAL+ +++++++| 8© 2015 Think Big, a Teradata Company
Data-Lake-(HDFS)
Core-Data-ScienceProduction
• Dashboards
• R+Shiny+Apps
• Predictive+model+
scoring
Plug+&+play+model+deployment
Data-Sources-
(Operations,+
Sales,+
marketing,+etc)
Ingestion
Realatime+
Optimization+with+
Multiaarmed+Bandit
Data
• Integration+of+R+and+
Python+with+Hadoop and+
Spark
• Leveraging+computing+
power+of+Hadoop cluster+
for+distributed+analytics
• Plug+&+play+model+
deployment+tools+for+
easy+and+robust+
productionising of+
analytics+models
Realatime+Data
Productionising Analytics

CONFIDENTIAL+ +++++++| 9© 2015 Think Big, a Teradata Company
Project-KickVoff
Data-Profiling-
and-Exploratory-
Analysis
Analytics-
Modeling
Model-Validation Model-Publishing Reporting
Data-Science-Project
Data+Science+and+Analytics+Overview

CONFIDENTIAL+ +++++++| 10© 2015 Think Big, a Teradata Company
We+leverage+our+expertise+across+industries
Dynamic-Pricing
Fraud-Detection
Customer-Segmentation
Recommendation-
Engine
Predictive-Asset-
Maintenance
Proactive-
Customer-
Support
Credit-Default-
Prediction
Churn-Modeling
Scenario-Simulation
A/B-Testing
Display-Targeting-Optimisation
Demand-Forecast
Cluster-Analysis-&-
Segmentation
Device-Analytics
Risk-Analytics
Customer-Analytics

CONFIDENTIAL+ +++++++| 11© 2015 Think Big, a Teradata Company
Thank+you

Mais conteúdo relacionado

Mais procurados

Case study: Hadoop as ELT for Leading US Retailer - Happiest Minds

Happiest Minds Technologies

Alliander robin hagemans daniel peyron

BigDataExpo

Why watch? Looking to analyze your growing data assets to unlock real business benefits today? But, are you sick of all the Big Data hype and whoopla? Watch this on-demand Webinar from Actian and Yellowfin – Big Data Analytics with Hadoop – to discover how we’re making Big Data Analytics fast and easy: Learn how a telecommunications provider has already transformed its business using Big Data Analytics with Hadoop. Hold on as we go from data in Hadoop to predictive analytics in just 40-minutes. Learn how to combine Hadoop with the most advanced Big Data technologies, and world’s easiest BI solution, to quickly generate real business value from Big Data Analytics. What will you learn? Discover how Actian’s market-leading Big Data Analytics technologies, combined with Yellowfin’s consumer-oriented platform for reporting and analytics, makes generating value from Big Data Analytics faster and easier than you thought possible. Join us as we demonstrate how to: • Connect to, prepare and optimize Big Data in Hadoop for reporting and analytics. • Perform predictive analytics on streaming Big Data: Learn how to empower all your analytics stakeholders to move from historical reports to predictive analytics and gain a sustainable competitive advantage. • Communicate insights attained from Big Data: Optimize the value of your Big Data insights by learning how to effectively communicate analytical information to defined user groups and types. This Webinar is ideal if… • You want to act on more data and data types in shorter timeframes • You want to understand the steps involved in achieving Big Data success – both front and back end • You want to see how market leaders are leveraging Big Data to become data-driven organizations today Looking to analyze and exploit Big Data assets stored in Hadoop? Then this Webinar is a must.

Big Data Analytic with Hadoop: Customer Stories

Yellowfin

Eneco Ronald Root

BigDataExpo

De groote de man Ingrid de Poorter

BigDataExpo

AURIN Data Hubs Supporting Smarter Cities - Phil Delaney, Locate14

Phillip Delaney

Neo4j Aura Enterprise

Neo4j

Importance of Big Data Analytics

Impetus Technologies

Drive Business Outcomes for Big Data Environments

Cisco Services

Big data analytics

iACT Global

Itility marianne faro

BigDataExpo

Six steps to leveraging location for the Canadian insurance industry

DMTI Spatial

Enable Advanced Analytics with Hadoop and an Enterprise Data Hub

Cloudera, Inc.

Cisco_Big_Data_Webinar_At-A-Glance_ABSOLUTE_FINAL_VERSION

Renee Yao

Multi Cloud Data Integration- Manufacturing Industry

alanwaler

Talend mike hirt

BigDataExpo

Mais procurados (16)

Case study: Hadoop as ELT for Leading US Retailer - Happiest Minds

Alliander robin hagemans daniel peyron

Big Data Analytic with Hadoop: Customer Stories

Eneco Ronald Root

De groote de man Ingrid de Poorter

AURIN Data Hubs Supporting Smarter Cities - Phil Delaney, Locate14

Neo4j Aura Enterprise

Importance of Big Data Analytics

Drive Business Outcomes for Big Data Environments

Big data analytics

Itility marianne faro

Six steps to leveraging location for the Canadian insurance industry

Enable Advanced Analytics with Hadoop and an Enterprise Data Hub

Cisco_Big_Data_Webinar_At-A-Glance_ABSOLUTE_FINAL_VERSION

Multi Cloud Data Integration- Manufacturing Industry

Talend mike hirt

Destaque

At Google Cloud Platform, we're combining the Apache Spark and Hadoop ecosystem with our software and hardware innovations. We want to make these awesome tools easier, faster, and more cost-effective, from 3 to 30,000 cores. This presentation will showcase how Google Cloud Platform is innovating with the goal of bringing the Hadoop ecosystem to everyone. Bio: "I love data because it surrounds us - everything is data. I also love open source software, because it shows what is possible when people come together to solve common problems with technology. While they are awesome on their own, I am passionate about combining the power of open source software with the potential unlimited uses of data. That's why I joined Google. I am a product manager for Google Cloud Platform and manage Cloud Dataproc and Apache Beam (incubating). I've previously spent time hanging out at Disney and Amazon. Beyond Google, love data, amateur radio, Disneyland, photography, running and Legos."

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop

huguk

Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...

huguk

Sean Owen – Director of Data Science @Cloudera Building machine learning models is all well and good, but how do they get productionized into a service? It's a long way from a Python script on a laptop, to a fault-tolerant system that learns continuously, serves thousands of queries per second, and scales to terabytes. The confederation of open source technologies we know as Hadoop now offers data scientists the raw materials from which to assemble an answer: the means to build models but also ingest data and serve queries, at scale. This short talk will introduce Oryx 2, a blueprint for building this type of service on Hadoop technologies. It will survey the problem and the standard technologies and ideas that Oryx 2 combines: Apache Spark, Kafka, HDFS, the lambda architecture, PMML, REST APIs. The talk will touch on a key use case for this architecture -- recommendation engines.

Lambda architecture on Spark, Kafka for real-time large scale ML

huguk

As Hadoop became mainstream, the need to simplify and speed up analytics processes grew rapidly. Data wrangling emerged as a necessary step in any analytical pipeline, and is often considered to be its crux, taking as much as 80% of an analyst's time. In this presentation we will discuss how data wrangling solutions can be leveraged to streamline, strengthen and improve data analytics initiatives on Hadoop, including use cases from Trifacta customers. Bio: Olivier is EMEA Solutions Lead at Trifacta. He has 7 years experience in analytics with prior roles as technical lead for business analytics at Splunk and quantitative analyst at Accenture and Aon.

Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

huguk

Stephen Taylor is the community manager for Ether Camp. They provide an analysis tool for the Ethereum blockchain, ‘Block Explorer’ and also an ‘Intergrated Development Environment’ (I.D.E) that empowers developers to build, test and deploy applications in a sandbox environment. This November they are launching their second annual hackathon, hack.ether.camp which is aiming to deliver a more sustained approach to the hackathon ideology, by utilising blockchain technology.

ether.camp - Hackathon & ether.camp intro

huguk

Big organisations have a wealth of rich customer data which opens up huge new opportunities. However, they have the challenge of how to extract value from this data while protecting the privacy of their individual customers. He will talk about the risks organisations face, and what they should do about it. He will survey the techniques which can be used to make data safe for analysis, and talk briefly about how they are solving this problem at Privitar.

Extracting maximum value from data while protecting consumer privacy. Jason ...

huguk

Jonathon Southam: Venture Capital, Funding & Pitching

huguk

We Are Social presents Social Brands: The eBook

We Are Social Singapore

Destaque (8)

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop

Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...

Lambda architecture on Spark, Kafka for real-time large scale ML

Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

ether.camp - Hackathon & ether.camp intro

Extracting maximum value from data while protecting consumer privacy. Jason ...

Jonathon Southam: Venture Capital, Funding & Pitching

We Are Social presents Social Brands: The eBook

Semelhante a Today’s reality Hadoop with Spark- How to select the best Data Science approach when using Big Data Platforms and Technologies?

Outsource Bigdata has been providing data abstraction services for years. We have years of experience in working with organizations from various countries and fields. Our team can easily extract information from a variety of documents such as product documents, corporate reports, customer, details, production reports, etc. Know More :https://outsourcebigdata.com/data-automation/data-processing-services/data-abstraction-services/ AIMLEAP - Outsource Bigdata is global technology consulting and service provider offering End-to-End IT Services, AI-Augmented Data Solutions, Automation, Digital Marketing, and Research & Analytics Services. We successfully delivered projects for more than 550 fast-growing companies in the USA, Europe, New Zealand, Australia, Canada: and more. - An ISO 9001:2015 and ISO/IEC 27001:2013 certified - Served 550+ customers - 9+ Years of industry experience - 98% Client Retention - Global Delivery Centers in the USA, Canada, India & Australia Email: sales@outsourcebigdata.com USA: 1-30235 14656 Canada: +1 902 279 1797 India: +91 810 527 1615 Australia: +61 402 576 615

Key benefits of outsourcing data abstraction services to our team

Outsource Bigdata

ZIGRAM is a data technology organization that operates in the Data Asset space. Our team is made up of professionals from varied domains like data science, technology, sales, risk, analytics, financial services, research, and business consulting. Our aim is to deliver value to clients by Building Solutions, Developing Data Assets, and Managing Projects across use cases - thereby boosting revenues and reducing the cost of doing business, in a data-driven world.

ZIGRAM Introduction September 2020

ZIGRAM

Denodo Data Virtualization - IT Days in Luxembourg with Oktopus

Denodo

The Art of Data Science - event slides

RedPixie

We at MetaOption LLC invite you to join us for advanced IT infrastructure management services. We are the leaders in the IT industry for serving advanced IT Solutions to our clients. We are serving innovative IT solutions & consultancy for more than 8 years to many leading enterprises running at very high levels. Our IT infrastructure management services package includes several innovative tools to manage and maintain your IT infrastructure assets for their profitable running.

Metaoption IT solutions

Abhishek Pathak

12th July GDPR event slides

Exponential_e

BIGITUAE201

Bryan C.

Intelligent Business Process Management Suites (iBPMS) - The Next-Generation ...

Kai Wähner

ZIGRAM Introduction July 2021

ZIGRAM

InfoVision is a Global IT Services and Solutions company offering Strategic Resources, Enterprise Applications and Technology Solutions. InfoVision’s specialized Technology Solution offerings include Mobility, Outsourced Product Development, Business Intelligence and Big Data Analytics, DevOps, IoT, Testing Services and Cloud Services. InfoVision’s Integrated Value Delivery capability is the combination of proven talent, technical depth, delivery discipline and domain expertise.

InfoVision Corporate Profile

InfoVisionHQ

ThousandEyes Webinar: How to see and resolve office 365 performance challenges

ThousandEyes

This presentation was made on June 16, 2020. A recording of the presentation can be viewed here: https://youtu.be/khjW1t0gtSA AI is unlocking new potential for every enterprise. Organizations are using AI and machine learning technology to inform business decisions, predict potential issues, and provide more efficient, customized customer experiences. The results can enable a competitive edge for the business. H2O.ai is a visionary leader in AI and machine learning and is on a mission to democratize AI for everyone. We believe that every company can become an AI company, not just the AI Superpowers. We are empowering companies with our leading AI and Machine Learning platforms, our expertise, experience and training to embark on their own AI journey to become AI companies themselves. All companies in all industries can participate in this AI Transformation. Tune into this virtual meetup to learn how companies are transforming their business with the power of AI and where to start. About Parul Pandey: Parul is a Data Science Evangelist here at H2O.ai. She combines Data Science , evangelism and community in her work. Her emphasis is to spread the information about H2O and Driverless AI to as many people as possible, She is also an active writer and has contributed towards various national and international publications.

Your AI Transformation

Sri Ambati

Digital Velocity London 2018 - James Morgan, Sainsbury's

Tealium

Chordify Brochure Staffing

varunberry

More than ever, marketing has become reliant on data. With the advent of the internet and digital media in the last 20 years, we are now awashed with data from different sources as well as ever increasing volume. Digital is not just advertising media; it has become part of business. Data-driven marketing is becoming the norm, but making sense of the data requires new technologies like AI. Joe will explore this rise of data science and how it may affect businesses.

The Rise of Data Science Master Class - Joe Nguyen, H Plus

DigiMarCon - Digital Marketing, Media and Advertising Conferences & Exhibitions

Making Big Data Work

Corecom Consulting

BICS empowers predictive analytics and customer centricity with a Hadoop base...

DataWorks Summit

Modern Enterprises are under more pressure than ever before, with increased demands and more competition, especially from native digital competitors, making digital transformation more vital than ever. Intelligent automation is your route to success with that transformation. With Automation Anywhere’s industry-leading RPA and Digital Workforce platform companies can dramatically increase the speed of deploying RPA, automate more business processes than ever before while decreasing the time in which they derive compelling business value.

Shift Money 2019 - Staying Competitive in a Digital World – moving to Intelli...

Shift Conference

Insight Presentation - Anil Kaul, AbsolutData

NorthWest MRA

From Paris Hilton to Walmart: welcome to the Big Data Revolution

William Visterin

Semelhante a Today’s reality Hadoop with Spark- How to select the best Data Science approach when using Big Data Platforms and Technologies? (20)

Key benefits of outsourcing data abstraction services to our team

ZIGRAM Introduction September 2020

Denodo Data Virtualization - IT Days in Luxembourg with Oktopus

The Art of Data Science - event slides

Metaoption IT solutions

12th July GDPR event slides

BIGITUAE201

Intelligent Business Process Management Suites (iBPMS) - The Next-Generation ...

ZIGRAM Introduction July 2021

InfoVision Corporate Profile

ThousandEyes Webinar: How to see and resolve office 365 performance challenges

Your AI Transformation

Digital Velocity London 2018 - James Morgan, Sainsbury's

Chordify Brochure Staffing

The Rise of Data Science Master Class - Joe Nguyen, H Plus

Making Big Data Work

BICS empowers predictive analytics and customer centricity with a Hadoop base...

Shift Money 2019 - Staying Competitive in a Digital World – moving to Intelli...

Insight Presentation - Anil Kaul, AbsolutData

From Paris Hilton to Walmart: welcome to the Big Data Revolution

Mais de huguk

IBM is developing the Watson Ecosystem to leverage its Developer Cloud, APIs, Content Store and Talent Hub. This is part of IBM's recent announcement of the $1B investment in Watson as a new business unit including Silicon Alley NYC headquarters. For the first time, IBM will open up Watson as a development platform in the Cloud to spur innovation and fuel a new ecosystem of entrepreneurial software app providers who will bring forward a new generation of applications infused with Watson's cognitive computing intelligence.

Intelligence Augmented vs Artificial Intelligence. Alex Flamant, IBM Watson

huguk

In this talk about Apache Flink we will touch on three main things, an introductory look at Flink, a look under the hood and a demo. * In the introduction we will briefly look at the history of Flink and then go on to the API and different use cases. Here we will also see how it can be deployed in practice and what some of the pitfalls in a cluster setting can be. * In the second section we will look at the streaming execution engine that lies at the heart of Flink. Here we will see what makes it tick and also what distinguishes it from other approaches, such as the mini-batch execution model. Ufuk Celebi - PMC member at Apache Flink and co-founder and software engineer at data Artisans * In the final section we will see a live demo of a fault-tolerant streaming job that performs analysis of the wikipedia edit-stream.

Streaming Dataflow with Apache Flink

huguk

Signal Media: Real-Time Media & News Monitoring

huguk

Dean Bryen: Scaling The Platform For Your Startup

huguk

Peter Karney: Intro to the Digital catapult

huguk

Cytora: Real-Time Political Risk Analysis

huguk

Cubitic: Predictive Analytics

huguk

Bird.i: Earth Observation Data Made Social

huguk

Aiseedo: Real Time Machine Intelligence

huguk

This talk will cover the design and implementation decisions that have been key to the success of Apache Spark over other competing cluster computing frameworks. It will be delving into the whitepaper behind Spark and cover the design of Spark RDDs, the abstraction enables the Spark execution engine to be extended to support a wide variety of use cases: Spark SQL, Spark Streaming, MLib and GraphX. RDDs allow Spark to outperform existing models by up to 100x in multi-pass analytics.

Secrets of Spark's success - Deenar Toraskar, Think Reactive

huguk

Technical developments in the area of data warehousing have allowed companies to push their analysis a step further and, therefore, allowed data scientists to deliver more value to business areas. In that session, we will focus on the case of performance marketing at King and demonstrate how we use Hadoop capabilities to exploit user-level data efficiently. That approach results in obtaining a more holistic view in a return-on-investment analysis of TV advertisement.

TV Marketing and big data: cat and dog or thick as thieves? Krzysztof Osiewal...

huguk

Hadoop - Looking to the Future By Arun Murthy (Founder of Hortonworks, Creator of YARN) The Apache Hadoop ecosystem began as just HDFS & MapReduce nearly 10 years ago in 2006. Very much like the Ship of Theseus (http://en.wikipedia.org/wiki/Ship_of_Theseus), Hadoop has undergone incredible amount of transformation from multi-purpose YARN to interactive SQL with Hive/Tez to machine learning with Spark. Much more lies ahead: whether you want sub-second SQL with Hive or use SSDs/Memory effectively in HDFS or manage Metadata-driven security policies in Ranger, the Hadoop ecosystem in the Apache Software Foundation continues to evolve to meet new challenges and use-cases. Arun C Murthy has been involved with Apache Hadoop since the beginning of the project - nearly 10 years now. In the beginning he led MapReduce, went on to create YARN and then drove Tez & the Stinger effort to get to interactive & sub-second Hive. Recently he has been very involved in the Metadata and Governance efforts. In between he founded Hortonworks, the first public Hadoop distribution company.

Hadoop - Looking to the Future By Arun Murthy

huguk

By Kevin Schmidt (Head of Data Science at Mind Candy) Luis Vicente (Senior Data Engineer at Mind Candy) For mobile games, constant tweaks are the difference between success and failure. Data and analytics have to be available in real-time, but calculating, for example, uniqueness or newness of a data point requires a list of seen data points - both memory intensive and tricky when using real-time stream processing like Spark Streaming. Probabilistic data structures allow approximation of these properties with a fixed memory representation, and are very well suited for this kind of stream processing. Getting from the theory of approximation to a useful metric at a low error rate even for many millions of users is another story. In our talk we will look at practical ways of achieving this: which approximation we used for selection of useful metrics, why we picked a specific probabilistic data structure, how we stored it in Cassandra as a time series and how we implemented it in Spark Streaming.

Fast real-time approximations using Spark streaming

huguk

The task of “data profiling”—assessing the overall content and quality of a data set—is a core aspect of the analytic experience. Traditionally, profiling was a fairly cut-and-dried task: load the raw numbers into a stat package, run some basic descriptive statistics, and report the output in a summary file or perhaps a simple data visualization. However, data volumes can be so large today that traditional tools and methods for computing descriptive statistics become intractable; even with scalable infrastructure like Hadoop, aggressive optimization and statistical approximation techniques must be used. In this talk Sean will cover technical challenges in keeping data profiling agile in the Big Data era. He will discuss both research results and real-world best practices used by analysts in the field, including methods for sampling, summarizing and sketching data, and the pros and cons of using these various approaches. Sean is Trifacta’s Chief Technical Officer. He completed his Ph.D. at Stanford University, where his research focused on user interfaces for database systems. At Stanford, Sean led development of new tools for data transformation and discovery, such as Data Wrangler. He previously worked as a data analyst at Citadel Investment Group.

Sean Kandel - Data profiling: Assessing the overall content and quality of a ...

huguk

Fraud Detection - This talk will outline NRT Fraud Detection and event detection using the EDH to achieve sub-second response times. This talk will focus specifically on Fraud and Event Detection architectures and tools necessary to build the system. Kevin O’Dell has been an HBase contributor since 2012. Kevin currently works as a Field Engineer for Cloudera building Big Data applications with a specialization in HBase. He is also the co-author on the upcoming Architecting HBase Applications book being published by O'Reilly.

Kevin O'Dell - Fraud and event detection using the Enterprise Data Hub

huguk

Matt will take a look at the EMR interface, and explore what additional value EMR provides for creating and managing Hadoop clusters. Matt is a principal technologist in the technical IT training team at QA. He is a Microsoft Certified Trainer and an authorized Amazon trainer, focusing on creating and delivering courses about cloud services, service-oriented architectures and enterprise application integration.

Matthew Bishop - A Quick Introduction to AWS Elastic MapReduce

huguk

Process Scheduling on Hadoop at Expedia

huguk

Developing Unit Testable Software with Hadoop at Expedia

huguk

SUSE, Hadoop and Big Data Update. Stephen Mogg, SUSE UK

huguk

Olivier from Hortonworks will introduce Apache Argus a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. Data security within Hadoop has evolved to support multiple use cases for data access, while also providing a framework for central administration of security policies and monitoring of user access. With the advent of Apache YARN and Argus, the Hadoop platform can now support a true secure data lake architecture.

Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...

huguk

Mais de huguk (20)

Intelligence Augmented vs Artificial Intelligence. Alex Flamant, IBM Watson

Streaming Dataflow with Apache Flink

Signal Media: Real-Time Media & News Monitoring

Dean Bryen: Scaling The Platform For Your Startup

Peter Karney: Intro to the Digital catapult

Cytora: Real-Time Political Risk Analysis

Cubitic: Predictive Analytics

Bird.i: Earth Observation Data Made Social

Aiseedo: Real Time Machine Intelligence

Secrets of Spark's success - Deenar Toraskar, Think Reactive

TV Marketing and big data: cat and dog or thick as thieves? Krzysztof Osiewal...

Hadoop - Looking to the Future By Arun Murthy

Fast real-time approximations using Spark streaming

Sean Kandel - Data profiling: Assessing the overall content and quality of a ...

Kevin O'Dell - Fraud and event detection using the Enterprise Data Hub

Matthew Bishop - A Quick Introduction to AWS Elastic MapReduce

Process Scheduling on Hadoop at Expedia

Developing Unit Testable Software with Hadoop at Expedia

SUSE, Hadoop and Big Data Update. Stephen Mogg, SUSE UK

Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...

Último

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Histor y of HAM Radio presentation slide

vu2urc

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Advantages of Hiring UIUX Design Service Providers for Your Business

Pixlogix Infotech

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

Scaling API-first – The story of a global engineering organization

Radu Cotescu

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Today’s reality Hadoop with Spark- How to select the best Data Science approach when using Big Data Platforms and Technologies?

1. DATA SCIENCE DATA ENGINEERS DATA SOLUTIONS Think&Big&Start&Smart&Scale&Fast Eliano Marques-– Senior-Data-Scientist Martin-Oberhuber-– Senior-Data-Scientist

2. CONFIDENTIAL+ +++++++| 2© 2015 Think Big, a Teradata Company Think+Big+History 1st SI+Solution+Provider+with+100%+focus+on+open+source+ and+Big+Data+Hadoop ecosystem • 100++Successful+Programs • 70++Clients • Global+Delivery+Capabilities • We-are-hiring

3. CONFIDENTIAL+ +++++++| 3© 2015 Think Big, a Teradata Company Think-Big-Clients Trusted&Analytics&Services&Provider&to&the&Fortune&1000 eCommerce 2+of+Global+Top+5 Internet-Transaction-Security Global #1 Retail 2+of+Global+Top+5 Brokerage &-Mutual-Funds 2+of+Global+Top+5 Social-Networking Global #1 Asset-Management Global #1 Credit-Issuer 2+of Global+Top+5 Semiconductor 2+of+Global Top+5 Banking 4+of+Global Top+10 Data Storage-Devices 3+of+Global Top+5 Financial Data-Services 2+of+Global+Top+5 Disk Manufacturing Global+#1 Financial-Exchanges Global #2 Telecommunications 2+of+Global Top+5 Media-& Advertising 2+of+Global+Top+5

4. CONFIDENTIAL+ +++++++| 4© 2015 Think Big, a Teradata Company Think+Big+VELOCITY Methodology Big+Data Strategy Think+Big Academy Big+Data Program+Mgt Business Analytics Managed+ Services Data+ Engineering Big+Data+Lab Think+Big+engages+with+it’s+client’s+business,+technical,+analyst+and+support+teams+in+ an+agile+inspired+VELOCITY+Methodology+to+continuously+develop+Big+Data+solutions+

5. CONFIDENTIAL+ +++++++| 5© 2015 Think Big, a Teradata Company What+is+Apache+Spark?+ • Open+source+Apache+project − Parallel+middleware+for+server+ clusters − Spark.apache.org+(2014) • Developed+by+UC+Berkeley’s+ AMPLab − Supported+by+Databricks • Top+use+cases − SQLaonaHadoop − Machine+learning − Streaming+data+miniabatches

6. CONFIDENTIAL+ +++++++| 6© 2015 Think Big, a Teradata Company Apache-Spark-Core-Engine Spark-SQL Spark- Streaming MLib (Machine-learning) GraphX (Graph) Scala,-R-(SparkR),-Python-(PySpark) What+is+Apache+Spark?+

7. CONFIDENTIAL+ +++++++| 7© 2015 Think Big, a Teradata Company Data+Science+Approaches 7 Single-Workstation - Small+data+sets - No+distributed+analytics+ across+multiple+nodes - Powerful+tools+are+R+or+ Python - Data+Scientist+can+focus+on+ business+problem Mixed Single/Workstation/+/Cluster - Small+or+large+data+sets - Data+wrangling+and+feature+ engineering+is+performed+on+ cluster - Predictive+analysis+and+ modeling+can+be+performed+on+ single+workstation - Powerful+tools+are+Hadoop Streaming+and+Spark combined+with+R+and+Python - Data+Scientist+now+have+to+ worry+about+parallelisation of+ some+data+mining+tasks+ (ususally the+ones+that+are+ embarrassingly+parallel) Cluster - Large+data+sets - Both+data+wrangling+and+ modeling+is+performed+on+ cluster - Spark+is+one+of+the+few+tools+ that+support+efficient+parallel+ machine+learning - Parallelising machine+learning+ algorithms+is+challenging

8. CONFIDENTIAL+ +++++++| 8© 2015 Think Big, a Teradata Company Data-Lake-(HDFS) Core-Data-ScienceProduction • Dashboards • R+Shiny+Apps • Predictive+model+ scoring Plug+&+play+model+deployment Data-Sources- (Operations,+ Sales,+ marketing,+etc) Ingestion Realatime+ Optimization+with+ Multiaarmed+Bandit Data • Integration+of+R+and+ Python+with+Hadoop and+ Spark • Leveraging+computing+ power+of+Hadoop cluster+ for+distributed+analytics • Plug+&+play+model+ deployment+tools+for+ easy+and+robust+ productionising of+ analytics+models Realatime+Data Productionising Analytics

9. CONFIDENTIAL+ +++++++| 9© 2015 Think Big, a Teradata Company Project-KickVoff Data-Profiling- and-Exploratory- Analysis Analytics- Modeling Model-Validation Model-Publishing Reporting Data-Science-Project Data+Science+and+Analytics+Overview

10. CONFIDENTIAL+ +++++++| 10© 2015 Think Big, a Teradata Company We+leverage+our+expertise+across+industries Dynamic-Pricing Fraud-Detection Customer-Segmentation Recommendation- Engine Predictive-Asset- Maintenance Proactive- Customer- Support Credit-Default- Prediction Churn-Modeling Scenario-Simulation A/B-Testing Display-Targeting-Optimisation Demand-Forecast Cluster-Analysis-&- Segmentation Device-Analytics Risk-Analytics Customer-Analytics

Today’s reality Hadoop with Spark- How to select the best Data Science approach when using Big Data Platforms and Technologies?

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (16)

Destaque

Destaque (8)

Semelhante a Today’s reality Hadoop with Spark- How to select the best Data Science approach when using Big Data Platforms and Technologies?

Semelhante a Today’s reality Hadoop with Spark- How to select the best Data Science approach when using Big Data Platforms and Technologies? (20)

Mais de huguk

Mais de huguk (20)

Último

Último (20)

Today’s reality Hadoop with Spark- How to select the best Data Science approach when using Big Data Platforms and Technologies?