Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

•

1 gostou•1,303 visualizações

As Hadoop became mainstream, the need to simplify and speed up analytics processes grew rapidly. Data wrangling emerged as a necessary step in any analytical pipeline, and is often considered to be its crux, taking as much as 80% of an analyst's time. In this presentation we will discuss how data wrangling solutions can be leveraged to streamline, strengthen and improve data analytics initiatives on Hadoop, including use cases from Trifacta customers. Bio: Olivier is EMEA Solutions Lead at Trifacta. He has 7 years experience in analytics with prior roles as technical lead for business analytics at Splunk and quantitative analyst at Accenture and Aon.

Tecnologia

Hadoop User Group London: Data Wrangling on Hadoop
September 8 2016
Olivier de Garrigues, EMEA Solutions Lead

Creating radical productivity
for people who analyze data.
JEFFREY HEER
Co-Founder & CXO
VISUALIZATION
JOE HELLERSTEIN
Co-Founder & CSO
BIG DATA
SEAN KANDEL
Co-Founder & CTO
HUMAN-COMPUTER INTERACTION

What is Data Wrangling?
4
QUESTION ANALYZE INSIGHTDISCOVER STRUCTURE CLEANSE ENRICH VALIDATE PUBLISH

The Bridge Between Raw Data & Analysis
5
v
Ingestion Storage Processing
ANALYSIS & VISUALIZATION
LOBCLEANING ENRICHMENT DISTILLATIONSTRUCTURINGDISCOVERY
End-User Capabilities
IT
GOVERNANCE INTEGRATION AVAILABILTIYSCALABILITYSECURITY
Technical Capabilities

Conventional Approaches Inhibit User Empowerment
Hand-Coding Technical Workﬂow Mapping

Trifacta Approach: It’s All About The Experience
Interact Predict
Preview

TRIFACTA
DATA WRANGLING WORKFLOW
Trifacta. Confidential & Proprietary.
Sample Scale Up
Reﬁne
Sample
Results
Identify/Register Data
1.
Predictive Interaction
2
.
Consume
Schedulers
Monitor and Adjust
3
.
Schedule
Visualization & Analysis
Secure Access

Ingestion Processing Storage
ANALYSIS & CONSUMPTION
v
Discover Structure Clean Enrich Distill
LOB
IT
News
Topics
Time
Trades
Tickers
Date
$
eMails
Recipients
Topics
Phone Logs
Call Details
Recipients
Corporations
Company Relations
Individuals
Financial Services use case: Trader Fraud

Data Wrangling Beneﬁts
➔  Empower the people who know the data best
➔  Accelerate time to value
➔  Lower business risk with more accurate data
➔  Unlock innovation using a wider variety of data

Mais conteúdo relacionado

Mais procurados

Apouc 2014-business-analytics-and-big-data

OUGTH Oracle User Group in Thailand

As a knowledge graph database, Grakn is ideal for storing metadata and data lineage information. Many applications, such as data discovery, data governance, and data marketplaces, depend upon metadata for management. User experiences can be enhanced by leveraging a hyper-scalable graph database like Grakn, rather than traditional graph databases. Additionally, inference-driven use cases predominantly depended on RDF Triple Stores, requiring additional plug-ins to derive the inferences. With Grakn, this can now be achieved natively.

Unified Information Governance, Powered by Knowledge Graph

Vaticle

Learn about the industry's new open metadata standard Egeria, introduced in September by ODPi, The Linux Foundation’s Open Data Platform initiative. Egeria supports the free flow of standardized metadata between different technologies and vendor platforms, enabling organizations to locate, manage and use their data resources more effectively. Explore how Egeria's set of open APIs, types and interchange protocols to allow all metadata repositories to share and exchange metadata. From this common base, it adds governance, discovery and access frameworks for automating the collection, management and use of metadata across an enterprise. The result is an enterprise catalog of data resources that are transparently assessed, governed and used in order to deliver maximum value to the enterprise. This presentation by ODPi Director John Mertic provides an introduction to Egeria, and explores how the standard provides a vendor-neutral approach to data governance. Learn how a group of companies led by ING, IBM and Hortonworks came together through the open source community to re-imagining data governance and delivered Egeria -- to automate the collection, management and use of metadata across organizations of any size and complexity. Learn how Egeria was built on open standards and delivered via Apache 2.0 open source license.

Streamline Data Governance with Egeria: The Industry's First Open Metadata St...

DataWorks Summit

Data Science Application in Business Portfolio & Risk Management

Data Science Thailand

Scalability and Graph Analytics with Neo4j - Stefan Kolmar, Neo4j

Neo4j

In this webinar, AnzoGraph’s graph database guru Barry Zane (former co-founder of Netezza) and data governance author Steve Sarsfield talk about how graph databases fit into the data warehouse modernization trend. They also explore how certain workloads can be better served with an analytical graph database and how today’s technology stacks offer new paradigms for deployment like the cloud, containers and graph analytics.

Should a Graph Database Be in Your Next Data Warehouse Stack?

Cambridge Semantics

In their webinar "Big Data Fabric 2.0 Drives Data Democratization" Ben Szekley, Cambridge Semantics’ SVP of Field Operations, and guest speaker, Forrester’s Noel Yuhanna, author of the Forrester report: “Big Data Fabric 2.0 Drives Data Democratization”, explored why data-driven businesses are making a big data fabric part of their data strategy to minimize data complexity, integrate siloed data, deliver real-time trusted insights, and to create new business opportunities. These are the slides from that webinar.

Big Data Fabric 2.0 Drives Data Democratization

Cambridge Semantics

Using Cloud Automation Technologies to Deliver an Enterprise Data Fabric

Cambridge Semantics

Infochimps + CloudCon: Infinite Monkey Theorem

Infochimps, a CSC Big Data Business

BI architecture drivers have to change to satisfy new requirements in format, volume, latency, hosting, analysis, reporting, and visualization. In this presentation delivered at the 2014 SATURN conference, SoftServe`s Serhiy and Olha showcased a number of reference architectures that address these challenges and speed up the design and implementation process, making it more predictable and economical: - Traditional architecture based on an RDMBS data warehouse but modernized with column-based storage to handle a high load and capacity - NoSQL-based architectures that address Big Data batch and stream-based processing and use popular NoSQL and complex event-processing solutions - Hybrid architecture that combines traditional and NoSQL approaches to achieve completeness that would not be possible with either alone The architectures are accompanied by real-life projects and case studies that the presenters have performed for multiple companies, including Fortune 100 and start-ups.

Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...

SoftServe

[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics

Infochimps, a CSC Big Data Business

Importance of Big Data Analytics

Impetus Technologies

# Automating Data Science over a Human Genomics Knowledge Base Radouane Oudrhiri, the CTO of Eagle Genomics, will talk about how Eagle Genomics is building a platform for automating data science over a human genomics knowledge base. Rad will dive into the architecture Eagle Genomics and also discuss how Grakn serves as the knowledge base foundation of the system. Rad also give a brief history of databases, semantic expressiveness and how Grakn fits in the big picture. # Radouane Oudrhiri, CTO, Eagle Genomics Radouane has an extensive experience in leading world-class software and data-intensive system developments in different industries from Telecom to Healthcare, Nuclear, Automotive, Financials. Radouane is Lean/Six Sigma Master Black Belt with speciality in high-tech, IT and Software engineering and he is recognised as the leader and early adaptor of Lean/Six Sigma and DFSS to IT and Software. He is a fellow of the Royal Statistical Society (RSS) and member of the ISO Technical Committee (TC69: Applications of Statistical methods) where he is co-author of the Lean & Six Sigma Standard (ISO 18404) as well as the new standard under development (Design for Six Sigma). He is also part of the newly formed international Group on Big Data (nominated by BSI as the UK representative/expert). Radouane has also been Chair of the working group on Measurement Systems for Automated Processes/Systems within the ISPE (International Society for Pharmaceutical Engineering).

Automating Data Science over a Human Genomics Knowledge Base

Vaticle

Data is both our most valuable asset and our biggest ongoing challenge. As data grows in volume, variety and complexity, across applications, clouds and siloed systems, traditional ways of working with data no longer work. Unlike traditional databases, which arrange data in rows, columns and tables, Neo4j has a flexible structure defined by stored relationships between data records. We'll discuss the primary use cases for graph databases Explore the properties of Neo4j that make those use cases possible Look into the visualisation of graphs Introduce how to write queries. Webinar, 23 July 2020

Introduction to Neo4j

Neo4j

Big Data & Analytics continues to redefine business. Data has transitioned from an underused asset to the lifeblood of the organisation, and a critical component of business intelligence, insight and strategy. Big Data Scotland is the largest annual data analytics conference held in Scotland: it is supported by ScotlandIS and The Data Lab and free for delegates to attend. The conference is geared towards senior technologists and business leaders and aims to provide a unique forum for knowledge exchange, discussion and cross-pollination. The programme will explore the evolution of data analytics; looking at key tools and techniques and how these can be applied to deliver practical insight and value. Presentations will span a wide array of topics from Data Wrangling and Visualisation to AI, Chatbots and Industry 4.0. Key Topics • Tools and techniques • Corporate data culture, business processes, digital transformation • Business intelligence, trends, decision making • AI, Real-time Analytics, IoT, Industry 4.0, Robotics • Security, regulation, privacy, consent, anonymization • Data visualisation, interpretation and communication • CRM and Personalisation

Big Data Scotland 2017

Ray Bugg

Big Data Ecosystem

Ivo Vachkov

Combining a Knowledge Graph and Graph Algorithms to Find Hidden Skills at NASA

Neo4j

Big data ecosystem

magda3695

Banking organizations are having to deal with highly complex fraud rings today. Fraudsters are spreading their activities across a large number of transactions and geographical regions in a more coordinated fashion, making traditional anomaly identification methods nearly obsolete. Join us for this on-demand webinar with Jie Wu, Director of Product Marketing, DataStax and guest speaker Scott Heath, Chief Revenue Officer, Expero for a discussion on how to use real-time graph database to effectively detect fraud and reduce risk in real time. View recording: https://youtu.be/XzpTICtJ2Fk Explore all DataStax webinars: https://www.datastax.com/resources/webinars

Webinar - Fighting Bank Fraud with Real-time Graph Database

DataStax

Risk Analytics Using Knowledge Graphs / FIBO with Deep Learning

Cambridge Semantics

Mais procurados (20)

Apouc 2014-business-analytics-and-big-data

Unified Information Governance, Powered by Knowledge Graph

Streamline Data Governance with Egeria: The Industry's First Open Metadata St...

Data Science Application in Business Portfolio & Risk Management

Scalability and Graph Analytics with Neo4j - Stefan Kolmar, Neo4j

Should a Graph Database Be in Your Next Data Warehouse Stack?

Big Data Fabric 2.0 Drives Data Democratization

Using Cloud Automation Technologies to Deliver an Enterprise Data Fabric

Infochimps + CloudCon: Infinite Monkey Theorem

Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...

[Webinar] Measure Twice, Build Once: Real-Time Predictive Analytics

Importance of Big Data Analytics

Automating Data Science over a Human Genomics Knowledge Base

Introduction to Neo4j

Big Data Scotland 2017

Big Data Ecosystem

Combining a Knowledge Graph and Graph Algorithms to Find Hidden Skills at NASA

Big data ecosystem

Webinar - Fighting Bank Fraud with Real-time Graph Database

Risk Analytics Using Knowledge Graphs / FIBO with Deep Learning

Destaque

At Google Cloud Platform, we're combining the Apache Spark and Hadoop ecosystem with our software and hardware innovations. We want to make these awesome tools easier, faster, and more cost-effective, from 3 to 30,000 cores. This presentation will showcase how Google Cloud Platform is innovating with the goal of bringing the Hadoop ecosystem to everyone. Bio: "I love data because it surrounds us - everything is data. I also love open source software, because it shows what is possible when people come together to solve common problems with technology. While they are awesome on their own, I am passionate about combining the power of open source software with the potential unlimited uses of data. That's why I joined Google. I am a product manager for Google Cloud Platform and manage Cloud Dataproc and Apache Beam (incubating). I've previously spent time hanging out at Disney and Amazon. Beyond Google, love data, amateur radio, Disneyland, photography, running and Legos."

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop

huguk

See the new release: https://www.talend.com/products/talend-6/ Talend Summer ’17 delivers the latest cloud and big data innovations so you can get a 360-degree view of your customer across multiple cloud platforms. Accelerate AWS, Microsoft Azure and Google Cloud Platform adoption, with the flexibility and portability to easily reuse development work across the cloud. In this presentation, we break down new features for data quality, ESB, cloud, big data and more.

Talend Summer '17 Release: New Features and Tech Overview

Talend

Role of Analytics in Consumer Packaged Goods Industry

Perceptive Analytics

Extend Governance in Hadoop with Atlas Ecosystem: Waterline, Attivo & Trifacta

DataWorks Summit/Hadoop Summit

Préparation de Données Hadoop avec Trifacta

Victor Coustenoble

Like all consumer packaged goods (CPG) companies, PepsiCo relies on huge volumes of data to accurately replenish its retailers with the appropriate amount and type of product. Across the CPG industry, most analysts exclusively rely on Excel and Access for data wrangling, but as PepsiCo’s data surpassed the capabilities of those tools, they knew they needed a better way. https://www.brighttalk.com/webcast/9573/227935

How PepsiCo's Big Data Strategy is Disrupting CPG Retail Analytics

Hortonworks

Apache Atlas: Tracking dataset lineage across Hadoop components

DataWorks Summit/Hadoop Summit

Comparison of Data Preparation vs. Data Wrangling Programming Languages, Frameworks and Tools in Machine Learning / Deep Learning Projects. A key task to create appropriate analytic models in machine learning or deep learning is the integration and preparation of data sets from various sources like files, databases, big data storages, sensors or social networks. This step can take up to 80% of the whole project. This session compares different alternative techniques to prepare data, including extract-transform-load (ETL) batch processing (like Talend, Pentaho), streaming analytics ingestion (like Apache Storm, Flink, Apex, TIBCO StreamBase, IBM Streams, Software AG Apama), and data wrangling (DataWrangler, Trifacta) within visual analytics. Various options and their trade-offs are shown in live demos using different advanced analytics technologies and open source frameworks such as R, Python, Apache Hadoop, Spark, KNIME or RapidMiner. The session also discusses how this is related to visual analytics tools (like TIBCO Spotfire), and best practices for how the data scientist and business user should work together to build good analytic models. Key takeaways for the audience: - Learn various options for preparing data sets to build analytic models - Understand the pros and cons and the targeted persona for each option - See different technologies and open source frameworks for data preparation - Understand the relation to visual analytics and streaming analytics, and how these concepts are actually leveraged to build the analytic model after data preparation Video Recording / Screencast of this Slide Deck: https://youtu.be/2MR5UynQocs

Data Preparation vs. Inline Data Wrangling in Data Science and Machine Learning

Kai Wähner

Destaque (8)

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop

Talend Summer '17 Release: New Features and Tech Overview

Role of Analytics in Consumer Packaged Goods Industry

Extend Governance in Hadoop with Atlas Ecosystem: Waterline, Attivo & Trifacta

Préparation de Données Hadoop avec Trifacta

How PepsiCo's Big Data Strategy is Disrupting CPG Retail Analytics

Apache Atlas: Tracking dataset lineage across Hadoop components

Data Preparation vs. Inline Data Wrangling in Data Science and Machine Learning

Semelhante a Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

Democratizing Advanced Analytics Propels Instant Analysis Results to the Ubiq...

Dana Gardner

Benefiting from Big Data - A New Approach for the Telecom Industry

Persontyle

In today’s context, the big data market is rapidly undergoing contortions that define market maturity, such as consolidation. Big data refers to large volumes of data. This can be both structured and unstructured data. Big data is data that is huge in size and grows exponentially with time. As the data is too large and complex, traditional data management tools are not sufficient for storing or processing it efficiently. But analyzing big data is crucial to know the patterns and trends to be adopted to improve your business.

Top 10 renowned big data companies

Robert Smith

Insights Success is The Best Business Magazine in the world for enterprises. Being a platform, it focuses distinctively on emerging as well as leading fastest growing companies, their confrontational style of doing businesses and the way of delivering effective and collaborative solutions to strengthen market share. Here, we talk about the leader’s viewpoints & ideas, latest products/services, etc. Insights Success magazine reaches out to all the ‘C’ Level Professionals, VPs, Consultants, VCs, Managers, and HRs of various industries.

Insights success the 10 best hadoop solution provider companies nov 2017

Insights success media and technology pvt ltd

Alitora Innovation Networks

alitora

Seminario OpenStack - Universita' Tor Vergata - Cattedra Cloud Computing

Fausto Pasqualetti

World’s 10 Best Data Integration Solution Providers 2022.pdf

InsightsSuccess4

The Long Road of IT Systems Management Enters the Domain of AIOps-Fueled Auto...

Dana Gardner

AI is moving from its academic roots to the forefront of business and industry

Digital Transformation EXPO Event Series

GraphTalks Frankfurt - Graph Database Überblick

Neo4j

Watch the full session: Denodo DataFest 2016 sessions: https://goo.gl/yVJnti Data virtualization starts with democratizing data access for business users, but goes well beyond to enable entire analytics life cycle. This session will discuss the critical role of data virtualization in the four key phases of big data analytics: Discovery of raw and enriched data, Analytic Exploration, Real-time Operationalization, and Predictive Intervention. In this session, you will learn: • Design of advanced analytics with view towards business goal realization • The role of data virtualization in enabling analytics through four key phases • How to exploit product capabilities relevant to each stage • Creating a system of governed self-service and collaborative analytics This session is part of the Denodo DataFest 2016 event. You can also watch more Denodo DataFest sessions on demand here: https://goo.gl/VXb6M6

Denodo DataFest 2016: Data Science: Operationalizing Analytical Models in Rea...

Denodo

As more companies grow their business in global markets, they discover the need to capture new opportunities in a matter of days rather than months to have competitive advantage and to capture new market share. Their machines are producing terabytes of various data types — video, audio, Microsoft® SharePoint®, sensor data, Microsoft Excel® files — and leaders are searching for the right technologies to capture this data and help provide a better understanding of their business. The HDS big data product roadmap will help customers build a big data enterprise plan that ingests data faster and correlate meaningful data sets to create intelligence that’s easy to consume and helps leaders make the right business decisions. View this webcast to learn about Hitachi’s product roadmap to big data. For more information on HDS Big Data Solutions please visit: http://www.hds.com/solutions/it-strategies/big-data/?WT.ac=us_mg_sol_bigdat

Hitachi Data Systems Big Data Roadmap

Hitachi Vantara

Record manager 8.0 presentation

Andrey Karpov

Modernizing Your IT Infrastructure with Hadoop - Cloudera Summer Webinar Seri...

Cloudera, Inc.

HP Software Performance Tour 2014 - Vincere i Big Data con HP HAVEn

HP Enterprise Italia

How HudsonAlpha Transforms Hybrid Cloud Deployment Complexity Into a Manageme...

Dana Gardner

SIMPosium presentation_Bardess Qlik

Bardess Group

Even from the “man in the street” perspective, there is a sense that we are living in an increasingly algorithmic world. Self-driving cars, pizza delivery by drone, and smart houses are commonplace. The technologies enabling this revolution are both simultaneously mature and evolving rapidly. In this session, we’ll took a look at a real world problem, the recent global outbreak of the ZIka virus, and used data analytics technologies to gain valuable insights that can assist authorities and the general public to understand and potentially prevent the spread of this disease. Bardess Group, a sponsor of the event and business analytics consulting firm, will demonstrate how huge, extremely jagged data from a variety of sources can be collected and prepared and rapidly made available for analysis. Advanced machine learning and predictive analysis further enhance the value of those insights. Finally, Bardess will make the case that using a systematic approach to conceptually visualize the strategic journey to insightful business analytics, the analytics value chain, can assist any organization prepare for this revolution in analytics. Also see http://cloudera.qlik.com for the demos.

Revolution in Business Analytics-Zika Virus Example

Bardess Group

It is clear that all employees must have access to data wherever they are to make decisions. However, tools that allow to share data just as easily as the best collaborative tools such as a google doc or an office 365 should be used. Open source driven by the big data ecosystem and a number of large companies have provided solutions to allow organizations to federate data systems and secure their access. After a quick overview of existing open source solutions, and how such projects can be organized, it will be necessary to detail Dremio implementation, a unique and centralized interface on all your data. Some real feedbacks will conclude the presentation.

All data accessible to all my organization - Presentation at OW2con'19, June...

OW2

Big Data Analytics

Sreedhar Chowdam

Semelhante a Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta (20)

Democratizing Advanced Analytics Propels Instant Analysis Results to the Ubiq...

Benefiting from Big Data - A New Approach for the Telecom Industry

Top 10 renowned big data companies

Insights success the 10 best hadoop solution provider companies nov 2017

Alitora Innovation Networks

Seminario OpenStack - Universita' Tor Vergata - Cattedra Cloud Computing

World’s 10 Best Data Integration Solution Providers 2022.pdf

The Long Road of IT Systems Management Enters the Domain of AIOps-Fueled Auto...

AI is moving from its academic roots to the forefront of business and industry

GraphTalks Frankfurt - Graph Database Überblick

Denodo DataFest 2016: Data Science: Operationalizing Analytical Models in Rea...

Hitachi Data Systems Big Data Roadmap

Record manager 8.0 presentation

Modernizing Your IT Infrastructure with Hadoop - Cloudera Summer Webinar Seri...

HP Software Performance Tour 2014 - Vincere i Big Data con HP HAVEn

How HudsonAlpha Transforms Hybrid Cloud Deployment Complexity Into a Manageme...

SIMPosium presentation_Bardess Qlik

Revolution in Business Analytics-Zika Virus Example

All data accessible to all my organization - Presentation at OW2con'19, June...

Big Data Analytics

Mais de huguk

Stephen Taylor is the community manager for Ether Camp. They provide an analysis tool for the Ethereum blockchain, ‘Block Explorer’ and also an ‘Intergrated Development Environment’ (I.D.E) that empowers developers to build, test and deploy applications in a sandbox environment. This November they are launching their second annual hackathon, hack.ether.camp which is aiming to deliver a more sustained approach to the hackathon ideology, by utilising blockchain technology.

ether.camp - Hackathon & ether.camp intro

huguk

Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...

huguk

Big organisations have a wealth of rich customer data which opens up huge new opportunities. However, they have the challenge of how to extract value from this data while protecting the privacy of their individual customers. He will talk about the risks organisations face, and what they should do about it. He will survey the techniques which can be used to make data safe for analysis, and talk briefly about how they are solving this problem at Privitar.

Extracting maximum value from data while protecting consumer privacy. Jason ...

huguk

IBM is developing the Watson Ecosystem to leverage its Developer Cloud, APIs, Content Store and Talent Hub. This is part of IBM's recent announcement of the $1B investment in Watson as a new business unit including Silicon Alley NYC headquarters. For the first time, IBM will open up Watson as a development platform in the Cloud to spur innovation and fuel a new ecosystem of entrepreneurial software app providers who will bring forward a new generation of applications infused with Watson's cognitive computing intelligence.

Intelligence Augmented vs Artificial Intelligence. Alex Flamant, IBM Watson

huguk

In this talk about Apache Flink we will touch on three main things, an introductory look at Flink, a look under the hood and a demo. * In the introduction we will briefly look at the history of Flink and then go on to the API and different use cases. Here we will also see how it can be deployed in practice and what some of the pitfalls in a cluster setting can be. * In the second section we will look at the streaming execution engine that lies at the heart of Flink. Here we will see what makes it tick and also what distinguishes it from other approaches, such as the mini-batch execution model. Ufuk Celebi - PMC member at Apache Flink and co-founder and software engineer at data Artisans * In the final section we will see a live demo of a fault-tolerant streaming job that performs analysis of the wikipedia edit-stream.

Streaming Dataflow with Apache Flink

huguk

Sean Owen – Director of Data Science @Cloudera Building machine learning models is all well and good, but how do they get productionized into a service? It's a long way from a Python script on a laptop, to a fault-tolerant system that learns continuously, serves thousands of queries per second, and scales to terabytes. The confederation of open source technologies we know as Hadoop now offers data scientists the raw materials from which to assemble an answer: the means to build models but also ingest data and serve queries, at scale. This short talk will introduce Oryx 2, a blueprint for building this type of service on Hadoop technologies. It will survey the problem and the standard technologies and ideas that Oryx 2 combines: Apache Spark, Kafka, HDFS, the lambda architecture, PMML, REST APIs. The talk will touch on a key use case for this architecture -- recommendation engines.

Lambda architecture on Spark, Kafka for real-time large scale ML

huguk

Martin Oberhuber and Eliano Marques, Senior Data Scientists @Think Big International In this talk Think Big International Lead Data Scientists will discuss the options that exist today for engineering and data science teams aiming to use big data patterns to solve new business problems. With the enterprise adoption of the Hadoop ecosystem and the emerging momentum of open source projects like Spark it is becoming mandatory to have an approach that solves for business results but remains flexible to adapt and change with the open source market.

Today’s reality Hadoop with Spark- How to select the best Data Science approa...

huguk

Jonathon Southam: Venture Capital, Funding & Pitching

huguk

Signal Media: Real-Time Media & News Monitoring

huguk

Dean Bryen: Scaling The Platform For Your Startup

huguk

Peter Karney: Intro to the Digital catapult

huguk

Cytora: Real-Time Political Risk Analysis

huguk

Cubitic: Predictive Analytics

huguk

Bird.i: Earth Observation Data Made Social

huguk

Aiseedo: Real Time Machine Intelligence

huguk

This talk will cover the design and implementation decisions that have been key to the success of Apache Spark over other competing cluster computing frameworks. It will be delving into the whitepaper behind Spark and cover the design of Spark RDDs, the abstraction enables the Spark execution engine to be extended to support a wide variety of use cases: Spark SQL, Spark Streaming, MLib and GraphX. RDDs allow Spark to outperform existing models by up to 100x in multi-pass analytics.

Secrets of Spark's success - Deenar Toraskar, Think Reactive

huguk

Technical developments in the area of data warehousing have allowed companies to push their analysis a step further and, therefore, allowed data scientists to deliver more value to business areas. In that session, we will focus on the case of performance marketing at King and demonstrate how we use Hadoop capabilities to exploit user-level data efficiently. That approach results in obtaining a more holistic view in a return-on-investment analysis of TV advertisement.

TV Marketing and big data: cat and dog or thick as thieves? Krzysztof Osiewal...

huguk

Hadoop - Looking to the Future By Arun Murthy (Founder of Hortonworks, Creator of YARN) The Apache Hadoop ecosystem began as just HDFS & MapReduce nearly 10 years ago in 2006. Very much like the Ship of Theseus (http://en.wikipedia.org/wiki/Ship_of_Theseus), Hadoop has undergone incredible amount of transformation from multi-purpose YARN to interactive SQL with Hive/Tez to machine learning with Spark. Much more lies ahead: whether you want sub-second SQL with Hive or use SSDs/Memory effectively in HDFS or manage Metadata-driven security policies in Ranger, the Hadoop ecosystem in the Apache Software Foundation continues to evolve to meet new challenges and use-cases. Arun C Murthy has been involved with Apache Hadoop since the beginning of the project - nearly 10 years now. In the beginning he led MapReduce, went on to create YARN and then drove Tez & the Stinger effort to get to interactive & sub-second Hive. Recently he has been very involved in the Metadata and Governance efforts. In between he founded Hortonworks, the first public Hadoop distribution company.

Hadoop - Looking to the Future By Arun Murthy

huguk

By Kevin Schmidt (Head of Data Science at Mind Candy) Luis Vicente (Senior Data Engineer at Mind Candy) For mobile games, constant tweaks are the difference between success and failure. Data and analytics have to be available in real-time, but calculating, for example, uniqueness or newness of a data point requires a list of seen data points - both memory intensive and tricky when using real-time stream processing like Spark Streaming. Probabilistic data structures allow approximation of these properties with a fixed memory representation, and are very well suited for this kind of stream processing. Getting from the theory of approximation to a useful metric at a low error rate even for many millions of users is another story. In our talk we will look at practical ways of achieving this: which approximation we used for selection of useful metrics, why we picked a specific probabilistic data structure, how we stored it in Cassandra as a time series and how we implemented it in Spark Streaming.

Fast real-time approximations using Spark streaming

huguk

The task of “data profiling”—assessing the overall content and quality of a data set—is a core aspect of the analytic experience. Traditionally, profiling was a fairly cut-and-dried task: load the raw numbers into a stat package, run some basic descriptive statistics, and report the output in a summary file or perhaps a simple data visualization. However, data volumes can be so large today that traditional tools and methods for computing descriptive statistics become intractable; even with scalable infrastructure like Hadoop, aggressive optimization and statistical approximation techniques must be used. In this talk Sean will cover technical challenges in keeping data profiling agile in the Big Data era. He will discuss both research results and real-world best practices used by analysts in the field, including methods for sampling, summarizing and sketching data, and the pros and cons of using these various approaches. Sean is Trifacta’s Chief Technical Officer. He completed his Ph.D. at Stanford University, where his research focused on user interfaces for database systems. At Stanford, Sean led development of new tools for data transformation and discovery, such as Data Wrangler. He previously worked as a data analyst at Citadel Investment Group.

Sean Kandel - Data profiling: Assessing the overall content and quality of a ...

huguk

Mais de huguk (20)

ether.camp - Hackathon & ether.camp intro

Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...

Extracting maximum value from data while protecting consumer privacy. Jason ...

Intelligence Augmented vs Artificial Intelligence. Alex Flamant, IBM Watson

Streaming Dataflow with Apache Flink

Lambda architecture on Spark, Kafka for real-time large scale ML

Today’s reality Hadoop with Spark- How to select the best Data Science approa...

Jonathon Southam: Venture Capital, Funding & Pitching

Signal Media: Real-Time Media & News Monitoring

Dean Bryen: Scaling The Platform For Your Startup

Peter Karney: Intro to the Digital catapult

Cytora: Real-Time Political Risk Analysis

Cubitic: Predictive Analytics

Bird.i: Earth Observation Data Made Social

Aiseedo: Real Time Machine Intelligence

Secrets of Spark's success - Deenar Toraskar, Think Reactive

TV Marketing and big data: cat and dog or thick as thieves? Krzysztof Osiewal...

Hadoop - Looking to the Future By Arun Murthy

Fast real-time approximations using Spark streaming

Sean Kandel - Data profiling: Assessing the overall content and quality of a ...

Último

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Delhi Call girls

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Presentation on how to chat with PDF using ChatGPT code interpreter

naman860154

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “The Role of Taxonomy and Ontology in Semantic Layers” at a webinar hosted by Progress Semaphore on April 16, 2024. Taxonomies at their core enable effective tagging and retrieval of content, and combined with ontologies they extend to the management and understanding of related data. There are even greater benefits of taxonomies and ontologies to enhance your enterprise information architecture when applying them to a semantic layer. A survey by DBP-Institute found that enterprises using a semantic layer see their business outcomes improve by four times, while reducing their data and analytics costs. Extending taxonomies to a semantic layer can be a game-changing solution, allowing you to connect information silos, alleviate knowledge gaps, and derive new insights. Hedden, who specializes in taxonomy design and implementation, presented how the value of taxonomies shouldn’t reside in silos but be integrated with ontologies into a semantic layer. Learn about: - The essence and purpose of taxonomies and ontologies in information and knowledge management; - Advantages of semantic layers leveraging organizational taxonomies; and - Components and approaches to creating a semantic layer, including the integration of taxonomies and ontologies

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Enterprise Knowledge

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Choosing the right accounts payable services provider is a strategic decision that can significantly impact your business's financial performance and operational efficiency. By considering factors such as expertise, range of services, technology infrastructure, scalability, cost, and reputation, businesses can make informed decisions and select a provider that aligns with their unique needs and objectives. Partnering with the right provider can streamline accounts payable processes, drive cost savings, and position your business for long-term success. https://katprotech.com/accounts-payable-and-purchase-order-automation/

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Katpro Technologies

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

CNv6 Instructor Chapter 6 Quality of Service

giselly40

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

In an era where artificial intelligence (AI) stands at the forefront of business innovation, Information Architecture (IA) is at the core of functionality. See “There’s No AI Without IA” – (from 2016 but even more relevant today) Understanding and leveraging how Information Architecture (IA) supports AI synergies between knowledge engineering and prompt engineering is critical for senior leaders looking to successfully deploy AI for internal and externally facing knowledge processes. This webinar be a high-level overview of the methodologies that can elevate AI-driven knowledge processes supporting both employees and customers. Core Insights Include: Strategic Knowledge Engineering: Delve into how structuring AI's knowledge base is required to prevent hallucinations, enable contextual retrieval of accurate information. This will include discussion of gold standard libraries of use cases support testing various LLMs and structures and configurations of knowledge base. Precision in Prompt Engineering: Learn the art of crafting prompts that direct AI to deliver targeted, relevant responses, thereby optimizing customer experiences and business outcomes. Unified Approach for Enhanced AI Performance: Explore the intersection of knowledge and prompt engineering to develop AI systems that are not only more responsive but also aligned with overarching business strategies. Guiding Principles for Implementation: Equip yourself with best practices, ethical guidelines, and strategic considerations for embedding these technologies into your business ecosystem effectively. This webinar is designed to empower business and technology leaders with the knowledge to harness the full potential of AI, ensuring their organizations not only keep pace with digital transformation but lead the charge. Join us to map a roadmap to fully leverage Information Architecture (IA) and AI chart a course towards a future where AI is a key pillar of strategic innovation and business success.

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Earley Information Science

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Explore 'The Codex of Business: Writing Software for Real-World Solutions,' a compelling SlideShare presentation that delves into digital transformation in healthcare. Discover through a detailed case study how Agile methodologies empower healthcare providers to develop, iterate, and refine digital solutions that address real-world challenges. Learn how strategic planning, user feedback, and continuous improvement drive success in deploying technologies that enhance patient care and operational efficiency. Ideal for healthcare professionals, IT specialists, and digital transformation advocates seeking actionable insights and practical examples of technology making a real difference.

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Malak Abu Hammad

Microsoft's Threat Matrix for Kubernetes helps organizations understand the attack surface a Kubernetes deployment introduces to their environments. This ensures that adequate detections and mitigations are in place. By covering over 40 different attacker techniques, defenders can learn about Kubernetes-specific mitigations and controls to deploy to their environments. In this session, we will explore the MS-TA9013 Host Path Mount technique, which is commonly used by attackers to perform privilege escalation in a Kubernetes cluster. Attendees will learn how attackers and defenders can: * Escape the container's host volume mount to gain persistence on an underlying node * Move laterally from the underlying node into the customer's cloud environment * Analyze Kubernetes audit logs to detect pods deployed with a hostPath mount * Deploy an admission controller that prevents new pods from using a hostPath mount

Breaking the Kubernetes Kill Chain: Host Path Mount

Puma Security, LLC

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

1. Hadoop User Group London: Data Wrangling on Hadoop September 8 2016 Olivier de Garrigues, EMEA Solutions Lead

2. Creating radical productivity for people who analyze data. JEFFREY HEER Co-Founder & CXO VISUALIZATION JOE HELLERSTEIN Co-Founder & CSO BIG DATA SEAN KANDEL Co-Founder & CTO HUMAN-COMPUTER INTERACTION

3. 3 3,000+ Companies 10,000+ Users

4. What is Data Wrangling? 4 QUESTION ANALYZE INSIGHTDISCOVER STRUCTURE CLEANSE ENRICH VALIDATE PUBLISH

5. The Bridge Between Raw Data & Analysis 5 v Ingestion Storage Processing ANALYSIS & VISUALIZATION LOBCLEANING ENRICHMENT DISTILLATIONSTRUCTURINGDISCOVERY End-User Capabilities IT GOVERNANCE INTEGRATION AVAILABILTIYSCALABILITYSECURITY Technical Capabilities

6. Conventional Approaches Inhibit User Empowerment Hand-Coding Technical Workﬂow Mapping

7. Trifacta Approach: It’s All About The Experience Interact Predict Preview

8. Data Wrangling for Financial Fraud

9. TRIFACTA DATA WRANGLING WORKFLOW Trifacta. Confidential & Proprietary. Sample Scale Up Reﬁne Sample Results Identify/Register Data 1. Predictive Interaction 2 . Consume Schedulers Monitor and Adjust 3 . Schedule Visualization & Analysis Secure Access

10. Ingestion Processing Storage ANALYSIS & CONSUMPTION v Discover Structure Clean Enrich Distill LOB IT News Topics Time Trades Tickers Date $ eMails Recipients Topics Phone Logs Call Details Recipients Corporations Company Relations Individuals Financial Services use case: Trader Fraud

11. Data Wrangling Beneﬁts ➔  Empower the people who know the data best ➔  Accelerate time to value ➔  Lower business risk with more accurate data ➔  Unlock innovation using a wider variety of data

Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (8)

Semelhante a Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

Semelhante a Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta (20)

Mais de huguk

Mais de huguk (20)

Último

Último (20)

Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta