Analyzing Large-Scale User Data with Hadoop and HBase

•

0 gostou•527 visualizações

WibiData

WibiData's presentation on personalization and large-scale user data at Structure:Data 2012

Tecnologia Negócios

Analyzing Large-Scale User Data
with Hadoop and HBase

Aaron Kimball – CTO

WibiData, Inc.

We can now collect more
data than at any time in
history.

Yesterday’s engineering challenge:
Fitting the problem into the
hardware.

Today’s constrained
resource is understanding.

How do we best apply data

…to better serving our users?

The best products are user-centric
• Intuitive UI
• Continuously learning
– Guided search
– Smarter recommendations
• More effective service

Requirements

1. Understand the user population

Requirements
2. Respond to
users in real time

Requirements

3. Support graceful data evolution

Large-scale data science is hard
• What does a user look like?
– What data is available about the user?
– Which features are important?
– Which features are correlated?
• How do I model this in MapReduce?
• How do I serve results in a timely fashion?

Tools of the trade
• Store all data about a user
in one place
• Support real-time get/put,
as well as MapReduce

Tools of the trade
• Use complex data types to
model complex data
• Support extended data
models over time
• Retain support for legacy
systems using older models

Tools of the trade
• Abstract computational
model away from MapReduce
• Support computation over all
users… or one user at a time

: for set-top boxes

Viewing/recording history

: for set-top boxes
Libraries
Device and User Analysis

Viewing/recording history

Personalized offers and
recommendations

: for set-top boxes
Libraries
Device and User Analysis

Viewing/recording history

Personalized offers and
recommendations

Analysis for
product roadmap

: for set-top boxes
Libraries
Device and User Analysis

Viewing/recording history

Personalized offers and
recommendations

Improved
Analysis for
reports for
product roadmap Tech support portal
advertisers

The future
• More personalization
• Adaptive UIs (self arranging dashboards)
• Targeted content, ads
• More effective customer service

Conclusions
• Applications are becoming increasingly user-
centric
• Data drives this capability, but harnessing it
requires a new distributed architecture
• The biggest challenge is allowing data
scientists to effectively leverage the data

www.wibidata.com / @wibidata
Aaron Kimball – aaron@wibidata.com

Mais conteúdo relacionado

Semelhante a Analyzing Large-Scale User Data with Hadoop and HBase

Software Programs for Data Analysis

unmgrc

Feature differences between OLAP tools used for BI applications, including Dynamic Cubes. View the webinar video recording and download this deck: http://www.senturus.com/resources/beyond-powerplay-choosing-right-olap-tool-bi-environment/ This presentation outlines important criteria that should be considered before investing significant hours (and dollars) in development of OLAP tools. As always, we give a candid review of relevant tool capabilities, show features in action and provide some pragmatic decision-making tips on how to make the right choice for your BI environment. Senturus, a business analytics consulting firm, has a resource library with hundreds of free recorded webinars, trainings, demos and unbiased product reviews. Take a look and share them with your colleagues and friends: http://www.senturus.com/resources/.

Beyond PowerPlay: Choose the Right OLAP Tool for Your BI Environment (Cognos...

Senturus

Knowage 8 presentation

KNOWAGE

Digitisation workshop pres 2009(v1)

Mal Booth

WibiData is a collaborative data mining and predictive modeling platform for large-scale, multi-structured, user-centric data. It leverages HBase to combine batch analysis and real time access within the same system, and integrates with existing BI, reporting and analysis tools. WibiData offers a set of libraries for common user-centric analytic tasks, and more advanced data mining libraries for personalization, recommendation, and other predictive modeling applications. Developers can write reusable libraries that are also accessible to data scientists and analysts alongside the WibiData libraries. In this talk, we will provide a technical overview of WibiData, and show how we used it to build FoneDoktor, a mobile app that collects data about device performance and app resource usage and offers personalized battery/performance improvement recommendations directly to users.

Hadoop World 2011: WibiData: Building Personalized Applications with HBase - ...

Cloudera, Inc.

Building Personalized Applications with HBase

WibiData

Discussion of practically implementing data science models on big data platforms from engineering perspective. An eye opener on the engineering factors associated with designing and working solution. We use a simple text mining example on social media analytics for brand marketing. At the first while, it seems simple solution however if you go deeply and think on implementation aspects of even a simple analytics model, you can discover the degree of complexity at each part of the solution. An Abstraction of the Big Data key advantages would be very helpful to select appropriate Big Data technology components out of very large landscape. Two examples with reference are given for using Lambda Architecture and unusual way of image processing using Big Data abstraction provided.

Engineering patterns for implementing data science models on big data platforms

Hisham Arafat

Self Service Reporting & Analytics For an Enterprise

Sreejith Madhavan

Business analytics and data visualisation

Shwetabh Jaiswal

Conceptual Design of TAPipedia

Nikos Manouselis

Big Data Introduction

Durga Gadiraju

1 introba

Claudia Gomez

Unbundling the Modern Streaming Stack With Dunith Dhanushka | Current 2022 Event-first thinking and streaming help organizations transition from followers to leaders in the market. A reliable, scalable, and economical streaming architecture helps them get there. This talk first explores the ""classic streaming stack,"" based on the Lambda architecture, its origin, and why it didn't pick up amongst data-driven organizations. The modern streaming stack (MSS) is a lean, cloud-native, and economical alternative to classic streaming architectures, where it aims to make event-driven real-time applications viable for organizations. The second half of the talk explores the MSS in detail, including its core components, their purposes, and how Kappa architecture has influenced it. Moreover, the talk lays out a few considerations before planning a new streaming application within an organization. The talk concludes by discussing the challenges in the streaming world and how vendors are trying to overcome them in the future.

Unbundling the Modern Streaming Stack With Dunith Dhanushka | Current 2022

HostedbyConfluent

How we use Hive at SnowPlow, and how the role of HIve is changing

yalisassoon

Mit der Architektur steht und fällt jedes IT-Projekt. Das gilt in noch stärkerem Maße für Big-Data-Projekte, denn hier konnten noch keine Standards über Jahrzehnte ihre Tauglichkeit beweisen. Dennoch verbreiten und etablieren sich auch hier gute und effektive Lösungen. Der Vortrag erklärt, welche Bausteine wichtig für die verschiedenen Einsatzmöglichkeiten im Big-Data-Umfeld sind, und wie sie in konkrete Lösungen gegossen werden können. Dabei beleuchtet er sowohl traditionelle Big-Data-Architekturen als auch aktuelle Ansätze, wie z. B. die Lambda- und die Kappa-Architektur. Ebenfalls ein Thema sind Stream-Processing-Infrastrukturen und ihre Kombination mit Big-Data-Technologien. Ausgehend von einer produkt- und technologieunabhängigen Referenzarchitektur stellt dieser Vortrag verschiedene Lösungsmöglichkeiten auf Basis von Open-Source-Komponenten vor.

Big Data Architectures @ JAX / BigDataCon 2016

Guido Schmutz

Offline and stream processing of big data sets can be done with tools such as Hadoop, Spark, and Storm, but what if you need to process big data at the time a user is making a request? This presentation introduces Vespa (http://vespa.ai) – the open source big data serving engine. Vespa allows you to search, organize, and evaluate machine-learned models from e.g TensorFlow over large, evolving data sets with latencies in the tens of milliseconds. Vespa is behind the recommendation, ad targeting, and search at Yahoo where it handles billions of daily queries over billions of documents and was recently open sourced at http://vespa.ai.

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Yahoo Developer Network

In memory analysis 衍華

Lawrence Huang

Birst for SAP HANA

Birst

Big data unit 2

RojaT4

Digitisation Workshop Pres 2008(V1)

Mal Booth

Semelhante a Analyzing Large-Scale User Data with Hadoop and HBase (20)

Software Programs for Data Analysis

Beyond PowerPlay: Choose the Right OLAP Tool for Your BI Environment (Cognos...

Knowage 8 presentation

Digitisation workshop pres 2009(v1)

Hadoop World 2011: WibiData: Building Personalized Applications with HBase - ...

Building Personalized Applications with HBase

Engineering patterns for implementing data science models on big data platforms

Self Service Reporting & Analytics For an Enterprise

Business analytics and data visualisation

Conceptual Design of TAPipedia

Big Data Introduction

1 introba

Unbundling the Modern Streaming Stack With Dunith Dhanushka | Current 2022

How we use Hive at SnowPlow, and how the role of HIve is changing

Big Data Architectures @ JAX / BigDataCon 2016

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

In memory analysis 衍華

Birst for SAP HANA

Big data unit 2

Digitisation Workshop Pres 2008(V1)

Último

The action of the next cyber saga takes place in the mystical lands of the Asia-Pacific region, where the main characters began their digital activities in the middle of 2021 and qualitatively strengthened it in 2022. Corporate espionage, document theft, audio recordings, and data leaks from messaging platforms were all a matter of one day for Dark Pink. Their geographical focus may have started in the Asia-Pacific region, but their ambitions knew no bounds, targeting a European government ministry in a bold move to expand their portfolio. Their victim profile was as diverse as a UN meeting, targeting military organizations, government agencies, and even a religious organization. Because discrimination is not a fashionable agenda. In the world of cybercrime, they serve as a reminder that sometimes the most serious threats come in the most unassuming packages with a pink bow.

Cyberprint. Dark Pink Apt Group [EN].pdf

Overkill Security

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

Exploring Multimodal Embeddings with Milvus

Zilliz

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Analyzing Large-Scale User Data with Hadoop and HBase

2. Analyzing Large-Scale User Data with Hadoop and HBase Aaron Kimball – CTO WibiData, Inc.

3. We can now collect more data than at any time in history.

4. Yesterday’s engineering challenge: Fitting the problem into the hardware.

5. Today’s constrained resource is understanding.

6. How do we best apply data …to better serving our users?

7. The best products are user-centric • Intuitive UI • Continuously learning – Guided search – Smarter recommendations • More effective service

8. What are we building toward?

9. What are we building toward?

10. What are we building toward?

11. What are we building toward?

12. What are we building toward?

13. Requirements 1. Understand the user population

14. Requirements 2. Respond to users in real time

15. Requirements 3. Support graceful data evolution

16. Large-scale data science is hard • What does a user look like? – What data is available about the user? – Which features are important? – Which features are correlated? • How do I model this in MapReduce? • How do I serve results in a timely fashion?

17.

18. Tools of the trade • Store all data about a user in one place • Support real-time get/put, as well as MapReduce

19. Tools of the trade • Use complex data types to model complex data • Support extended data models over time • Retain support for legacy systems using older models

20. Tools of the trade • Abstract computational model away from MapReduce • Support computation over all users… or one user at a time

21.

22.

23.

24.

25. : for set-top boxes Viewing/recording history

26. : for set-top boxes Libraries Device and User Analysis Viewing/recording history Personalized offers and recommendations

27. : for set-top boxes Libraries Device and User Analysis Viewing/recording history Personalized offers and recommendations Analysis for product roadmap

28. : for set-top boxes Libraries Device and User Analysis Viewing/recording history Personalized offers and recommendations Analysis for product roadmap Tech support portal

29. : for set-top boxes Libraries Device and User Analysis Viewing/recording history Personalized offers and recommendations Improved Analysis for reports for product roadmap Tech support portal advertisers

30. The future • More personalization • Adaptive UIs (self arranging dashboards) • Targeted content, ads • More effective customer service

31. Conclusions • Applications are becoming increasingly user- centric • Data drives this capability, but harnessing it requires a new distributed architecture • The biggest challenge is allowing data scientists to effectively leverage the data

32. www.wibidata.com / @wibidata Aaron Kimball – aaron@wibidata.com

Analyzing Large-Scale User Data with Hadoop and HBase

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Analyzing Large-Scale User Data with Hadoop and HBase

Semelhante a Analyzing Large-Scale User Data with Hadoop and HBase (20)

Último

Último (20)

Analyzing Large-Scale User Data with Hadoop and HBase