SlideShare uma empresa Scribd logo
1 de 36
Developing Data Products
Uber Tech Talk
Pete Skomoroch @peteskomoroch
December 5 2012


©2012 LinkedIn Corporation. All Rights Reserved.
Examples, Techniques, & Lessons Learned

Developing Data Products
Our Mission
                          Connect the world’s professionals to make them
                             more productive and successful.

Our Vision
Create economic opportunity for every
   professional in the world.


Members First!
LinkedIn is the leading professional network site



         187M+
                                           1




           LinkedIn Members

                                          2


         640M+        Worldwide
                   Professionals
                                           2


 3,300M+
      Worldwide Workforce




©2012 LinkedIn Corporation. All Rights Reserved.    4
LinkedIn profiles represent our professional identity




                                 1                             2




       187M                                        Members   187M   Member
                                                                    Profiles


©2012 LinkedIn Corporation. All Rights Reserved.                               5
We have a lot of data.




©2012 LinkedIn Corporation. All Rights Reserved.
We have a lot of data.
    And (like everyone else), we store it in Hadoop.




©2012 LinkedIn Corporation. All Rights Reserved.
We have a lot of data.
  And (like everyone else), we store it in Hadoop.
  And people build awesome things with that data.




©2012 LinkedIn Corporation. All Rights Reserved.
What do we mean by data
products?
Building products from data at LinkedIn

A few examples:

    People You May Know
    Skills and Endorsements
    Year in Review
    Network Updates Digest
    InMaps
    Who’s viewed my profile
    Collaborative Filtering
    Groups You May Like
    and more…




©2012 LinkedIn Corporation. All Rights Reserved.
Collaborative Filtering: LinkedIn Skill Pages




©2012 LinkedIn Corporation. All Rights Reserved.
Classification: giving structure to unstructured data




          Extract




©2012 LinkedIn Corporation. All Rights Reserved.
Clustering & Disambiguation




©2012 LinkedIn Corporation. All Rights Reserved.
De-duplication and Normalization




©2012 LinkedIn Corporation. All Rights Reserved.
Network Algorithms: Relevance & Ranking




©2012 LinkedIn Corporation. All Rights Reserved.   15
Prediction: Personalized Skill Recommendations




©2012 LinkedIn Corporation. All Rights Reserved.
Skill Endorsements




©2012 LinkedIn Corporation. All Rights Reserved.
Social Proof and the Skill Endorsement Graph




©2012 LinkedIn Corporation. All Rights Reserved.   20
The Economic Graph: Skills, Jobs, People, Locations…




                                                   Location



©2012 LinkedIn Corporation. All Rights Reserved.              21
Lessons learned developing data
products
Collect the right data at the right time
Large amounts of data can reveal new patterns
 Probability of Job Title




                                                   Months since graduation
©2012 LinkedIn Corporation. All Rights Reserved.                             24
Be wary of “black-box” approaches




©2012 LinkedIn Corporation. All Rights Reserved.   25
Look at your data




©2012 LinkedIn Corporation. All Rights Reserved.   26
Aggregate statistics can be misleading

       12




       10




        8




        6




        4




        2




        0
                 1             2            3      4   5   6   7   8   9   10




©2012 LinkedIn Corporation. All Rights Reserved.                                27
Build a viewer app, “micro-listen”




©2012 LinkedIn Corporation. All Rights Reserved.   28
Algorithmic intuition: include data geeks in design




©2012 LinkedIn Corporation. All Rights Reserved.      29
OODA: Think like a jet fighter




©2012 LinkedIn Corporation. All Rights Reserved.   30
OODA: Observe, Orient, Decide, Act




©2012 LinkedIn Corporation. All Rights Reserved.   31
OODA: The speed you can move determines victory




©2012 LinkedIn Corporation. All Rights Reserved.   32
Red teaming: what can go wrong likely will




©2012 LinkedIn Corporation. All Rights Reserved.   33
Error data is super valuable, analyze it and adapt




©2012 LinkedIn Corporation. All Rights Reserved.     34
Conclusion: tips for developing data products

    Collect the right data at the right time
    Large amounts of data can reveal new patterns
    Be wary of “black box” approaches
    Look at your raw data
    Aggregate statistics can be misleading
    Build and use viewer apps
    Include data geeks in design process
    OODA: Think like a jet fighter
    Red-teaming: anticipate edge cases
    Find opportunity in your error data




©2012 LinkedIn Corporation. All Rights Reserved.
Questions?

More info: data.linkedin.com
@peteskomoroch


©2012 LinkedIn Corporation. All Rights Reserved.   36

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Data Architecture Brief Overview
Data Architecture Brief OverviewData Architecture Brief Overview
Data Architecture Brief Overview
 
Data Catalog as the Platform for Data Intelligence
Data Catalog as the Platform for Data IntelligenceData Catalog as the Platform for Data Intelligence
Data Catalog as the Platform for Data Intelligence
 
5 Steps for Architecting a Data Lake
5 Steps for Architecting a Data Lake5 Steps for Architecting a Data Lake
5 Steps for Architecting a Data Lake
 
data-analytics-strategy-ebook.pptx
data-analytics-strategy-ebook.pptxdata-analytics-strategy-ebook.pptx
data-analytics-strategy-ebook.pptx
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)
 
Data Product Management by Tinder Group PM
Data Product Management by Tinder Group PMData Product Management by Tinder Group PM
Data Product Management by Tinder Group PM
 
Delivering Data Democratization in the Cloud with Snowflake
Delivering Data Democratization in the Cloud with SnowflakeDelivering Data Democratization in the Cloud with Snowflake
Delivering Data Democratization in the Cloud with Snowflake
 
Screw DevOps, Let's Talk DataOps
Screw DevOps, Let's Talk DataOpsScrew DevOps, Let's Talk DataOps
Screw DevOps, Let's Talk DataOps
 
BigID & Collibra Joint Deck: Using BigID’s Privacy-centric Data Discovery to...
BigID & Collibra Joint Deck: Using BigID’s Privacy-centric Data  Discovery to...BigID & Collibra Joint Deck: Using BigID’s Privacy-centric Data  Discovery to...
BigID & Collibra Joint Deck: Using BigID’s Privacy-centric Data Discovery to...
 
Data Product Architectures
Data Product ArchitecturesData Product Architectures
Data Product Architectures
 
Introduction to Azure Databricks
Introduction to Azure DatabricksIntroduction to Azure Databricks
Introduction to Azure Databricks
 
AI Data Acquisition and Governance: Considerations for Success
AI Data Acquisition and Governance: Considerations for SuccessAI Data Acquisition and Governance: Considerations for Success
AI Data Acquisition and Governance: Considerations for Success
 
Data Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and GovernanceData Catalog for Better Data Discovery and Governance
Data Catalog for Better Data Discovery and Governance
 
Business intelligence
Business intelligenceBusiness intelligence
Business intelligence
 
BI-Analytics-Overview.pptx
BI-Analytics-Overview.pptxBI-Analytics-Overview.pptx
BI-Analytics-Overview.pptx
 
Enterprise guide to building a Data Mesh
Enterprise guide to building a Data MeshEnterprise guide to building a Data Mesh
Enterprise guide to building a Data Mesh
 
Data Architecture Strategies
Data Architecture StrategiesData Architecture Strategies
Data Architecture Strategies
 
Datorama - NOAH15 London
Datorama - NOAH15 LondonDatorama - NOAH15 London
Datorama - NOAH15 London
 
Business Intelligence & Data Analytics– An Architected Approach
Business Intelligence & Data Analytics– An Architected ApproachBusiness Intelligence & Data Analytics– An Architected Approach
Business Intelligence & Data Analytics– An Architected Approach
 
Data Lake Overview
Data Lake OverviewData Lake Overview
Data Lake Overview
 

Semelhante a Developing Data Products

Big Data World 2013 - How LinkedIn leveraged its data to become the world's l...
Big Data World 2013 - How LinkedIn leveraged its data to become the world's l...Big Data World 2013 - How LinkedIn leveraged its data to become the world's l...
Big Data World 2013 - How LinkedIn leveraged its data to become the world's l...
Vitaly Gordon
 
7 Badass Tactics for SlideShare Content Domination
7 Badass Tactics for SlideShare Content Domination7 Badass Tactics for SlideShare Content Domination
7 Badass Tactics for SlideShare Content Domination
LinkedIn
 
7 Badass SlideShare Tactics - Jason Miller (Social Fresh WEST 2013)
7 Badass SlideShare Tactics - Jason Miller (Social Fresh WEST 2013)7 Badass SlideShare Tactics - Jason Miller (Social Fresh WEST 2013)
7 Badass SlideShare Tactics - Jason Miller (Social Fresh WEST 2013)
Social Fresh Conference
 
Are Your LinkedIn or Linked Out?
Are Your LinkedIn or Linked Out?Are Your LinkedIn or Linked Out?
Are Your LinkedIn or Linked Out?
virallysocial
 
Enterprise Search, more relevant now than ever
Enterprise Search, more relevant now than everEnterprise Search, more relevant now than ever
Enterprise Search, more relevant now than ever
Mike Davis
 
Big data arch_analytics
Big data arch_analyticsBig data arch_analytics
Big data arch_analytics
Srinu Adira
 
The Data Effect: Canadian Big Data & Analytics Update - Dr. Alison Brooks Dir...
The Data Effect: Canadian Big Data & Analytics Update - Dr. Alison Brooks Dir...The Data Effect: Canadian Big Data & Analytics Update - Dr. Alison Brooks Dir...
The Data Effect: Canadian Big Data & Analytics Update - Dr. Alison Brooks Dir...
CityAge
 
Partner - Talent Solutions - Staffing Agency
Partner - Talent Solutions - Staffing AgencyPartner - Talent Solutions - Staffing Agency
Partner - Talent Solutions - Staffing Agency
LinkedIn_EMEA_Events
 
Partner - Talent Solutions - Corporate
Partner - Talent Solutions - CorporatePartner - Talent Solutions - Corporate
Partner - Talent Solutions - Corporate
LinkedIn_EMEA_Events
 
Tamm & kitt
Tamm & kittTamm & kitt
Tamm & kitt
Jeff Roy
 
Office 2012 LinkedIn slides
Office 2012 LinkedIn slidesOffice 2012 LinkedIn slides
Office 2012 LinkedIn slides
Luke Williams
 
Business Networking on LinkedIn
Business Networking on LinkedInBusiness Networking on LinkedIn
Business Networking on LinkedIn
Realty Profiler
 

Semelhante a Developing Data Products (20)

SF Data Science: Developing Data Products
SF Data Science: Developing Data ProductsSF Data Science: Developing Data Products
SF Data Science: Developing Data Products
 
Big Data World 2013 - How LinkedIn leveraged its data to become the world's l...
Big Data World 2013 - How LinkedIn leveraged its data to become the world's l...Big Data World 2013 - How LinkedIn leveraged its data to become the world's l...
Big Data World 2013 - How LinkedIn leveraged its data to become the world's l...
 
Computing Professional Identity for the Economic Graph
Computing Professional Identity for the Economic GraphComputing Professional Identity for the Economic Graph
Computing Professional Identity for the Economic Graph
 
7 Badass Tactics for SlideShare Content Domination
7 Badass Tactics for SlideShare Content Domination7 Badass Tactics for SlideShare Content Domination
7 Badass Tactics for SlideShare Content Domination
 
7 Badass Tactics for Slideshare Content Domination
7 Badass Tactics for Slideshare Content Domination 7 Badass Tactics for Slideshare Content Domination
7 Badass Tactics for Slideshare Content Domination
 
Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.
Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips. Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.
Crowdsourcing Series: LinkedIn. By Vitaly Gordon & Patrick Philips.
 
7 Badass SlideShare Tactics - Jason Miller (Social Fresh WEST 2013)
7 Badass SlideShare Tactics - Jason Miller (Social Fresh WEST 2013)7 Badass SlideShare Tactics - Jason Miller (Social Fresh WEST 2013)
7 Badass SlideShare Tactics - Jason Miller (Social Fresh WEST 2013)
 
Big Data Ecosystem @ LinkedIn
Big Data Ecosystem @ LinkedInBig Data Ecosystem @ LinkedIn
Big Data Ecosystem @ LinkedIn
 
Are Your LinkedIn or Linked Out?
Are Your LinkedIn or Linked Out?Are Your LinkedIn or Linked Out?
Are Your LinkedIn or Linked Out?
 
Enterprise Search, more relevant now than ever
Enterprise Search, more relevant now than everEnterprise Search, more relevant now than ever
Enterprise Search, more relevant now than ever
 
Big data arch_analytics
Big data arch_analyticsBig data arch_analytics
Big data arch_analytics
 
The Data Effect: Canadian Big Data & Analytics Update - Dr. Alison Brooks Dir...
The Data Effect: Canadian Big Data & Analytics Update - Dr. Alison Brooks Dir...The Data Effect: Canadian Big Data & Analytics Update - Dr. Alison Brooks Dir...
The Data Effect: Canadian Big Data & Analytics Update - Dr. Alison Brooks Dir...
 
Building the perfect profile on LinkedIn
Building the perfect profile on LinkedInBuilding the perfect profile on LinkedIn
Building the perfect profile on LinkedIn
 
Solving Biz Problems with SugarExchange: Session 9: How to Run Contributor Ca...
Solving Biz Problems with SugarExchange: Session 9: How to Run Contributor Ca...Solving Biz Problems with SugarExchange: Session 9: How to Run Contributor Ca...
Solving Biz Problems with SugarExchange: Session 9: How to Run Contributor Ca...
 
Partner - Talent Solutions - Staffing Agency
Partner - Talent Solutions - Staffing AgencyPartner - Talent Solutions - Staffing Agency
Partner - Talent Solutions - Staffing Agency
 
Remarkable Content for AMA New Orleans, Nov 2014
Remarkable Content for AMA New Orleans, Nov 2014Remarkable Content for AMA New Orleans, Nov 2014
Remarkable Content for AMA New Orleans, Nov 2014
 
Partner - Talent Solutions - Corporate
Partner - Talent Solutions - CorporatePartner - Talent Solutions - Corporate
Partner - Talent Solutions - Corporate
 
Tamm & kitt
Tamm & kittTamm & kitt
Tamm & kitt
 
Office 2012 LinkedIn slides
Office 2012 LinkedIn slidesOffice 2012 LinkedIn slides
Office 2012 LinkedIn slides
 
Business Networking on LinkedIn
Business Networking on LinkedInBusiness Networking on LinkedIn
Business Networking on LinkedIn
 

Mais de Peter Skomoroch

Executive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you thinkExecutive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you think
Peter Skomoroch
 

Mais de Peter Skomoroch (15)

Bridging the AI Gap: Building Stakeholder Support
Bridging the AI Gap: Building Stakeholder SupportBridging the AI Gap: Building Stakeholder Support
Bridging the AI Gap: Building Stakeholder Support
 
Managing Machines: The New AI Dev Stack
Managing Machines: The New AI Dev StackManaging Machines: The New AI Dev Stack
Managing Machines: The New AI Dev Stack
 
Product Management for AI
Product Management for AIProduct Management for AI
Product Management for AI
 
Executive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you thinkExecutive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you think
 
Building Competitive Moats With Data
Building Competitive Moats With DataBuilding Competitive Moats With Data
Building Competitive Moats With Data
 
O'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data ExhaustO'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data Exhaust
 
Skills, Reputation, and Search
Skills, Reputation, and SearchSkills, Reputation, and Search
Skills, Reputation, and Search
 
LinkedIn Endorsements: Reputation, Virality, and Social Tagging
LinkedIn Endorsements: Reputation, Virality, and Social TaggingLinkedIn Endorsements: Reputation, Virality, and Social Tagging
LinkedIn Endorsements: Reputation, Virality, and Social Tagging
 
Practical Problem Solving with Data - Onlab Data Conference, Tokyo
Practical Problem Solving with Data - Onlab Data Conference, TokyoPractical Problem Solving with Data - Onlab Data Conference, Tokyo
Practical Problem Solving with Data - Onlab Data Conference, Tokyo
 
Street Fighting Data Science
Street Fighting Data ScienceStreet Fighting Data Science
Street Fighting Data Science
 
Data Mashups -Data Science Summit
Data Mashups -Data Science SummitData Mashups -Data Science Summit
Data Mashups -Data Science Summit
 
Geo Analytics Tutorial - Where 2.0 2011
Geo Analytics Tutorial - Where 2.0 2011Geo Analytics Tutorial - Where 2.0 2011
Geo Analytics Tutorial - Where 2.0 2011
 
Rapid Data Exploration With Hadoop
Rapid Data Exploration With HadoopRapid Data Exploration With Hadoop
Rapid Data Exploration With Hadoop
 
Prototyping Data Intensive Apps: TrendingTopics.org
Prototyping Data Intensive Apps: TrendingTopics.orgPrototyping Data Intensive Apps: TrendingTopics.org
Prototyping Data Intensive Apps: TrendingTopics.org
 
Elasticwulf Pycon Talk
Elasticwulf Pycon TalkElasticwulf Pycon Talk
Elasticwulf Pycon Talk
 

Developing Data Products

  • 1. Developing Data Products Uber Tech Talk Pete Skomoroch @peteskomoroch December 5 2012 ©2012 LinkedIn Corporation. All Rights Reserved.
  • 2. Examples, Techniques, & Lessons Learned Developing Data Products
  • 3. Our Mission Connect the world’s professionals to make them more productive and successful. Our Vision Create economic opportunity for every professional in the world. Members First!
  • 4. LinkedIn is the leading professional network site 187M+ 1 LinkedIn Members 2 640M+ Worldwide Professionals 2 3,300M+ Worldwide Workforce ©2012 LinkedIn Corporation. All Rights Reserved. 4
  • 5. LinkedIn profiles represent our professional identity 1 2 187M Members 187M Member Profiles ©2012 LinkedIn Corporation. All Rights Reserved. 5
  • 6. We have a lot of data. ©2012 LinkedIn Corporation. All Rights Reserved.
  • 7. We have a lot of data. And (like everyone else), we store it in Hadoop. ©2012 LinkedIn Corporation. All Rights Reserved.
  • 8. We have a lot of data. And (like everyone else), we store it in Hadoop. And people build awesome things with that data. ©2012 LinkedIn Corporation. All Rights Reserved.
  • 9. What do we mean by data products?
  • 10. Building products from data at LinkedIn A few examples:  People You May Know  Skills and Endorsements  Year in Review  Network Updates Digest  InMaps  Who’s viewed my profile  Collaborative Filtering  Groups You May Like  and more… ©2012 LinkedIn Corporation. All Rights Reserved.
  • 11. Collaborative Filtering: LinkedIn Skill Pages ©2012 LinkedIn Corporation. All Rights Reserved.
  • 12. Classification: giving structure to unstructured data Extract ©2012 LinkedIn Corporation. All Rights Reserved.
  • 13. Clustering & Disambiguation ©2012 LinkedIn Corporation. All Rights Reserved.
  • 14. De-duplication and Normalization ©2012 LinkedIn Corporation. All Rights Reserved.
  • 15. Network Algorithms: Relevance & Ranking ©2012 LinkedIn Corporation. All Rights Reserved. 15
  • 16. Prediction: Personalized Skill Recommendations ©2012 LinkedIn Corporation. All Rights Reserved.
  • 17.
  • 18.
  • 19. Skill Endorsements ©2012 LinkedIn Corporation. All Rights Reserved.
  • 20. Social Proof and the Skill Endorsement Graph ©2012 LinkedIn Corporation. All Rights Reserved. 20
  • 21. The Economic Graph: Skills, Jobs, People, Locations… Location ©2012 LinkedIn Corporation. All Rights Reserved. 21
  • 22. Lessons learned developing data products
  • 23. Collect the right data at the right time
  • 24. Large amounts of data can reveal new patterns Probability of Job Title Months since graduation ©2012 LinkedIn Corporation. All Rights Reserved. 24
  • 25. Be wary of “black-box” approaches ©2012 LinkedIn Corporation. All Rights Reserved. 25
  • 26. Look at your data ©2012 LinkedIn Corporation. All Rights Reserved. 26
  • 27. Aggregate statistics can be misleading 12 10 8 6 4 2 0 1 2 3 4 5 6 7 8 9 10 ©2012 LinkedIn Corporation. All Rights Reserved. 27
  • 28. Build a viewer app, “micro-listen” ©2012 LinkedIn Corporation. All Rights Reserved. 28
  • 29. Algorithmic intuition: include data geeks in design ©2012 LinkedIn Corporation. All Rights Reserved. 29
  • 30. OODA: Think like a jet fighter ©2012 LinkedIn Corporation. All Rights Reserved. 30
  • 31. OODA: Observe, Orient, Decide, Act ©2012 LinkedIn Corporation. All Rights Reserved. 31
  • 32. OODA: The speed you can move determines victory ©2012 LinkedIn Corporation. All Rights Reserved. 32
  • 33. Red teaming: what can go wrong likely will ©2012 LinkedIn Corporation. All Rights Reserved. 33
  • 34. Error data is super valuable, analyze it and adapt ©2012 LinkedIn Corporation. All Rights Reserved. 34
  • 35. Conclusion: tips for developing data products  Collect the right data at the right time  Large amounts of data can reveal new patterns  Be wary of “black box” approaches  Look at your raw data  Aggregate statistics can be misleading  Build and use viewer apps  Include data geeks in design process  OODA: Think like a jet fighter  Red-teaming: anticipate edge cases  Find opportunity in your error data ©2012 LinkedIn Corporation. All Rights Reserved.
  • 36. Questions? More info: data.linkedin.com @peteskomoroch ©2012 LinkedIn Corporation. All Rights Reserved. 36

Notas do Editor

  1. Mission: For us, fundamentally changing the way the world works begins with our mission statement: To connect the world’s professionals to make them more productive and successful. This means not only helping people to find their dream jobs, but also enabling them to be great at the jobs they’re already in. Vision: But, we’re just getting started. By our measure,there are more than 640 million professionals in the world. And roughly 3.3 billion people in the global workforce. Ultimately, our vision is to create economic opportunity for every professional, which we believe is an especially crucial objective in light of current macroeconomic trends.Our most important core value is that members come first.