SlideShare uma empresa Scribd logo
1 de 36
Baixar para ler offline
Smart Canvas®
Hybrid Recommender System
Architecture and Evaluation
Gilmar Souza
Gabriel Moreira
We are global with people in Brazil, USA,
Europe, Australia, Japan and China
Our excellence has been recognized by the
market with awards since our foundation in 1995
We are truly multi-cultural, an army of over
2000 talented employees and their great stories
Strong presence in strategic regions
USA
• Atlanta
• Philadelphia
• Houston
• San Francisco
• Somerset NJ
• New York
Brazil
• Campinas
(headquarters)
• Belo Horizonte
• Rio de Janeiro
• São Paulo
UK
• London
China
• Ningbo
• Shanghai
Japan
• Tokyo
3
Australia
• Sydney
Smart Canvas  @ Large Scale Recommender Systems Workshop 2015
Smart Canvas is a platform that delivers
web and mobile user experiences using
machine learning algorithms.
Smart CanvasReordered Stacks
Desktop mobile sites
and mobile apps
Card deck
(Chronological
content inventory)
Big Picture: 1 aggregate -> 2. cardify -> 3. curate
Your environment
(internal and external)
Public Data
RSS
Social
Networks
All of your portals,
apps and people
3rd party systems
through custom
connections
1 3
2
metrics and insights
Algorithms
Context
● Cloud-based SaaS, hosted on Google
● Multi-tenant
● Used in different domains (websites, intranets,
collaboration portals)
● Content produced by the users or ingested from
external services.
● Hybrid recommender system
● Implicit and explicit feedback (touchpoints)
Some Context
Following concepts of material design, presents content as cards and allow like, dislike, pins and
share interactions. Curation merges search relevance with personalized recommendations.
Smart Canvas User Interface
Websites: Ancar Ivanhoe Shopping Malls
Digital strategy revamp
Collaboration Portal: Whirpool Winning Workplace Transformation
The place to learn about Google Apps
Collaboration Portal: SulAmérica Seguros: Innovation Portal
Capturing ideas and inspiration to overcome new market challenges
Collaboration Portal: Motorola Smart Analytics Portal
Reports repository reimagined
Designing for Choice:
The Making of the
Wood-Back Moto X
[VIDEO]New Year, New You
Publisher Name
30 december 2014
Get to Know Rhea
Jeong, a Motorola
Designer of Moto Hint
Search
Analytics
999 999 999
Inside Motorola Inside Motorola Inside Motorola Inside Motorola
999 999 999 999 999 999 999 999 999
...
...
Smart Analytics
Moto X Device Activations:
Latin America North
source.com
Customer Satisfaction
4Q2014
source.com
Model Sales per region
source.com
All
Number of support calls per
region
source.com
What's new Communities AnalyticsPeople ToolsEvents
Smart Analytics
Inside Motorola
999 999 999
Number of support calls
per region
source.com
● 23 Tenants (customer portals).
● 33K cards ingested.
● 8.5M users (anonymous and logged).
● 30M touchpoints (user events) captured.
● 14M recommendations provided.
● 300 ms average recommendation response time.
Last 12 months In Numbers
Architecture
Cards
Touchpoints
Users
Data Layer Batch Layer
Recommenders
Offline Components
Similarity
Algorithms
Serving Layer
Recommenders
Online Components
Hybrid
Composition
Recommendation Layers
Search
Engine
search by terms.
(context: url,
locale
device and
person id)
cards
satisfying the
search criteria
Dynamic
Recommender
Strategy
Recommenders
cards reordered by
relevance.
Recommenders are
created and configured
based the context pattern
(locale, device, time)
1
2
3
4
5
search results curated by
personalized recommenders
6
user clickstream (touchpoints) is
used to refine recommendations
Recommendation Process
Online Serving
Google App Engine (Java)
Batch Processing
Google Compute Engine
Google Cloud Storage
Dynamic
Recommender
Factory
Recommenders
HDFS
Memcache On-demand Jobs
Datastore
(NoSQL)
The recommender system is implemented on Google Cloud Platform and Hadoop Ecosystem
to ensure its scalability and performance at large.
Pig Mahout Python
Smart Canvas
Users
Technology Components
Algorithms
HYBRID RECOMMENDER STRATEGY IS AN AGGREGATION OF THESE ALGORITHMS
Personalized
USER-BASED CF - Find similar users and recommend cards those users liked
CONTENT-BASED FILTERING - Recommends cards with content similar to the user's preference
vector (weighted average of the most relevant words in the content of the cards user interacted with).
ITEM-ITEM FREQUENCY - Recommends X cards frequently read together with the last Y cards user
has read.
DISLIKE FILTER - Reduces relevance of cards user has disliked
Non-Personalized
POPULARITY - Recommends most popular cards, decreasing its relevance as them become older
RECENCY - Recommends most recent cards
FIXED PACK - Human curation, where cards are fixed on top by the admin users (marketers)
ESSENTIALS PACK - Recommends a defined set of cards for new users (< Z interactions)
Recommender Algorithms Overview
Evaluation Framework
● Normalized Discounted Cumulative Gain (nDCG)
Verify whether the cards are ranked accordingly to the relevance
for the users
● Top-N Accuracy
Verify whether the more relevant cards for the user are among
the Top-N
○ Precision - How many Top-N cards are relevant?
○ Recall - How many relevant cards are among the Top-N?
Offline Metrics
Cognition Trainer
Offline Evaluation workflow
Offline Evaluator
1 - Filters touchpoints from train and test sets
periods
2 - Run cognition job
3 - Gets recommendations for each user using a
SmartCanvas WebService
Users
Items
Interactions
Cognition
WebService
Pre-generated
recommendations
(Utility Matrix)
2
1
3
4
5
4 - Compare recommendations ranking with real
user interactions in test set, and calculate offline
metrics and statistical significance
5 - Stores results in BigQuery tables, which can be
accessed by a Tableau dashboard
Offline
Metrics
Database
1 - Users sessions are split among control group and variants of hybrid recommender settings
2 - User interactions and recommendation logs are recorded as touchpoints in BigQuery
3 - Online Evaluator Job calculates engagement, accuracy, and coverage metrics
4 - Results are stored in BigQuery and accessible from Tableau Dashboards
Online metrics extraction workflow
4
1
3
Online Evaluator Job
Traffic Split
(A/B Tests)
Touchpoints
bigquery
2
Online Metrics - Engagement
1. Average sessions by user
Considers only engagement of active users -
with at least one visit (session) in the period
2. Average PageViews by session
Objective: Improve User Engagement with best Recommendations
Aggregation: Daily and Weekly
3. Average session strength
where S are the sessions and w the interaction strength
4. Average session minimum navigation time
Minimum navigation is calculated taking elapsed seconds between
the first and last touchpoints in the session.
When there is only one touchpoint in the session, 60 seconds will be
considered as navigation time.
Online Metrics - Engagement
Online Metrics - Conversion
2. CTR for Personalized and Non-
Personalized recommendations
Objective: Increase Click-Through Rate (CTR) for cards
recommended to the user
1. CTR by Recommender
#interactions / #visualizations
where # visualizations assumes that all cards
were viewed till at least the last card position
(ranking order) the user has interacted in the
page,
or a minimum position of 10, considering the
user has made at least one scroll in the page on
a desktop browser.
Online Metrics - Conversion
Online Metrics - Conversion
Online Metrics - Conversion
Online Metrics - Coverage
3. General Users
Coverage
% of the users that receives at list
one personalized recommendation
during a given period
Objective: Increase the number of users and cards with
personalized recommendations
1. Users Coverage by Recommender
#unique_users_with_at_least_a_personalized_
recommendation / #unique_users_accesses
2. Cards Coverage by Recommender
#cards_with_at_least_a_personalized_recomme
ndation / #active_recommendable_cards
where #active_recommendable_cards assumes all cards that
were active (status=CREATED, approvalStatus=APPROVED) at a
given date
Online Metrics - Coverage
A/B Testing
● Recommendations embedded in the search results using
elasticsearch
● Real-time and Near real-time recommendations delivered via
message bus.
● New product focused on enterprise collaboration
Under Construction
● Cloud computing allow us to scale up and down easily
● A hybrid approach helped us to deal with a multi-domain
scenario and to balance algorithms pros and cons.
● Our evaluation framework (offline and online) allows us to assess
hypothesis, tune hyperparameters, and provides a deep
understanding of recommenders effectiveness.
Conclusions
Gilmar Souza
gilmarj@ciandt.com
Gabriel Moreira
gabrielpm@ciandt.com
Thank you!
www.smartcanvas.com

Mais conteúdo relacionado

Destaque

Algorithmic Music Recommendations at Spotify
Algorithmic Music Recommendations at SpotifyAlgorithmic Music Recommendations at Spotify
Algorithmic Music Recommendations at SpotifyChris Johnson
 
Recsys 2014 Tutorial - The Recommender Problem Revisited
Recsys 2014 Tutorial - The Recommender Problem RevisitedRecsys 2014 Tutorial - The Recommender Problem Revisited
Recsys 2014 Tutorial - The Recommender Problem RevisitedXavier Amatriain
 
Collaborative Filtering at Spotify
Collaborative Filtering at SpotifyCollaborative Filtering at Spotify
Collaborative Filtering at SpotifyErik Bernhardsson
 
Interactive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and SpotifyInteractive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and SpotifyChris Johnson
 
Scaling Deep Learning with MXNet
Scaling Deep Learning with MXNetScaling Deep Learning with MXNet
Scaling Deep Learning with MXNetAI Frontiers
 
Recommendation system
Recommendation system Recommendation system
Recommendation system Vikrant Arya
 

Destaque (6)

Algorithmic Music Recommendations at Spotify
Algorithmic Music Recommendations at SpotifyAlgorithmic Music Recommendations at Spotify
Algorithmic Music Recommendations at Spotify
 
Recsys 2014 Tutorial - The Recommender Problem Revisited
Recsys 2014 Tutorial - The Recommender Problem RevisitedRecsys 2014 Tutorial - The Recommender Problem Revisited
Recsys 2014 Tutorial - The Recommender Problem Revisited
 
Collaborative Filtering at Spotify
Collaborative Filtering at SpotifyCollaborative Filtering at Spotify
Collaborative Filtering at Spotify
 
Interactive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and SpotifyInteractive Recommender Systems with Netflix and Spotify
Interactive Recommender Systems with Netflix and Spotify
 
Scaling Deep Learning with MXNet
Scaling Deep Learning with MXNetScaling Deep Learning with MXNet
Scaling Deep Learning with MXNet
 
Recommendation system
Recommendation system Recommendation system
Recommendation system
 

Semelhante a Smart Canvas @ Large Scale Recommender Systems Workshop 2015

Epam BI - Near Realtime Marketing Support System
Epam BI - Near Realtime Marketing Support SystemEpam BI - Near Realtime Marketing Support System
Epam BI - Near Realtime Marketing Support SystemDmitry Tolpeko
 
Productionalize content recommendation engine
Productionalize content recommendation engine Productionalize content recommendation engine
Productionalize content recommendation engine Kim Ming Teh
 
Vijayasarathy_Ranganathan-Portfolio-Apr2022.pdf
Vijayasarathy_Ranganathan-Portfolio-Apr2022.pdfVijayasarathy_Ranganathan-Portfolio-Apr2022.pdf
Vijayasarathy_Ranganathan-Portfolio-Apr2022.pdfVijayasarathy Ranganathan
 
Matthias Bettag - Challenges for each the multi-channel, multi-device and mul...
Matthias Bettag - Challenges for each the multi-channel, multi-device and mul...Matthias Bettag - Challenges for each the multi-channel, multi-device and mul...
Matthias Bettag - Challenges for each the multi-channel, multi-device and mul...Marketing Festival
 
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer Community
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer CommunityCrowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer Community
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer CommunityCrowdsourcing Week
 
Connecting the "dots" around your Consumers
Connecting the "dots" around your ConsumersConnecting the "dots" around your Consumers
Connecting the "dots" around your ConsumersDigital Air Strike
 
Planning For The Personalization Journey: From Empathy to Engagement
 Planning For The Personalization Journey: From Empathy to Engagement Planning For The Personalization Journey: From Empathy to Engagement
Planning For The Personalization Journey: From Empathy to EngagementRachel Wandishin
 
Time-to-Event Models, presented by DataSong and Revolution Analytics
Time-to-Event Models, presented by DataSong and Revolution AnalyticsTime-to-Event Models, presented by DataSong and Revolution Analytics
Time-to-Event Models, presented by DataSong and Revolution AnalyticsRevolution Analytics
 
Boosting Personalization In SaaS Using Machine Learning
Boosting Personalization  In SaaS Using Machine LearningBoosting Personalization  In SaaS Using Machine Learning
Boosting Personalization In SaaS Using Machine LearningGlorium Tech
 
Customer Experience Management
Customer Experience ManagementCustomer Experience Management
Customer Experience ManagementTigerGraph
 
Tarun datascientist affle
Tarun datascientist affleTarun datascientist affle
Tarun datascientist affleTarun Aditya
 
Tools for customer engagement
Tools for customer engagement Tools for customer engagement
Tools for customer engagement Ankit Chaudhary
 
A Modern Digital Experience Platform
A Modern Digital Experience PlatformA Modern Digital Experience Platform
A Modern Digital Experience PlatformPhase2
 
User definition and user journey poc ux with marketing approach
User definition and user journey poc ux with marketing approachUser definition and user journey poc ux with marketing approach
User definition and user journey poc ux with marketing approachSushmita Dutt
 
Boosting Personalization In SaaS Using Machine Learning.pdf
Boosting Personalization  In SaaS Using Machine Learning.pdfBoosting Personalization  In SaaS Using Machine Learning.pdf
Boosting Personalization In SaaS Using Machine Learning.pdfReza Rahimi
 

Semelhante a Smart Canvas @ Large Scale Recommender Systems Workshop 2015 (20)

Epam BI - Near Realtime Marketing Support System
Epam BI - Near Realtime Marketing Support SystemEpam BI - Near Realtime Marketing Support System
Epam BI - Near Realtime Marketing Support System
 
Productionalize content recommendation engine
Productionalize content recommendation engine Productionalize content recommendation engine
Productionalize content recommendation engine
 
Analytics in Online Retail
Analytics in Online RetailAnalytics in Online Retail
Analytics in Online Retail
 
ROI on Personalized Experiences - Greg Kihlstrom, GK5A
ROI on Personalized Experiences - Greg Kihlstrom, GK5AROI on Personalized Experiences - Greg Kihlstrom, GK5A
ROI on Personalized Experiences - Greg Kihlstrom, GK5A
 
Vijayasarathy_Ranganathan-Portfolio-Apr2022.pdf
Vijayasarathy_Ranganathan-Portfolio-Apr2022.pdfVijayasarathy_Ranganathan-Portfolio-Apr2022.pdf
Vijayasarathy_Ranganathan-Portfolio-Apr2022.pdf
 
Cam specialist-unit-web-analytics-and-social-media
Cam specialist-unit-web-analytics-and-social-mediaCam specialist-unit-web-analytics-and-social-media
Cam specialist-unit-web-analytics-and-social-media
 
Matthias Bettag - Challenges for each the multi-channel, multi-device and mul...
Matthias Bettag - Challenges for each the multi-channel, multi-device and mul...Matthias Bettag - Challenges for each the multi-channel, multi-device and mul...
Matthias Bettag - Challenges for each the multi-channel, multi-device and mul...
 
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer Community
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer CommunityCrowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer Community
Crowdtesting 2.0 - From Anonymous Crowd to Target Audience & Customer Community
 
Connecting the "dots" around your Consumers
Connecting the "dots" around your ConsumersConnecting the "dots" around your Consumers
Connecting the "dots" around your Consumers
 
Gartner-Digital-Experience-Platforms-MQ-q118
Gartner-Digital-Experience-Platforms-MQ-q118Gartner-Digital-Experience-Platforms-MQ-q118
Gartner-Digital-Experience-Platforms-MQ-q118
 
Planning For The Personalization Journey: From Empathy to Engagement
 Planning For The Personalization Journey: From Empathy to Engagement Planning For The Personalization Journey: From Empathy to Engagement
Planning For The Personalization Journey: From Empathy to Engagement
 
Time-to-Event Models, presented by DataSong and Revolution Analytics
Time-to-Event Models, presented by DataSong and Revolution AnalyticsTime-to-Event Models, presented by DataSong and Revolution Analytics
Time-to-Event Models, presented by DataSong and Revolution Analytics
 
Boosting Personalization In SaaS Using Machine Learning
Boosting Personalization  In SaaS Using Machine LearningBoosting Personalization  In SaaS Using Machine Learning
Boosting Personalization In SaaS Using Machine Learning
 
Customer Experience Management
Customer Experience ManagementCustomer Experience Management
Customer Experience Management
 
Tarun datascientist affle
Tarun datascientist affleTarun datascientist affle
Tarun datascientist affle
 
Tools for customer engagement
Tools for customer engagement Tools for customer engagement
Tools for customer engagement
 
A Modern Digital Experience Platform
A Modern Digital Experience PlatformA Modern Digital Experience Platform
A Modern Digital Experience Platform
 
User definition and user journey poc ux with marketing approach
User definition and user journey poc ux with marketing approachUser definition and user journey poc ux with marketing approach
User definition and user journey poc ux with marketing approach
 
Boosting Personalization In SaaS Using Machine Learning.pdf
Boosting Personalization  In SaaS Using Machine Learning.pdfBoosting Personalization  In SaaS Using Machine Learning.pdf
Boosting Personalization In SaaS Using Machine Learning.pdf
 
Mar-Tech Oversight
Mar-Tech OversightMar-Tech Oversight
Mar-Tech Oversight
 

Último

The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceMartin Humpolec
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 

Último (20)

The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Things you didn't know you can use in your Salesforce
Things you didn't know you can use in your SalesforceThings you didn't know you can use in your Salesforce
Things you didn't know you can use in your Salesforce
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 

Smart Canvas @ Large Scale Recommender Systems Workshop 2015

  • 1. Smart Canvas® Hybrid Recommender System Architecture and Evaluation Gilmar Souza Gabriel Moreira
  • 2. We are global with people in Brazil, USA, Europe, Australia, Japan and China Our excellence has been recognized by the market with awards since our foundation in 1995 We are truly multi-cultural, an army of over 2000 talented employees and their great stories
  • 3. Strong presence in strategic regions USA • Atlanta • Philadelphia • Houston • San Francisco • Somerset NJ • New York Brazil • Campinas (headquarters) • Belo Horizonte • Rio de Janeiro • São Paulo UK • London China • Ningbo • Shanghai Japan • Tokyo 3 Australia • Sydney
  • 5. Smart Canvas is a platform that delivers web and mobile user experiences using machine learning algorithms.
  • 6. Smart CanvasReordered Stacks Desktop mobile sites and mobile apps Card deck (Chronological content inventory) Big Picture: 1 aggregate -> 2. cardify -> 3. curate Your environment (internal and external) Public Data RSS Social Networks All of your portals, apps and people 3rd party systems through custom connections 1 3 2 metrics and insights Algorithms
  • 8. ● Cloud-based SaaS, hosted on Google ● Multi-tenant ● Used in different domains (websites, intranets, collaboration portals) ● Content produced by the users or ingested from external services. ● Hybrid recommender system ● Implicit and explicit feedback (touchpoints) Some Context
  • 9. Following concepts of material design, presents content as cards and allow like, dislike, pins and share interactions. Curation merges search relevance with personalized recommendations. Smart Canvas User Interface
  • 10. Websites: Ancar Ivanhoe Shopping Malls Digital strategy revamp
  • 11. Collaboration Portal: Whirpool Winning Workplace Transformation The place to learn about Google Apps
  • 12. Collaboration Portal: SulAmérica Seguros: Innovation Portal Capturing ideas and inspiration to overcome new market challenges
  • 13. Collaboration Portal: Motorola Smart Analytics Portal Reports repository reimagined Designing for Choice: The Making of the Wood-Back Moto X [VIDEO]New Year, New You Publisher Name 30 december 2014 Get to Know Rhea Jeong, a Motorola Designer of Moto Hint Search Analytics 999 999 999 Inside Motorola Inside Motorola Inside Motorola Inside Motorola 999 999 999 999 999 999 999 999 999 ... ... Smart Analytics Moto X Device Activations: Latin America North source.com Customer Satisfaction 4Q2014 source.com Model Sales per region source.com All Number of support calls per region source.com What's new Communities AnalyticsPeople ToolsEvents Smart Analytics Inside Motorola 999 999 999 Number of support calls per region source.com
  • 14. ● 23 Tenants (customer portals). ● 33K cards ingested. ● 8.5M users (anonymous and logged). ● 30M touchpoints (user events) captured. ● 14M recommendations provided. ● 300 ms average recommendation response time. Last 12 months In Numbers
  • 16. Cards Touchpoints Users Data Layer Batch Layer Recommenders Offline Components Similarity Algorithms Serving Layer Recommenders Online Components Hybrid Composition Recommendation Layers
  • 17. Search Engine search by terms. (context: url, locale device and person id) cards satisfying the search criteria Dynamic Recommender Strategy Recommenders cards reordered by relevance. Recommenders are created and configured based the context pattern (locale, device, time) 1 2 3 4 5 search results curated by personalized recommenders 6 user clickstream (touchpoints) is used to refine recommendations Recommendation Process
  • 18. Online Serving Google App Engine (Java) Batch Processing Google Compute Engine Google Cloud Storage Dynamic Recommender Factory Recommenders HDFS Memcache On-demand Jobs Datastore (NoSQL) The recommender system is implemented on Google Cloud Platform and Hadoop Ecosystem to ensure its scalability and performance at large. Pig Mahout Python Smart Canvas Users Technology Components
  • 20. HYBRID RECOMMENDER STRATEGY IS AN AGGREGATION OF THESE ALGORITHMS Personalized USER-BASED CF - Find similar users and recommend cards those users liked CONTENT-BASED FILTERING - Recommends cards with content similar to the user's preference vector (weighted average of the most relevant words in the content of the cards user interacted with). ITEM-ITEM FREQUENCY - Recommends X cards frequently read together with the last Y cards user has read. DISLIKE FILTER - Reduces relevance of cards user has disliked Non-Personalized POPULARITY - Recommends most popular cards, decreasing its relevance as them become older RECENCY - Recommends most recent cards FIXED PACK - Human curation, where cards are fixed on top by the admin users (marketers) ESSENTIALS PACK - Recommends a defined set of cards for new users (< Z interactions) Recommender Algorithms Overview
  • 22. ● Normalized Discounted Cumulative Gain (nDCG) Verify whether the cards are ranked accordingly to the relevance for the users ● Top-N Accuracy Verify whether the more relevant cards for the user are among the Top-N ○ Precision - How many Top-N cards are relevant? ○ Recall - How many relevant cards are among the Top-N? Offline Metrics
  • 23. Cognition Trainer Offline Evaluation workflow Offline Evaluator 1 - Filters touchpoints from train and test sets periods 2 - Run cognition job 3 - Gets recommendations for each user using a SmartCanvas WebService Users Items Interactions Cognition WebService Pre-generated recommendations (Utility Matrix) 2 1 3 4 5 4 - Compare recommendations ranking with real user interactions in test set, and calculate offline metrics and statistical significance 5 - Stores results in BigQuery tables, which can be accessed by a Tableau dashboard Offline Metrics Database
  • 24. 1 - Users sessions are split among control group and variants of hybrid recommender settings 2 - User interactions and recommendation logs are recorded as touchpoints in BigQuery 3 - Online Evaluator Job calculates engagement, accuracy, and coverage metrics 4 - Results are stored in BigQuery and accessible from Tableau Dashboards Online metrics extraction workflow 4 1 3 Online Evaluator Job Traffic Split (A/B Tests) Touchpoints bigquery 2
  • 25. Online Metrics - Engagement 1. Average sessions by user Considers only engagement of active users - with at least one visit (session) in the period 2. Average PageViews by session Objective: Improve User Engagement with best Recommendations Aggregation: Daily and Weekly 3. Average session strength where S are the sessions and w the interaction strength 4. Average session minimum navigation time Minimum navigation is calculated taking elapsed seconds between the first and last touchpoints in the session. When there is only one touchpoint in the session, 60 seconds will be considered as navigation time.
  • 26. Online Metrics - Engagement
  • 27. Online Metrics - Conversion 2. CTR for Personalized and Non- Personalized recommendations Objective: Increase Click-Through Rate (CTR) for cards recommended to the user 1. CTR by Recommender #interactions / #visualizations where # visualizations assumes that all cards were viewed till at least the last card position (ranking order) the user has interacted in the page, or a minimum position of 10, considering the user has made at least one scroll in the page on a desktop browser.
  • 28. Online Metrics - Conversion
  • 29. Online Metrics - Conversion
  • 30. Online Metrics - Conversion
  • 31. Online Metrics - Coverage 3. General Users Coverage % of the users that receives at list one personalized recommendation during a given period Objective: Increase the number of users and cards with personalized recommendations 1. Users Coverage by Recommender #unique_users_with_at_least_a_personalized_ recommendation / #unique_users_accesses 2. Cards Coverage by Recommender #cards_with_at_least_a_personalized_recomme ndation / #active_recommendable_cards where #active_recommendable_cards assumes all cards that were active (status=CREATED, approvalStatus=APPROVED) at a given date
  • 32. Online Metrics - Coverage
  • 34. ● Recommendations embedded in the search results using elasticsearch ● Real-time and Near real-time recommendations delivered via message bus. ● New product focused on enterprise collaboration Under Construction
  • 35. ● Cloud computing allow us to scale up and down easily ● A hybrid approach helped us to deal with a multi-domain scenario and to balance algorithms pros and cons. ● Our evaluation framework (offline and online) allows us to assess hypothesis, tune hyperparameters, and provides a deep understanding of recommenders effectiveness. Conclusions