WWW 2018 tutorial on Understanding User Needs & Tasks Details: https://task-ir.github.io/Task-based-Search/

1. Inferring User Tasks and Needs
Rishabh Mehrotra1, Emine Yilmaz2, Ahmed Hassan Awadallah3
1Spotify, London
2University College London
3Microsoft Research

2. Phase I: Understanding User Tasks & Needs
1. What is a task & why are they important?
2. Characterizing Tasks across devices & interfaces: desktop search, digital assistants &
voice-only assistants
3. Understanding User Tasks in Web Search
a. Extracting Query Intents
b. Search Task Understanding
a. Task extraction
b. Subtask extraction
c. Hierarchies of tasks & subtasks
c. Evaluating task extraction algorithms
4. Recommendation Systems
a. User intents & goals
b. Defining user intents
c. Predicting user intents

3. Section 1: Introduction

4. Introduction
Search is Everywhere!
Understanding users’
needs is HARD!

5. Why “Tasks” are Important?
• Long sessions are common
– 40% of sessions contain multiple
queries
– 30% of sessions take 5+ minutes
– 7% of these take 30+ minutes
71%
12%
10%
7%
Percentage of Sesions of Different Time Lengths
< 5 mins 5 – 15 mins 15 – 30 mins > 30 mins
[Dumais, NSF Search Tasks Workshop 2013]

6. • Long sessions account for most of
the search time
– 50% of time spent in sessions of 30+
mins
4%
15%
29%
52%
Time in Sessions of Different Time Lengths
< 5 mins 5 – 15 mins 15 – 30 mins > 30 mins
Why “Tasks” are Important?
[Dumais, NSF Search Tasks Workshop 2013]

7. • Long sessions are more likely
to continue
– Probability of issuing another
query, given session of length
N-1
Why “Tasks” are Important?
[Bailey et al., NII Shannon 2012]

8. People use search to accomplish variety of task
[Bailey et al., NII Shannon 2012]

9. Task Minutes per
task (Avg.)
Queries per
task (Avg.)
Compare products/
services
25 7
Plan travel 12 5
Find real estate 15 5
Buy a product 9 3
Many tasks require significant effort to achieve
[Bailey et al., NII Shannon 2012]

10. What is a “Task”?
Online credit
check
Mortgage
in principle
Facebook House buying
guide
Quit smoking
benefits
Solicitors
near me
…
Houses
for sale
Loans for
house
…
10:00am 10:03am 10:07am 12:30pm
17:00pm 17:02pm 17:06pm 18:15pm
Session 1 Session 2
Session 3 Session 4
A task is an atomic information need resulting in one or more queries [Jones and Klinkner, CIKM 2008]

11. Why identify “Tasks”?

12. Outline of the Tutorial
• Section 1: Introduction
• Section 2: Characterizing Tasks
• Section 3: Tasks Extraction Algorithms
• Section 4: Task based Evaluation
• Section 5: Applications

13. Section 2: Characterizing Tasks
• Understanding Intents & Tasks
– Query intents in IR
– Search sessions
– Sessions à Tasks
• Characterizing Tasks across devices
– Desktop based search
• Taxonomy of browsing & querying behavior
– Assistants
• Digital Assistants
– Use cases
– User engagement
• Voice-only Assistants
– Use cases
– User engagement

14. Query Intents in IR
• Queries often
– carry some degree of ambiguity
– are underspecified, or
– have various aspects/interpretations
• Identifying & understanding the intents helps
distinguish ambiguous or multi-faceted queries.

15. • Query classification aims to classify queries into pre-
defined intents
Pre-defined intents include:
– Query groups: Navigational/Informational/Transactional
– Topical categorization
• Manual (ODP topics)
• Automatic (LDA topics)
• Query clustering for non pre-defined intents
Query Intents in IR

16. Pre-defined query groups:
– Navigational
– Informational
– Transactional
Query Intents in IR
[Broder, SIGIR Forum 2002; Jansen et al., IPM 2008]

17. – Query terms
• Navigational: Company/business/org/people
• Informational: ways to/how to
– Click entropy
• Navigational clicks: low entropy
– URL domains clicked
• Navigational: usually primary domain (facebook.com)
– Query length
• Informational: longer queries (>2 words)
– Post-click user behavior
• Informational: longer sessions, reformulations
Characteristics & features used for automatic
classification:

18. Topical Categorization
• Open Directory Project (ODP)

19. • No longer updated but a static copy publicly available at:
http://www.dmoz.org/
• Crowd-sourced effort
• 15 top level categories
• 1031533 categories overall
• 91810 editors
• 3882684 websites
• 90 languages
Open Directory Project (ODP) [Sherman, ERIC’ 00]

20. • Topical categorization of user queries:
– Make use of Open Directory Project (ODP) topic hierarchy
– Crawl data from ODP to obtain training data
– Train Logistic Regression to obtain topic classier
• ODP topics widely used:
– developing user models for personalization
– ranking features
Open Directory Project (ODP) [Sherman, ERIC’ 00]

21. Automatic Topic Identification
• Many different models proposed
• Most popular: Latent Dirichlet Allocation (LDA) topic models [Blei
et al, JMLR ’03]
• Unsupervised extraction of topics
• Wide array of tools available

22. Latent Dirichlet Allocation [Blei et al, JMLR’03]
• LDA is a generative probabilistic model of a corpus. The basic idea
is that the documents are represented as random mixtures over
latent topics, where a topic is characterized by a distribution over
words.

23. For each document:
(a) draw a topic distribution, θd ∼ Dirichlet(α),
(b) for each word in the document:
(i) Draw a specific topic zd,n ∼ multinomial(θd) (ii)
Draw a word wd,n ∼ multinomial(βzd,n)
Latent Dirichlet Allocation [Blei et al, JMLR’03]

24. Query Intents in IR
• Query classification aims to classify queries into pre-defined intents
Pre-defined intents include:
– Query groups: Navigational/Informational/Transactional
– Topical categorization
• Manual (ODP topics)
• Automatic (LDA topics)
• Query clustering techniques based on:
– k-nearest neighbors
– Sequence clustering
– Using session, ad-click and sponsored search data
– Random walks on click graph
– Reformulations & clicks

25. Random Walks on Click Graph [Craswell et al., SIGIR’07]
• Consider the click graph:
– Bipartite with different types of nodes (queries, documents)
– An edge connects a query and a document if a click for that
query-document pair is observed
– The edge may be weighted according to the total number of
clicks from all users

26. • Cjk : click counts associating node j to k
• Define transition probabilities Pt+1|t (kij) from j to k:
• s is the self-transition probability, which corresponds to
the user favoring the current query or document
Random Walks on Click Graph [Craswell et al., SIGIR’07]

27. Inferring Query Intent from Reformulations & Clicks
Given an input query q, combine click d reformulation information to
find likely user intents using three steps:
1. Expand
– identify a set of possibly related queries to q (Recall is key here)
– Use session co-occurrence
2. Filter
– reduces the query neighborhood to more closely related queries,
improving precision
– Use 2 step Random walk on the the bipartite query-document click graph
– All pairs of queries with a random walk similarity above a fixed threshold
are connected
1. Cluster
– use the random walk similarities to find intent clusters
[Radlinski et al., WWW’10]

28. Section 2: Characterizing Tasks
• Understanding Intents & Tasks
– Query intents in IR
– Search sessions
– Sessions à Tasks
• Characterizing Tasks across devices
– Desktop based search
• Taxonomy of browsing & querying behavior
– Intelligent Assistants
• Digital Assistants
– Use cases
– User engagement
• Voice-only Assistants
– Use cases
– User engagement

29. Search Sessions as Tasks
Online credit
check
Mortgage
in principle
Facebook House buying
guide
Quit smoking
benefits
Solicitors
near me
…
Houses
for sale
Loans for
house
…
10:00am 10:03am 10:07am 12:30pm
17:00pm 17:02pm 17:06pm 18:15pm
Session 1 Session 2
Session 3 Session 4
Session continuation detection [Jon08, Agi12]
Session based task extraction [Luc11, Hua13, Wan13, Awa14, Luc13]
A Session is a chronologically ordered sequence of user interactions with the
search engine resulting in one or more queries – aimed at solving a single
information need.

30. Session based IR
Lot of research into the problem of segmenting and organizing query
logs into semantically coherent structures
Sessions allow us to:
– Look beyond queries
– Preserve semantic associations between adjacent query trails
– Maintain context of user activity
• Sessions assumed to be corresponding to model atomic search
tasks

31. Session based IR
Session for understanding user needs:
– Query context understanding:
• Prior queries in the session help lessen query ambiguity
– Search result re-ranking
• Session contexts helps in re-ranking results, and showing better
related search suggestions
– Personalization models
• Session level personalization often outperforms long term user
models
– User satisfaction
• Understanding session level user needs help in developing
appropriate metrics for gauging user satisfaction

32. Section 2: Characterizing Tasks
• Understanding Intents & Tasks
– Query intents in IR
– Search sessions
– Sessions à Tasks
• Characterizing Tasks across devices
– Desktop based search
• Taxonomy of browsing & querying behavior
– Assistants
• Digital Assistants
– Use cases
– User engagement
• Voice-only Assistants
– Use cases
– User engagement

33. Sessions vs. Tasks
0"
10"
20"
30"
40"
50"
60"
No#of#Users#(In#%)#
Avg#Number#of#Tasks#Performed#in#a#Session#
1" 2" 3" 4" 5" 6" 7" 8" 9" 10" 10+"
• Sessions don’t represent atomic information needs
• Users often multi-task in a session
tϕ = 30 minutes
An impression
finding flight ticketsbooking hotels

34. From Sessions to Tasks
• Sessions should NOT be the focal units in IR
• More than 70% of sessions are multi-task
– Users differ based on their multi-tasking habits
– Some topics prone to multi-tasking than others
• 75% of search tasks tend to span across multiple sessions [Jones
and Klinkner, CIKM '08]

35. What is a Task?
• A search task is an atomic information need resulting in one or
more queries [Jones and Klinkner, CIKM '08]
• Complex search task: A set of related information needs,
resulting in one or more (possibly complex) tasks.
Credit
check
House buying
guide …
Houses
for sale
Loans for
house
17:00pm 17:02pm 17:06pm 18:15pm
Session 1 Session 2
Improve
credit
score
18:25pm

36. Rishabh takes over

37. Section 2: Characterizing Tasks
• Understanding Intents & Tasks
– Query intents in IR
– Search sessions
– Sessions à Tasks
• Characterizing Tasks across devices
– Desktop based search
• Taxonomy of browsing & querying behavior
– Intelligent Assistants
• Digital Assistants
– Use cases
– User engagement
• Voice-only Assistants
– Use cases
– User engagement

38. Taxonomy of Tasks
[Broder, SIGIR Forum 2002; Russell et al., HICSS 2009; Bailey et al., NII Shannon 2012]

39. Typical Web Tasks by Session
• Definition: All web activities including browsing behavior and search behavior
[Bailey et al., NII Shannon 2012]

40. Top Search Tasks by Session
[Bailey et al., NII Shannon 2012]

41. Complex Tasks and Sub-tasks
[Bailey et al., NII Shannon 2012]
Comparison Task broken down into
the following sub-tasks:
• Explore dimensions for comparison (size,
color, capacity, megapixel)
• Compile and refine a list of choices
(comparable models)
• Find details about a choice
• Read reviews about a choice
• Read side by side comparisons
• Act on a comparison decision

42. Section 2: Characterizing Tasks
• Understanding Intents & Tasks
– Query intents in IR
– Search sessions
– Sessions à Tasks
• Characterizing Tasks across devices
– Desktop based search
• Taxonomy of browsing & querying behavior
– Intelligent Assistants
• Digital Assistants
– Use cases
– User engagement
• Voice-only Assistants
– Use cases
– User engagement

43. Digital Assistants: Use Cases
• Different from traditional web search
– Verbal & textual interactions
– Plethora of tasks (files, reminders, surf the web, etc)
• Motivates the need to develop better understanding of user
interactions
– Better user interest models
– Help in task support & completion
– Develop metrics & gauge satisfaction
– Envision future use case scenarios

44. Investigating use cases of a Digital Assistant

45. Session Characteristics
• Analysis by session: Over 80% sessions have 1-5 queries
– Similar to traditional web search
• Analysis by traffic: longer sessions bring in over 50% of traffic
– Traditional IR: longer sessions ~ exploring or struggling [Odijk et al. CIKM’15]
– Open Question: Is that the case here too?
0
0.1
0.2
0.3
0.4
0.5
0.6
1 "2-5" "6-10" "11-20" 21-50 51+
% Sessions
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1 "2-5" "6-10" "11-20" 21-50 51+
% IMPRESSIONS

46. Input Modality
• Desktop search is primarily text based
• Interaction with intelligent assistants is mostly speech based
• Longer sessions are mostly speech driven
• Speech queries are much longer
1.8
1.85
1.9
1.95
2
2.05
2.1
Speech Text
Avg No of Words
9.6
9.8
10
10.2
10.4
10.6
10.8
11
11.2
11.4
11.6
11.8
Speech Text
Avg No of Characters

47. Common Use-Case Scenarios
• General search
• Commands
– Alarms, camera, system
settings
• Answers
– Find instant answers
• MyStuff
– Search local files &
folders
0 0.1 0.2 0.3 0.4 0.5
MyStuff
Answers
Commands
GeneralSearch
Percentage Interactions
Cortana Desktop Use-cases
[Mehrotra et al., CAIR 2017]

48. Common Use-Case Scenarios
Spread across session length
• MyStuff
– Over 70% single query sessions
originate from MyStuff
– Proportion on MyStuff steadily
decreases as the session length
increases
• Instant Answers
– Increasing proportion of
sessions with session length
• Steady proportions of General
Search & Commands
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Use-case Mix-Up in Sessions
MyStuff Answers
CommandControl GeneralSearch
[Mehrotra et al., CAIR 2017]

49. Beyond Traditional IR
Commands Answers
0 0.1 0.2 0.3 0.4 0.5
Open App
Weather
How To
BilingualDict
Math
News Answers
Time zone
Entity Lookup
Percentage Interactions
Beyond Chitchat & General Search
• Top Commands: reminders, alarms, music
• Users seek direct answers: weather, HowTos, Math etc
• Envision appropriate changes as system evolves
[Mehrotra et al., CAIR 2017]

50. User Interaction Signals
Traditional
CortanaDesktop
Has Scroll
Traditional
CortanaDesktop
Scroll Distance
Traditional
CortanaDesktop
Pointer Distance
Traditional
CortanaDesktop
Hover Count
Click Count
SCC
Click based differences
CortanaDesktop Traditional
Traditional
CortanaDesktop
Query DT

51. User Interaction with Proactive Cards
• Similar trends to search queries:
– More clicks at the top cards
– Cards at lower ranks get relatively less
engagement
[Shokouhi and Guo, SIGIR 2015]

52. User Interaction with Proactive Cards
• Temporal trends
– Sports: weekend / Traffic: weekdays
[Shokouhi and Guo, SIGIR 2015]

53. Desktop vs Mobile vs Proactive
[Shokouhi and Guo, SIGIR 2015]

54. Section 2: Characterizing Tasks
• Understanding Intents & Tasks
– Query intents in IR
– Search sessions
– Sessions à Tasks
• Characterizing Tasks
– Desktop based search
• Taxonomy of browsing & querying behavior
– Digital Assistants
• Use cases
• User engagement
– Voice-only Assistants
• Use cases
• User engagement

55. Voice-only Assistants
• No textual input
• More conversational in nature

56. Tasks people do on Voice-only Assistants
Source: comScore

57. Where do people use the Smart Speakers?
*Other devices may not be as easily accessible with occupied hands.
Source: NPR, Edison Research

58. Summary: Characterizing Tasks
• Understanding User Intents & Tasks
– Different abstractions of understanding user needs: Queries à Sessions à Tasks
– Methods for intent identification
• Classification
• Clustering
– Search sessions and Tasks
• Multitasking sessions
• Tasks provide a better abstraction
• Characterizing Tasks
– Taxonomy of web search tasks
– Traditional desktop based search tasks are different than emerging tasks on smart
assistants
– User interactions differ across different devices
– More contextual information available for smart assistants

59. Outline of the Tutorial
• Section 1: Introduction
• Section 2: Characterizing Tasks
• Section 3: Tasks Extraction Algorithms
• Section 4: Task based Evaluation
• Section 5: Applications