A Beginners Guide to Building a RAG App Using Open Source Milvus
CrowdSearcher. Reactive and multiplatform Crowdsourcing. keynote speech at DBCrowd2013 workshop @ vldb2013
1. CROWDSEARCHER
Marco Brambilla, Stefano Ceri,
Andrea Mauri, Riccardo Volonterio
Politecnico di Milano
Dipartimento di Elettronica, Informazione e BioIngegneria
Crowdsearcher 1
2. Crowd-based Applications
• Emerging crowd-based applications:
• opinion mining
• localized information gathering
• marketing campaigns
• expert response gathering
• General structure:
• the requestor poses some questions
• a wide set of responders are in charge of providing answers
(typically unknown to the requestor)
• the system organizes a response collection campaign
• Include crowdsourcing and crowdsearching
Crowdsearcher 2
3. The “system” is a wide concept
• Crowd-based applications may use social networks and Q&A
websites in addition to crowdsourcing platforms
• Our approach: a coordination engine which keeps an overall
control on the application deployment and execution
Crowdsearcher 3
CrowdSearcher
APIAccess
4. CrowdSearcher
• Combines a conceptual framework, a specification
paradigm and a reactive execution control environment
• Supports designing, deploying, and monitoring
applications on top of crowd-based systems
• Design is top-down, platform-independent
• Deployment turns declarative specifications into platform-specific
implementations which include social networks and crowdsourcing
platforms
• Monitoring provides reactive control, which guarantees
applications’ adaptation and interoperability
• Developed in the context of Search Computing
(SeCo, ERC Advanced Grant, 2008-2013)
Crowdsearcher 4
5. An example of crowd-based application:
crowd-search
• People do not trust web search completely
• Want to get direct feedback from people
• Expect recommendations, insights, opinions, reassurance
Crowdsearcher 7
6. Crowd-searching after conventional
search
• From search results to friends and experts feedback
Social Platform
initial query
Human
Search
System
Search
System
Social PlatformSocial Platform
Crowdsearcher 8
14. Deployment: search on the social network
• Multi-platform deployment
Crowdsearcher 16
15. Deployment: search on the social network
• Multi-platform deployment
Crowdsearcher 17
16. Deployment: search on the social network
• Multi-platform deployment
Crowdsearcher 18
17. From social workers to communities
• Issues and problems
• Motivation of the responders
• Intensity of social activity of the asker
• Topic appropriateness
• Timing of the post (hour of the day, day of the
week)
• Context and language barrier
Crowdsearcher 19
19. • A simple task design and deployment process, based on specific data
structures
• created using model-driven transformations
• driven by the task specification
The Design Process
Task
Specification
Task Planning
Control
Specification
Crowdsearcher 21
• Task Specification: task operations, objects, and performers
• Task Planning: work distribution
• Control Specification: task control policies
20. Task Specification
• Which are the input objects of the crowd interaction?
• Do they have a schema (record of named and typed fields)?
• Which operations should the crowd perform?
• Like, label, comment, add new instances, verify/modify data, order, etc.
• Who are the performers of the task? How should they be
selected? And invited?
• e.g. push vs pull model
• Which quality criteria should be used for deciding the task
outcome?
• e.g., majority weighting, with/without spam detection
• Which platforms should be used? Which execution
interface should be used?
Crowdsearcher 22
21. Operations
• In a Task, performers are required to execute logical operations on input objects
• e.g. Locate the faces of the people appearing in the following 5 images
• CrowdSearcher offers pre-defined operation types:
• Like: Ask a performer to express a preference (true/false)
• e.g. Do you like this picture?
• Comment: Ask a performer to write a description / summary / evaluation
• e.g. Can you summarize the following text using your own words?
• Tag: Ask a performer to annotate an object with a set of tags
• e.g. How would you label the following image?
• Classify: Ask a performer to classify an object within a closed-set of alternatives
• e.g. Would you classify this tweet as pro-right, pro-left, or neutral?
• Add: Ask a performer to add a new object conforming to the specified schema
• e.g. Can you list the name and address of good restaurants nearby Politecnico di Milano?
• Modify: Ask a performer to verify/modify the content of one or more input object
• e.g. Is this wine from Cinque Terre? If not, where does it come from?
• Order: Ask a performer to order the input objects
• e.g. Order the following books according to your taste
Crowdsearcher 23
22. Task planning
Typical problems:
• Task structuring: the task is too complex or too critical to
be executed as a single operation.
• Task splitting: the input data collection is too large to be
presented to a user.
• Task routing: a query can be distributed according to the
values of some attribute of the collection.
Crowdsearcher 24
23. Micro Tasks
• The actual unit of interaction with a performer.
• Mapping of objects to Micro Tasks:
• How many objects in each MicroTask?
• Which objects should appear in each MicroTask?
• How often an object should appear in MicroTasks?
• Which objects cannot appear together?
• Should objects be presented always in some order?
Crowdsearcher 25
24. Assignment Strategy
• Given a set of MicroTasks, which performers are
assigned to them?
• Pull vs Push:
• Pull: The performer choses
• Push: The performer is chosen
• Online vs offline
• Micro Tasks dynamically assigned to performers
• First come / First served
• Based on performer’s performance
• MicroTasks statically assigned to performers
• Based on performers’ priority
• Based on matching
Crowdsearcher 26
25. Invitation Strategy
• The process of inviting performers to perform Micro Tasks
• Can use very different mechanisms
• Essential in order to generate the appropriate performer reaction / reward.
• Examples:
• Send an email to a mailing list
• Publish a HIT on Mechanical Turk
• Create a new challenge in your game
• Publish a post/tweet on your social network profile
• Publish a post/tweet on your friends' profile
Crowdsearcher 27
26. Steps in Crodw-based Application Design
1) Task Design
2) Object and Performer Design
3) Micro Task Design
32. Application instatiation (for Italian Politics)
• Given the picture and name of a politician, specify his/her political
affiliation
• No time limit
• Performers are encouraged to look up online
• 2 set of rules
• Majority Evaluation
• Spammer Detection
Crowdsearcher 34
34. Crowd Control is tough…
• There are several aspects that makes crowd
engineering complicated
• Task design, planning, assignment
• Workers discovery, assessment, engagement
Crowdsearcher 36
35. Crowd Control is tough…
• There are several aspects that makes crowd
engineering complicated
• Task design, planning, assignment
• Workers discovery, assessment, engagement
• Controlling crowdsourcing tasks is a
fundamental issue
• Cost
• Time
• Quality
• Need for higher level abstrasction and tools
Crowdsearcher 37
36. Reactive Crowdsourcing
• A conceptual framework for controlling the execution of
crowd-based computations. Based on:
• Control Marts
• Active Rules
• Classical forms of controls:
• Majority control (to close object computations)
• Quality control (to check that quality constraints are met)
• Spam detection (to detect / eliminate some performers)
• Multi-platform adaptation (to change the deployment platform)
• Social adaptation (to change the community of performers)
Crowdsearcher 38
37. Why Active Rules?
• Ease of Use: control is easily expressible
• Simple formalism, simple computation
• Power: arbitrarily complex controls is supported
• Extensibility mechanisms
• Automation: active rules can be system-generated
• Well-defined semantics
• Flexibility: localized impact of changes on the rules set
• Control isolation
• Known formal properties descending from known theory
• Termination, confluence
Crowdsearcher 39
38. Control Mart
• Data structure for controlling application execution, inspired by data
marts (for data warehousing); content is automatically built from task
specification & planning
• Central entity: MicroTask Object Execution
• Dimensions: Task / Operations, Performer, Object
Crowdsearcher 40
Task Specification Task Planning Control Specification
40. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
Crowdsearcher 42
41. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
• Events: data updates / timer
• ROW-level granularity
• OLD before state of a row
• NEW after state of a row
Crowdsearcher 43
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
42. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
• Events: data updates / timer
• ROW-level granularity
• OLD before state of a row
• NEW after state of a row
• Condition: a predicate that must be satisfied (e.g. conditions on
control mart attributes)
Crowdsearcher 44
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
43. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
• Events: data updates / timer
• ROW-level granularity
• OLD before state of a row
• NEW after state of a row
• Condition: a predicate that must be satisfied (e.g. conditions on
control mart attributes)
• Actions: updates on data structures (e.g. change attribute value,
create new instances), special functions (e.g. replan)
Crowdsearcher 45
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
a: SET ObjectControl[oID == NEW.oID].#Eval+= 1
44. e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
a: SET ObjectControl[oID == NEW.oID].#Eval+= 1
Crowdsearcher 46
Rule Example 1
45. e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
a: SET ObjectControl[oID == NEW.oID].#Eval+= 1
Crowdsearcher 47
Rule Example 1
46. e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
a: SET ObjectControl[oID == NEW.oID].#Eval+= 1
Crowdsearcher 48
Rule Example 1
47. Crowdsearcher 49
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example 2
48. Crowdsearcher 50
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example 2
49. Crowdsearcher 51
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example 2
50. Crowdsearcher 52
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example 2
52. Rule Programming Best Practice
Crowdsearcher 54
• We define three classes of rules
• Control rules: modifying the control tables;
53. Rule Programming Best Practice
Crowdsearcher 55
• We define three classes of rules
• Control rules: modifying the control tables;
• Result rules: modifying the dimension tables (object, performer, task);
54. Rule Programming Best Practice
Crowdsearcher 56
• Top-to-bottom, left-to-right, evaluation
• Guaranteed termination
• We define three classes of rules
• Control rules: modifying the control tables;
• Result rules: modifying the dimension tables (object, performer, task);
55. Rule Programming Best Practice
• We define three classes of rules
• Control rules: modifying the control tables;
• Result rules: modifying the dimension tables (object, performer, task);
• Execution rules: modifying the execution table, either directly or through re-planning
Crowdsearcher 57
• Termination must be proven (rule precedence graph has cycles)
57. Crowdsearcher Experiment 1
• Goal: Test engagement on social networks
• Some 150 users
• Two classes of experiments:
• Random questions on fixed topics: interests (e.g. restaurants in the
vicinity of Politecnico), to famous 2011 songs, or to top-quality EU
soccer teams
• Questions manually submitted by the users
• Different invitation strategies:
• Random invitation
• Explicit selection of responders by the asker
• Outcome
• 175 like and insert queries
• 1536 invitations to friends
• 230 answers
• 95 questions (~55%) got at least one answer
Crowdsearcher 59
63. Crowdsearcher Experiment 2
• GOAL: demonstrate the flexibility and expressive power
of reactive crowdsourcing
• 3 experiments, focused on Italian politicians
• Parties: Human Computation affiliation classification
• Law: Game With a Purpose guess the convicted politician
• Order: Pure Game hot or not
• 1 week (November 2012)
• 284 distinct performers
• Recruited through public mailing lists and social networks
announcements
• 3500 Micro Tasks
Crowdsearcher 66
64. Politician Affiliation
• Given the picture and name of a politician, specify his/her political
affiliation
• No time limit
• Performers are encouraged to look up online
• 2 set of rules
• Majority Evaluation
• Spammer Detection
Crowdsearcher 67
65. Results – Majority Evaluation_1/3
Crowdsearcher 68
30 object; object redundancy = 9;
Final object classification as simple majority after 7 evaluations
66. Results - Majority Evaluation_2/3
Crowdsearcher 69
Final object classification as total majority after 3 evaluations
Otherwise, re-plan of 4 additional evaluations. Then simple majority at 7
67. Results - Majority Evaluation_3/3
Crowdsearcher 70
Final object classification as total majority after 3 evaluations
Otherwise, simple majority at 5 or at 7 (with replan)
68. Results – Spammer Detection_1/2
Crowdsearcher 71
New rule for spammer detection without ground truth
Performer correctness on final majority. Spammer if > 50% wrong classifications
69. Results – Spammer Detection_1/2
Crowdsearcher 72
New rule for spammer detection without ground truth
Performer correctness on current majority. Spammer if > 50% wrong classifications
71. Problem
• Ranking the members of a social group according
to the level of knowledge that they have about a
given topic
• Application: crowd selection (for Crowd Searching
or Sourcing)
• Available data
• User profile
• behavioral trace that users leave behind them through
their social activities
Crowdsearcher 74
72. Considered Features
• User Profiles
• Plus Linked Web Pages
• Social Relationships
• Facebook Friendship
• Twitter mutual following relationship
• LinkedIn Connections
• Resource Containers
• Groups, Facebook Pages
• Linked Pages
• Users who are followed by a given user are resource containers
• Resources
• Material published in resource containers
Crowdsearcher 75
78. Resource Processing
• Extraction from Social
Network APIs
• Extraction of Text from linked
Web Pages
• Alchemy Text Extraction APIs
• Language Identification
• Text Processing
• Sanitization, tokenization,
stopword, lemmatization
• Entity Extraction and
Disambiguation
• TagMe
Crowdsearcher 81
79. Dataset
• 7 kinds of expertises
• Computer Engineering, Location, Movies & TV, Music, Science,
Sport, Technology & Videogames
• 40 volunteer users (on Facebook & Twitter & LinkedIN)
• 330.000 resources (70% with URL to external resources)
• Groundtruth created trough self-assessment
• For expertise need, vote on 7 Likert Scale
• EXPERTS expertise above average
Crowdsearcher 84
80. Metrics
• We obtain lists of candidate experts and assess them
against the ground truth, using:
• For precision:
• Mean Average Precision (MAP)
• 11-Point Interpolated Average Precision (11-P)
• For ranking:
• Mean Reciprocal Rank (MRR) – for the first value
• Normalized Discounted Cumulative Gain (DCG) – for more values, can
be set @N for the first N values
Crowdsearcher 86
82. Friendship Relationship not useful
• Inspecting friend’s resources does not improve metrics!
Crowdsearcher 88
83. Social Network Analysis
• a
Comparison of the results obtained with All the social networks, or separately by
FaceBook, TWitter, and LinkedIn.
Crowdsearcher 89
84. Main Results
• Profiles are less effective than level-1 resources
• Resources produced by others help in describing each individual’s
expertise
• Twitter is the most effective social network for expertise
matching – sometimes it outperforms the other social
networks
• Twitter most effective in Computer Engineering, Science, Technology &
Games, Sport
• Facebook effective in Locations, Sport, Movies & TV, Music
• Linked-in never very helpful in locating expertise
Crowdsearcher 90
86. Summary
• Results
• An integrated framework for crowdsourcing task design and control
• Well-structured control rules with guarantees of termination
• Support for cross-platform crowd interoperability
• A working prototype crowdsearcher.search-computing.org
• Forthcoming
• Publication of Web Interface + API
• Support of declarative options for automatic rule generation
• Integration with more social networks and human computation
platforms
• Providing vertical solutions for specific markets
• More applications and experiments (e.g. in Expo 2015)
Crowdsearcher 96