Discovering Real-World Usage for a Multimodal Math Search Interface
•Transferir como PPTX, PDF•
1 gostou•717 visualizações
Thesis Defense Presentation by Keita (Del Valle) Wangari for Masters of Science in Human-Computer Interaction at Rochester Institute of Technology.
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Discovering Real-World Usage for a Multimodal Math Search Interface
Semelhante a Discovering Real-World Usage for a Multimodal Math Search Interface (20)
Último
Último (20)
Discovering Real-World Usage for a Multimodal Math Search Interface
- 1. Discovering Real World Usage for a Multimodal Math Search Interface Thesis Defense | Keita Del Valle Wangari Rochester Institute of Technology December 16, 2013
- 2. THE PROBLEM The Intention Gap.
- 6. How do you “Google it”?
- 7. Query options • n choose k (verbal expression) • binomial coefficient (related term/concept) • {{n}choose{k}} (LaTeX) • … (template editor)
- 8. Target Audience Non-experts in the math domain: • less likely to know math expression names • less likely to know a math encoding language • less likely to be familiar with math template editors or the sites that use them
- 9. Intention Gap Intention Gap (Zha et al, 2010)
- 10. Intention Gap
- 11. To use math expressions in search, current search engines require knowing expression names or using a structure editor or encoding language (e.g., LaTeX) to enter expressions. For people who are not math experts, this can lead to an “intention gap” between the math query they wish to express, and what the interface will allow. Problem Statement
- 12. THE STUDY GOALS
- 13. min (Sasarak et al, 2012)
- 14. Demo 1: drawn input
- 15. Demo 2: typed input
- 16. Demo 3: search
- 17. Study goals 1. To observe whether min changes user search behavior 2. Discover real-world scenarios for math search interfaces
- 18. What do we know about... • …the visual aspects of math? • …inputting math on a computer? • …math search?
- 19. PREVIOUS WORK Perceiving, inputting, & searching for math.
- 20. Perceiving math • Appearance affects reasoning – cognitive illusion (Landy & Goldstone, 2007) • Grounded in visual structure (Landy & Goldstone, 2007) • Designed to encourage us to think visually (Anthony, Yang & Koedinger, 2005) • Diagrammatic – derive meaning from layout (Landy & Goldstone, 2007)
- 21. Inputting math • Typed input not optimal (Anthony, Yang & Koedinger, 2005) • Handwriting most natural and satisfactory (Anthony, Yang & Koedinger, 2005) • Equation editors tedious (Smithies, Novins & Arvo, 2001)
- 22. Math search interfaces • Text keywords • Expressions coded in LaTeX, TeX, MathML • Expressions built using a template-based editor
- 23. LaTeX input in Wolfram Alpha www.wolframalpha.com
- 24. Editor input in MathFind (Munavalli & Miner, 2006)
- 25. Zhao et al study • written math expression not useful as a search term • doubt value of query-by- expression capability • prefer inputting LaTeX • text most viable form of searching • specialized input modalities unwieldy (Zhao, Kan, & Theng, 2008)
- 26. min 1. To observe whether min changes user search behavior 2. Discover real-world scenarios for math search interfaces
- 27. THE STUDY Design & execution.
- 28. Design considerations • Observational – min is in prototype phase • Peer-assist style – reduce math anxiety • Math professor input – ensure tasks are level-appropriate • Pilot – test and refine the protocol
- 29. Participants • 16 participants • 18 or older • first- or second-year college math course @ RIT • Beginner or Intermediate level in math knowledge • Comfortable or Very Comfortable using the internet • Recruited via email
- 30. The test session • In the Usability Lab, Golisano Hall @ RIT • 1 hour duration • 1 moderator & 1 observer • Recorded • $20 compensation
- 31. Artifacts • Screener Survey (online) • Orientation Script (printout) • Consent Form (printout) • Background Survey (online) • Task Sheets (printouts) • Pre-demo Questionnaire (printout) • Post-Study Questionnaire (online) • Post-Study Interview Sheet (printout) • Project Information Sheet (printout)
- 33. Task Keywords • polynomials • Pascal‟s triangle • binomial coefficients • prime counting function
- 34. Tasks 1 & 2 •
- 35. Tasks 3 & 4 •
- 36. Counterbalanced Group Task Order 1 1 2 3 4 2 4 1 2 3 3 3 4 1 2 4 2 3 4 1
- 37. Search Conditions • Text books, notes, websites, and/or online search • Online search only without the min interface • min interface only • Online search only with the option of using the min interface
- 38. Introducing min • In between search condition 2 & 3 • Participant impression noted first • Keyboard & mouse-drawn modalities demoed • Upload modality described • All tools and search function demoed
- 39. min – hands-on use • Supports diagrammatic aspects of math notation • Affords preferred handwritten method of math input
- 40. 1. Does min change behavior? Metrics • Expression use in search query • Query length • # of query reformulations • Task time
- 41. 2. Real-world use? Metrics • Input modality used • Self-rated task success • Satisfaction ratings • Participant ideas
- 42. What else can we observe?
- 44. Goal 1 1. To observe whether min changes user search behavior • Yes.
- 45. Expression Use 2 0 16 10 0 12 16 0 2 4 0 2 4 6 8 10 12 14 16 18 Cond. 1: no min, open resources Cond. 2: no min, search Cond. 3: min req. Cond. 4: min (min opt.) Cond. 4: no min (min opt) n=14 n=16 n=16 n=12 n=4 TotalNumberofInitialQueries Without Expression With Expression
- 46. Average Time by Condition 112 209 133 132133 96 136 108 343 289 517 112 158 173 405 164 79 50 0 29 0 180 360 540 720 900 prime counting function polynomial pascals triangle binomial coefficient Seconds Condition 1 - no min, open resources Condition 2 - no min, online search Condition 3 - min, min req. Condition 4 - min canvas, min opt. Condition 4 - no min canvas, min opt. n=3 n=3 n=3 n=1 n=3 n=1 n=3 n=1 n=4 n=0
- 47. Task Success 0 4 8 12 16 20 24 28 32 Successful Somewhat successful Unsuccessful Without min With min
- 48. Finding a resource Without min • submit search query then “cherry pick” from search results With min • submit search query to multiple databases
- 49. Reformulating Without min • when reformulating a search query containing an expression, modifications made to the expression, as well as any text keywords With min • when reformulating a search query containing an expression, modifications made only to the keywords.
- 50. Goal 2 2. Discover real-world scenarios for math search interfaces • Yes.
- 51. Real-world use • 12 out of 16 participants (75%) identified scenarios where they would use min or could have used min in the past.
- 52. Participant Comment “when they start to get nasty and use a lot of Greek letters, it's hard to search a Greek letter”
- 53. Participant Comment “for more complex problems…even over wolfram alpha, a lot easier to put problems in and can still search wolfram”
- 54. Participant Comment “if searching for something with a radical or some weird symbol that‟s really hard to enter that in… I like that you can draw it knows what you're talking about and can detect it and you can search it right there … I don't have to Google and type in the term in place of the symbol”
- 55. Participant Comment “it would be really nice when you have a really long equation … like when using wolfram alpha a lot of the equations I put in there you have to put like 10 parentheses in it just to get it to work and it ends up taking at least 10 minutes to make sure you have it right so this would be nice to be able to actually just draw it out and have it recognize what you draw”
- 56. Participant Comment “thought it was neat you can just write it in because it's hard to Google or wolfram alpha equations”
- 58. Increased Expression Use • The affordance of the interface • The novelty of the interface • The ability for the interface to bridge the “intention gap”
- 59. Intention Gap
- 60. Intention Gap – bridged? “I was so surprised when it picked up on 4 choose 2.” “Like 4 choose 2 – that‟s really hard to „write‟ but it knew what I meant and it accurately translated what I was trying to say to it.”
- 61. 122.75 152.13 134.86 121.57 0 60 120 180 240 300 360 420 480 540 600 660 720 780 Seconds 263.71 242.00 461.00 134.29 0 60 120 180 240 300 360 420 480 540 600 660 720 780 Seconds Task Time by Expression Without min With min
- 62. FUTURE WORK min development & areas to explore.
- 63. min improvements • Typed and recognized expressions are now rendered using the online MathJax service • Handwritten strokes are now hidden after recognition • Now a button on toolbar brings up correction menu • Now allows operator shorthand in text input such as „x^2‟
- 64. min improvements – demo 1
- 65. min improvements – demo 2
- 66. Ideas for future work • testing the improved min interface • expressions with complex structure and notation • experimental comparisons tests • different populations (e.g., age, education, income) • usage patterns over time
- 67. Ideas for future work • field use rather than lab use • actual success rather than perceived success • other domains that use diagrammatic terms and non- keyboard characters
- 68. Acknowledgements For expertise, guidance and support.
- 69. Dr. Zanibbi | Primary Advisor
- 70. Drs. Yacci & Rozanski | Advisors
- 71. Dr. Agarwal | Task design
- 72. Awelemdy Orakue | Assistant
- 73. DPRL members | Technical Support
- 74. AAUW | 2009 CD Grant
- 75. Thank you.
Notas do Editor
- Hello and thank you for attending. My name is Keita Wangari and today I’m presenting my thesis titled Discovering Real World Usage for a Multimodal Math Search Interface.
- I’ll jump right into the problem I’m addressing. The Intention Gap.
- What do you know about this math expression?Do you know how to say it in words?Do you know the name of it?How did you learn it? In words? Or visually, just like it’s pictured here?How important is the visual layout to the meaning of this math expression?Could this actually be considered a diagram of sorts?If you’re a first year college math student, how could you get more information on it?
- Most people connected to the Internet are familiar with doing online searches for information. If we want to know something, we quote “Google it”.Searching is a familiar activity and this slim, rectangular search field is a familiar sight.
- In fact, most search interfaces include a slim, rectangular query field that seems to only want to accommodate text in a one-dimensional, sentence-like format.(click)Including 2 of the top math databases
- So given a 2-dimensional math expression like this.(click)How do you get it in the search field? How exactly do you formulate your query? How do you “google it”?
- If you know how to say the expression verbally (n choose k) you could type those words inIf you know what math terms or concepts relate to that expression (for example, binomial coefficient), you could type those inIf you know an encoding language like LaTeX, some search fields will accept thatAnd some search sites have a template editor, similar to what you might see in Microsoft Equation Editor, that would allow you to build the expression in the search field
- To start to understand whether those querying options are sufficient, we need to ask sufficient for who?And in this study, we’re concerned with the non-experts in the math domain and more specifically, college students in lower level college math courses.(click through)If this is truly the case, then the current querying options available to them may not be sufficient.
- When there is a gap between a user’s intentions and the system’s allowable actions, it is known as the Gulf of Execution (Norman & Draper, 1986). This study hypothesizes that a Gulf of Execution occurs at the point where a user has formed the intent to search for information related to a math expression and yet the interface does not support a method for doing so in a natural way or in a way that matches the user’s mental model. In the domain of search, the point at which a user is unable to precisely express their search intent as a keyword query has specifically been dubbed the Intention Gap (Zha et al, 2010)
- So how would you “Google it?”
- min prototype was created to allow a more natural option for users to express their mathematical search needweb-based multimodal interface that allows users to input a math equation by drawing it on a blackboard-like canvas using a mouse, pen or fingerthe canvas also accepts keyboard input and uploaded imagesmin application displays its recognition of the user’s input on the canvas, allowing for correction and/or manipulation. min offers expression manipulation tools such as symbol selection, stroke selection, and undo & redo, as well as an inline OCR correction menumin allows the user to submit the expression to various search engines directly from the interface. The interface also provides a conventional text search field allowing the user to submit keywords along with the expression
- “How do people use this new search interface?” and “How does this new search interface change search perception, motivation and behavior?”, in addition to observing how non-experts currently search for math-related information, the study is designed to take the additional step of actually putting multimodal query-by-expression technology in the hands of users to see what real-world scenarios can be discovered and to note any behavior changes introduced by the technology
- people draw conclusions from the visual detail of the diagram as well as from the information being presented in the diagram
- In the learning domain, one study of college students (that was later generalized to middle- and high-schoolers) compared various input modes such as keyboard, handwriting-only, and handwriting plus speech, and found that typed input was not optimal for representing mathematical notation and that test participants found handwriting-only to be the most natural and satisfactory.Additionally working with equation editors can be tedious as one hunts for appropriate symbols and equation layout changes are difficult.
- But existing math search interfaces constrict users to either type in keywords or an code via the keyboard, both missing the diagrammatic aspect of math(click)or use an editor
- study did not focus on mathematical non-experts and participants were not given a prototype for hands-on evaluation
- , it is hypothesized that query languages are not, in fact, natural for non-expert math searchersexplore this idea of “natural” expression of a math search need a bit deeper by observing the search behavior of non-expert math searchers both with and without the use of the min interface.The intent is to observe any change in search behavior when users are presented with a math search interface that provides affordances for the preferred handwritten method of math input and the familiar, diagrammatic aspects of math notation and discover any other new affordances as well as real-world use cases.
- Just a quick introduction to the expressions used in the study before I show how they were used in the tasks presented to participants. Expressions used were intended to be consistent with math challenges non-expert students face in a freshman or sophomore college math course(click click)The first two expressions used all standard keyboard characters with some used as superscripts(click)The third expression used all standard keyboard characters and superscripts as well and additionally gets more 2-dimensional used a division symbol(click)The fourth expression is completely sentential but uses a non-standard keyboard character, pi(click)The fifth expression uses all standard keyboard characters but uses a layout not easily replicated using a standard keyboard
- As you’ll see in the next 2 slides, keywords were also incorporated into the tasks expressions to allow us to observe which the participant would use in their search. These are thekeywords that were used … they reflect the math concepts that are associated with the expressions in the previous slide
- Participants were reminded that they did not need to solve any math problems nor explain any of the math concepts to the study moderator – their job was to find resources that would help them explain the math concepts in each task to their classmate
- The tasks were counterbalanced across the groups to allow each task to be the first task for an equal number of participants and to present each task uniformly across the search tool conditions
- Search conditions were presented to all participants in the same order
- So before moving on the results, I want re-emphasize our goals.We now have a tool that(click)And(click)We want to explore this idea of “natural” expression of a math search need a bit deeper by observing the search behavior of non-expert math searchers. Does it come “natural” for them to use an encoding language? Or a template editor? What will happen when they have a tool like this in their hands?
- Will it change their behavior? One of our goals is to observe that. In this study we decided to measure the following to find out.
- Is there a real-world use for it? Our second goal is to discover the answer to that. The things we decided to measure are:
- In addition to our main study goals, we were also interested in seeing anything else we could observe about the intention gap we feel our target audience faces … being unable to express a search query in a way that matches their mental model, in a way that feels natural, in a way that supports their level of math knowledge.
- There was a large increase in expressions used in the query when using min both in initial queries (the red column) and query reformulations (the blue column)(click)Before being introduced to min, only 2 out of 14 participants (14.3%) who chose online search in Search Condition 1 used an expression in their initial search query. (click)None of the 16 participants used an expression in their initial query in Search Condition 2 where online search was required.(click)However, when required to use min for the first time in Search Condition 3, expression use increased to 16 of the 16 participants (100%)(click)And even though most participants experienced a glitch of some type with min, in Search Condition 4, where min was optional, 12 still chose to use min and 10 of the 16 participants (62.5%) used an expression in their initial search.Figure 17 also shows that in non- min conditions (i.e., Search Conditions 1 and 2), participants made more use of expressions in query reformulations than in initial queries and that the use of expressions in query reformulations
- There was a noticeable increase in average task time when participants used min. In Search Condition 1 using open resources, the average task time across all tasks was 147.4 seconds (σ ± 88.7) and in Search Condition 2 using online search only, the average task time across all tasks was 118.87 seconds (σ ± 62.8). Both of those non- min conditions averaged roughly between 2 to 2 ½ minutes. However, when required to use min for the first time in Search Condition 3, average task time across all tasks jumped to an average of 315.19 seconds (σ ± 239.8) .This high average and variance in Search Condition 3 was caused by one participant spending almost 18 minutes on the Pascal’s Triangle task (Task 2) and another spending 11 minutes on the Prime Counting Function task (Task 4), while participants completing the Binomial Coefficient task (Task 3) under the same condition were all able to finish relatively quickly in roughly 2 minutes or less.
- Participants were asked to rate themselves as Successful, Somewhat Successful, or Not Successful upon completion of each task.The number of participants who rated themselves Successful in a task differed little between the non- min conditions (27 Successful) and min conditions (25 Successful)The biggest difference occurred in the self-ratings of “Somewhat Successful.” Twice as many participants rated themselves as “Somewhat Successful” when using min compared to not using min.
- There was also a behavior change noted in the way participants honed in on the resource they were looking forPerhaps enabled and encouraged by the drop-down list of search engines built into the min interface
- Perhaps difficulties encountered in inputting an expression made participants less likely to want to bother with it again. But it could also be that because participants were able to express the expression in diagrammatic form – a form more compatible with their consumption of math expressions – that they were much more comfortable with the expression portion of the search query when using min
- As opposed to the Zhao study where the expert participants could not imagine any use for query-by-expression capabilities, 12 out of 16 of our non-expert participants, after using min, could identify scenarios where they could have or would use min.
- The min interface is designed with a large open, blank area with several tools located horizontally across the top. Upon viewing the interface for the first time, without using it or knowing its function, several participants mentioned how similar in appearance it was to a smartboard. When asked what the purpose of the interface was, the majority said “drawing.”When provided with a demo of the interface, several participants clearly showed visible and audible signs of being fascinated and impressed and seemed eager to try it themselves. In the post-study interview, about a quarter of the participants who chose drawing over typing cited “new and intriguing” as being their reason for making that choice.
- none of the participants were observed using any such methods (e.g., LaTeX, template editors) for expressing their math search need although a few claimed to be familiar with them. Admittedly, the structure of the expressions in 3 of the 4 tasks did not require any special encoding in order to be entered into a standard search engine text box. But for the one task that did, the Binomial Coefficient task containing the expression “4 choose 2”, most participants not only did not attempt to use any special encoding languages, but many were not familiar with how to say the expression in words. They recognized it visually but beyond that, were unclear how to express it in sentential form – the only form afforded by the conventional search text box
- has a relatively simple visual layoutparticipants had difficulty expressing it verbally and textuallyimpressed at min’s ability to recognize correctlynon-experts in the math domain deserve special consideration in the design of math search interfaces. They are simply not familiar enough with math encoding languages and template editors to employ them readily in a math search queryThis study was designed to target the non-expert math searcher as the feeling was that this is not a population familiar with LaTeX and other math encoding languages nor with template editors nor with expression names. Our observations may have proven that feeling correct – none of the participants were observed using any such methods (e.g., LaTeX, template editors) for expressing their math search need although a few claimed to be familiar with them. Admittedly, the structure of the expressions in 3 of the 4 tasks did not require any special encoding in order to be entered into a standard search engine text box. But for the one task that did, the Binomial Coefficient task containing the expression “4 choose 2”, most participants not only did not attempt to use any special encoding languages, but many were not familiar with how to say the expression in words. They recognized it visually but beyond that, were unclear how to express it in sentential form – the only form afforded by the conventional search text box. This observation seemed to indicate that we had, indeed, targeted an audience where a tool such as min might be beneficial and also confirmed the need to look at math experts and non-experts separately and confirmed the suspicion that math encoding languages and structure editors are not well within the grasp of non-expert math students.
- Additionally, when looking at average task time by task, rather than by condition, the times were all higher using min (Figure 20) compared to not using minWorth noting, particularly for understanding how min is most useful and where it can be improved, is that the biggest increase in average task time was with the Pascal’s Triangle task (Task 2). That task, whose expression contained several terms and had the most superscripts in the study, averaged 134.86 seconds (σ ± 74.25) across both non- min conditions and increased more than 179% to an average of 376.43 seconds (σ ±167.72) across both minconditions.In comparison, the Binomial Coefficient task (Task 3) had the smallest task time increase when using min. That task, whose expression ( 42 ) has a smaller number of terms and a relatively simple visual layout, averaged 121.57 seconds (σ ± 71.28) across both non- min conditions and increased only 10% to an average of 134.29 (σ ± 45.05) seconds across both min conditions. When completed under the min conditions, this task also had the lowest variance of all the per task time averages in the study – participants were more consistent in executing this task using min.Additionally, when looking at average task time by task, rather than by condition, the times were all higher using min (Figure 20) compared to not using minWorth noting, particularly for understanding how min is most useful and where it can be improved, is that the biggest increase in average task time was with the Pascal’s Triangle task (Task 2). That task, whose expression contained several terms and had the most superscripts in the study, averaged 134.86 seconds (σ ± 74.25) across both non- min conditions and increased more than 179% to an average of 376.43 seconds (σ ±167.72) across both minconditions.In comparison, the Binomial Coefficient task (Task 3) had the smallest task time increase when using min. That task, whose expression ( ■8(4@2) ) has a smaller number of terms and a relatively simple visual layout, averaged 121.57 seconds (σ ± 71.28) across both non- min conditions and increased only 10% to an average of 134.29 (σ ± 45.05) seconds across both min conditions. When completed under the min conditions, this task also had the lowest variance of all the per task time averages in the study – participants were more consistent in executing this task using min.
- The previous algorithm often produced unfamiliar symbol layouts(click)There was some participant feedback about visual clutter causing difficulty(click)method for correcting the grouping of strokes into symbols was overlooked by some participants and not easily used by others – correcting OCR results required a press-and-hold gesture on a selected symbol and, along with interpreting the recognized expression(click)Participants were observed trying to type in operator shorthand on the canvas and the min interface wasn’t designed to handle that. One participant even requested that capability during the discussion.(click)We think these improvements in the min interface may reduce observed increases in search task completion time
- Here you can see the drawn strokes being hidden as min recognizes the inputHowever those strokes are retained and are visible when the user goes into edit mode as you see hereAnd if I want to access a menu of similar symbols to correct what I have, there’s now a button on the toolbar allowing that