Learn how Machine Learning influences UX in this presentation by Smashing Boxes Designer Bonnie Cook.

1. UX for Machine Learning
Presented by Bonnie Cook

2. THE (WO)MAN AND THE MACHINE

4. STARTER UX ML QUESTIONS
I. Where do I find information about UX for ML?
II. Are there resources like tutorials? Demos? Thought
leaders?
III. How is machine learning going to change the future of UX?
a. Opportunities and setbacks
IV. Revolutions will take hold when tools become highly
mature and more accessible (for designer/dev/user needs)
a. We have not reached this level of expertise yet

5. ML RENAISSANCE
I. “Machine learning is in the midst of a renaissance that will
transform countless industries and provide designers with
a wide assortment of new tools for better engaging with
and understanding users. These technologies will give rise
to new design challenges and require new ways of
thinking about the design of user interfaces and
interactions.” - Patrick Hebron, Machine Learning for Designers

6. UX FOR MACHINE
LEARNING
SECTION ONE
1

7. LET’S START WITH “WHAT’S UX?”
I. “User Experience (UX) design is the process of creating
products that provide meaningful and relevant
experiences to users. This involves the design of the entire
process of acquiring and integrating the product, including
aspects of branding, design, usability, and function.”
-The Interaction Design Foundation

8. THEN, MACHINE LEARNING?
I. “Machine learning is a field of artificial intelligence that
uses statistical techniques to give computer systems the
ability to “learn” (e.g., progressively improve performance
on a specific task) from data, without being explicitly
programmed.”
-Wikipedia
II. Just in case: Algorithms are a process or set of rules to be followed in calculations or
other problem-solving operations, especially by a computer.

9. THE EVOLUTION OF HCI
SECTION TWO
2

10. USES OF HCI OVER TIME
I. 1980’s - 1990’s: Desktop and Mental Models
II. 1990’s - Early 2000’s: Collaboration and Communication
III. Mid 2000’s - 2010: Self Expression and Social Change
IV. Now - Future: Crafting technologies to the human
experience

11. ML - A DIFFERENT KIND OF LOGIC
I. Fuzzy logic - logic formed on approximations versus exact
a. Need auxiliary knowledge of range of values
i. Car sizes
1. Approximation of range
2. Can meaningfully discuss with other people
II. Computers have “lived a life of experiential deprivation.”
- Patrick Hebron

12. “PRIOR” TO MACHINE LEARNING
I. Systems are used to Boolean logic where every expression
has to ultimately evaluate as either true or false
II. Machine Learning challenges this logic by giving machines
experiential knowledge, which then helps deal with the
fuzzier/human logic
a. “Design challenges the problem of working with
imprecise technology and unpredictable behavior.” - Ibid

16. FACE VALUE
I. Infinite number of combos, genders, vantages,
environments, races, etc…
II. Examples of how we cannot reach perfect recognition with
faces, despite a lifetime of examples

17. 20 QUESTIONS
I. Ways we network with others
a. Not sure what things about you will overlap (job etc)
II. For conversational user interfaces
a. Make sure you have the ability to create common
ground with the users and the things the system can
do to become better acquainted
III. Works as a learning decision tree

18. DESIGN THINKING
SECTION THREE
3

19. DESIGN THINKING
“Design thinking is a human-centered approach to
innovation that draws from the designer’s toolkit to integrate
the needs of people, the possibilities of technology, and the
requirements for business success.” - Tim Brown, president and CEO, IDEO

20. UX & DESIGN THINKING
I. Design Thinking is a process or methodology. It is about
applying a specific way of thinking to a situation.
II. Design Thinking methods can be utilized not only by UX
designers, but also developers, product owners and
marketers.
III. Creative insight isn’t limited to just designers
a. Designer’s emphasis on criticism

21. DT & ML STAGES
I. Design Thinking Stages
a. Empathize
i. Who is my user?
ii. Pain points
b. Define
i. Point of View
c. Ideate
i. Brainstorming
d. Prototype
i. Working and ready
e. Test
i. Monitor Use
ii. Effectiveness
I. Machine Learning Stages
a. Analyze
i. Capture key decisions to find out
variables/metrics
b. Synthesize
i. Combo of separate elements to create new
c. Ideate
i. Small sample of data applied to various
analytical models/algorithms for insight
d. Tuning
i. Additional data capture
e. Validate
i. UX and analytic model tuning perspectives
Source: Abhay Pandey - Medium

22. WHO’S LISTENING?
SECTION FOUR
4

24. YOU AND ML ON THE DAILY
I. Siri - taught to understand the nuances of our language
II. Facebook - uses algorithms to recognize faces from contact list
III. Google Maps - traffic speed/time and best possible route
IV. Google Search - recommendations based on searches
a. 2012 - Introduced Knowledge Graph - algorithm used to decipher semantic
content of a search query
V. PayPal - ML to fight off fraud. Analyzes tons of consumer data and evaluate risks
VI. Netflix - video recommendation engine
VII. Uber - algorithms to determine arrival times, pick up locations, and UberEATS’
VIII. Lyst - match customer searches with relevant rec. Meta-data tags for visual comp.
IX. Spotify - ML to establish likes and dislikes and provides list of related tracks

25. Phoneme - Any of the perceptually distinct units of sound in a specified language that distinguish one
word from another, for example p, b, d, and t in the English words pad, pat, bad, and bat.

26. Image: Sam Drozdov

28. CURATION VS ALGORITHMS
I. Apple Music has DJs who custom create playlists rather
than how Spotify or Pandora works
a. Designers need to be able to design BOTH algorithmic
data and curated data - they’re equally important
personas
b. What data do you have?
c. What data do you want?

29. ML LANGUAGE
SECTION FIVE
5

30. UNDERSTANDING ML LANGUAGE
I. Machines like Siri understanding spoken language is a big
deal because it allows development of more natural
interaction paradigms
II. Diverse vocals/speech patterns
III. Speech-to-text sometimes struggles for humans and
computers alike (Shazam and SoundHound)

31. “GAMING THE ALGORITHM”
I. Pamela Pavilscak, author of Emotionally Intelligent Design, talks about the
strategic evasion of computers knowing too much about us
a. Then we’re disappointed that we get a bunch of info not related to us
i. Private browsing, ads vs ad Google search
b. Algorithms face backwards, not forward
i. “Making predictions on past behaviors doesn’t take into
account what is essentially human. We change a little every
day. We get interested in new things. We change our minds. We
feel a little conflicted. Algorithms have a hard time with that.”

32. OTHER ALGORITHMIC SETBACKS
I. They’re working with the big picture of what you do
a. Topics based on behavior
II. Incomplete information - algorithms keep up with big
purchases but other data points can be inaccurate
III. Resources - history of sites (anticipatory) but doesn’t
understand personalization completely

33. ML EMPATHY
SECTION SIX
6

34. A BEHAVIORAL PROBLEM
I. “We are more than just the sum of our behaviors. We don’t
behave rationally, maybe not even predictable irrationally. A
human presence can improve the algorithmic experience.” -
Pamela Pavilscak

35. ALGORITHMIC EMPATHY
I. Algorithms change the way we explore and understand
empathy with users
II. Pamela Pavilscak’s process to
“Cultivate Algorithmic Empathy”
a. Research practices: Data Role Play, Algorithm Swap,
Data Doubles, Algorithmic Personas, and Shared
Mythologies (broken out on next slides)

36. DATA ROLE PLAY
I. It acknowledges the abstraction of algorithms
a. Good way to start the conversation about this new type
of empathy
b. Would it be awkward to ask a stranger for private
information like email and phone number?
i. Even worse - getting their info without them
knowing it

37. ALGORITHM SWAP
I. We have very private experiences with our devices
II. Might change once we start interacting with voices rather
than hands
a. Right now it feels unnatural to spend time with
someone else’s private self
i. YET, can be meaningful to see that side of the
person you’re designing for

38. ALGORITHMIC PERSONAS
I. Design teams chose a base persona on a combination of:
a. Demographics
b. Interviews
c. Behavioral data that is collected

39. SHARED MYTHOLOGIES
I. When we reach an algorithmic disconnect, people start to
speculate as to WHY it’s no longer working
II. So INSTEAD of letting myth after myth float around
a. Design to reveal the algorithm
i. “See yourself through the ads you encounter” like on
Facebook etc…

40. ML iOT DEVICES
SECTION SEVEN
7

41. DESIGNERS AND iOT DEVICES
I. “Allows designers to discover implicit patterns within
numerous facets of a users behavior. These patterns carry
inherent meanings, which can be learned from and acted
upon, even if the user is not expressly aware of having
communicated them. In this sense, these implicit patterns
can be thought of as input modalities that, in practice,
serve a very similar purpose to more tangible input modes.”
- Patrick Hebron

43. DISNEY AND MACHINE LEARNING
I. Eventually Disney wants “a system that has gathered so
much data about a user over such a long period of time
that the anticipatory design creates a completely
different experience for each person, despite the fact
that all the users are in the same physical location.”
-Bryon Houwens

44. GO WITH THE FLOW
SECTION EIGHT
8

45. TRADITIONAL UX FLOW Graphic: VladcDesign

46. DESIGN THINKING FLOW Graphic: IDEO

47. ML/PREDICTIVE MODELING FLOW
Graphic: Class Central

48. DATA SCIENCE
I. Look like familiar UX pattern/flows?
a. Divergence and convergence
II. Predictive modeling is the scientific process within
machine learning that is incredibly important for an
aligned UX methodology and practice
III. We can only do but so much with current material
a. Present opportunities for growth

49. CONCEPT TO CREATION
SECTION NINE
9

50. USING DEDUCTIVE & INDUCTIVE
REASONING
I. Helps designers with holistic understanding of actions
II. General rules for observing and collecting data
a. “Deductive - broad theory about rules governing a
system, distill theory into more specific hypotheses,
gather specific observations and test against our own
hypotheses to confirm if original theory was right or
wrong.”- Patrick Hebron

51. USING DEDUCTIVE & INDUCTIVE
REASONING
I. “Inductive reasoning starts with a group of specific
observations that look for patterns in those observations,
formulate tentative hypotheses, and ultimately try to
produce a general theory that encompasses original
observations.” - Patrick Hebron

52. STORYTELLING
I. Storytelling for developers and designers to visualize something together
a. More than just screens that user interacts with, but understand why
things are happening behind the curtains
II. Importance of storytelling in the beginning of product creation
a. The experience impacting the user
i. Storyboards
ii. Prototypes
iii. Strategy Decks
iv. Diagrams
III. Empowerment as a shared vision and language with fewer steps

53. AIRBNB
I. Were creating a model to answer question “what will the booked
price of a listing be on any given day in the future?”
a. Developers were talking alphas and betas - foreign material to
many designers
II. Asked developers to sketch out the idea in a diagram and do a
walk-through
a. THIS is where the connection was made
III. Language barrier broken down and then created a shared
language about the product and moving forward (storytelling)

54. SKETCHES AND DIAGRAMMING
Amber
Cartwright,
AirBnB
*Smart Price
Regression
model next to a
visualization
explaining the
model is made
of three parts
that vary by host.

55. UX ML BEST
PRACTICES
SECTION TEN
10

56. ML UX BEST STEPS - PATRICK H.
1. Design tasks explicitly so that users can catch errors and redirect system
behavior.
2. Fallback mechanisms - circumvent in ML functionality and perform tasks
with explicit logic.
3. Test in as many environments and limited audience release.
4. Use metrics—confidence scores—to assess feature. Realistic expectations.
5. Consider impressive sounding feature if unreliable. Failure vs feature.
6. Make risks obvious. Allow users to decide if benefit outweighs risk.
7. Serious consequences—even death—in system failure. Take extreme
caution to assess risk/liabilities.

57. ML PROTOTYPING TOOLS
SECTION ELEVEN
11

58. WHY PROTOTYPE?
I. To understand - “freedom to think through all the different
ways you could solve the problem, discover new problems
that need to be addressed, and help you refine your ideas
with the feedback you receive.”
II. To test and improve - “the main reason to prototype”
III. To communicate - “invest team, stakeholders, or end users”
IV. To advocate - “for the design or direction”
- Kathryn McElroy, Prototyping for Physical and Digital Products

59. ML PROTOTYPING - WEKINATOR
I. ML challenges rapid prototyping because of architectures and training
II. Requisite of code or data sets limits tools (evolving)
III. Currently, there are some existing tools we can use
a. Wekinator - it’s free and open source
i. Allows development of experimental gesture recognition and
interface controllers for microphones, webcams, Kinect etc...
ii. Programming free!
iii. “Walks designer through process where defines particular
gesture by demoing it to machine and then associates gesture
with the desired output action.” -Ibid

60. QUICK DEMO - WEKINATOR
http://www.wekinator.org/walkthrough/

61. MATHEMATICA
I. “Tool features a polished UI and does not require deep
understanding of programming, though some basic
familiarity with text-based scripting is helpful
II. Machine learning features applied to wide range of data
types and auto data preprocessing and model selection
features will help users get good results without a great
deal of trial-and-error or deep knowledge of a particular
model’s training parameters.” -Patrick Hebron

62. MATHEMATICA CONTINUED
I. “Provides turnkey support for range of common ML tasks
like image recognition, text classification, and classification
or regression of generic data.
II. Datasets can be loaded through interactive, visual interface
III. Extremely well documented and embeds assistive tools like
feature suggestion and autocompletion directly to its
interface” -Ibid
IV. Not free www.wolfram.com/mathematica

63. QUICK DEMO - MATHEMATICA
http://www.wolfram.com/broadcast/video.php?c=89&v=358

65. WRAPPING IT UP
I. Time to make that prototype and test with people
a. Are they happy? Are they freaked out? Do they trust
you? Do they trust the device(s)?
b. Observe, listen, chronicle.
c. Why? What can you do? What can’t you do?

66. KEEP EXPLORING
I. HCI designers prototyping process post product release
II. UX for Machine Learning is growing at a rapid pace and
keep in mind we’re all figuring it out—experts included
III. Use tried and true methods where it is appropriate
IV. Remember, YOU already use machine learning—think like
another user
V. Errors are opportunities

67. THAT’S ALL, FOLKS!
Bonnie Cook
bonnie@smashingboxes.com

68. TAKEAWAYS
READ
Patrick Hebron - @patrickhebron - www.patrickhebron.com
Pamela Pavliscak - @paminthelab - Emotionally Intelligent Design
LEARN
Wekinator - http://www.wekinator.org/
Mathematica - http://www.wolfram.com/mathematica/
LET US HELP!
SB Design Sprints - 2-hour to 2-day discovery sprint to help identify your
needs, personas and design schemas.
Prototype Workshop - Come with an idea, leave with a clickable prototype
Dave Shepley - dave.shepley@smashingboxes.com @smashingboxes