The document discusses how computational methods and machine learning could potentially be used to automate aspects of interface design. It begins by providing background on how previous research has explored using optimization algorithms and predictive models from psychology to generate keyboard layouts and other interface designs. The document then discusses challenges, noting that interface design problems involve large search spaces and complex evaluation functions. It argues that a coordinated effort is needed to develop methods that can produce good outcomes while being compatible with design practice. Overall, the document explores the possibility of using computational approaches to automate parts of interface design, while acknowledging the difficulties in developing generally applicable methods.

1. CAN COMPUTERS
Antti Oulasvirta
Associate Professor
userinterfaces.aalto.fi
design?
Hands-on demo of
Sketchplorer at CHI’16
in San Jose; May 2-7

2. IS A COMBINATION OF
DESIGN THINKING AND
COMPUTATIONAL THINKING
COMPUTATIONAL INTERFACE DESIGN
This talk shows how algorithms can use predictive models
to design user interfaces. If successful, this program has
significant implications to interaction design practice.

3. 26/02/16 08:38How Many of Your Daily Tasks Could Be Automated?
OPERATIONS
How Many of Your Daily Tasks Could Be
Automated?
by Michael Chui, James Manyika, and Mehdi Miremadi
DECEMBER 14, 2015
A sea of ink and hours of video have been dedicated to the looming battle between humans and machines, often concluding that the machines
win and human workers lose. It is a nearly irresistible narrative, which has been captured in books such as Martin Ford’s Rise of the Robots and
The Second Machine Age, by Erik Brynjolfsson and Andrew McAfee. It has also has inspired scholarship by academics such as Carl Benedikt Frey
and Michael Osborne of Oxford University, who estimate that 47% of occupations in the United States could be automated within 20 years, and
David Autor of MIT, who argues that the ability of machines to take on human jobs is vastly overstated.
At McKinsey, we are conducting our own research into the impact of automation on jobs and organizations and recently published a brief paper
previewing some of our findings. The most important insight has been that it is far more useful to think about the activities that can be
automated rather than entire occupations—handing over discrete bits to machines, like pulling out the most useful data from an analyst’s report,
generating a report on the latest sales figures, or moving products around a warehouse. Smart machines have already demonstrated the ability to
see patterns in information, understand what humans are saying (answering a question like “show me where sales rose the most last week”),
and manipulate physical objects. Once these capabilities are applied to various work activities, few occupations or organizations will remain
untouched.
Let’s look at what happens if we approach automation at the level of activities, rather than occupations. Take, for example, the role of a

Want more out of
HBR.org?
Sign in to get more free content
month and build your own personal
library on HBR.org.
O K , T H A N K S

4. HOW MIGHT DESIGN BE AUTOMATED?
26/02/16 08:38How Many of Your Daily Tasks Could Be Automated?
Page 1 of 4https://hbr.org/2015/12/how-many-of-your-daily-tasks-could-be-automated
OPERATIONS
How Many of Your Daily Tasks Could Be
Automated?
by Michael Chui, James Manyika, and Mehdi Miremadi
DECEMBER 14, 2015
A sea of ink and hours of video have been dedicated to the looming battle between humans and machines, often concluding that the machines
win and human workers lose. It is a nearly irresistible narrative, which has been captured in books such as Martin Ford’s Rise of the Robots and
The Second Machine Age, by Erik Brynjolfsson and Andrew McAfee. It has also has inspired scholarship by academics such as Carl Benedikt Frey
and Michael Osborne of Oxford University, who estimate that 47% of occupations in the United States could be automated within 20 years, and
David Autor of MIT, who argues that the ability of machines to take on human jobs is vastly overstated.
At McKinsey, we are conducting our own research into the impact of automation on jobs and organizations and recently published a brief paper
previewing some of our findings. The most important insight has been that it is far more useful to think about the activities that can be
automated rather than entire occupations—handing over discrete bits to machines, like pulling out the most useful data from an analyst’s report,
generating a report on the latest sales figures, or moving products around a warehouse. Smart machines have already demonstrated the ability to
see patterns in information, understand what humans are saying (answering a question like “show me where sales rose the most last week”),
and manipulate physical objects. Once these capabilities are applied to various work activities, few occupations or organizations will remain
untouched.
Let’s look at what happens if we approach automation at the level of activities, rather than occupations. Take, for example, the role of a
marketing manager or executive in a consumer products company. A marketing manager is well educated and well compensated—he or she

Want more out of
HBR.org?
Sign in to get more free content
month and build your own personal
library on HBR.org.
O K , T H A N K S
loading activities to machines be used most effectively? When you’re bringing in automation you need to think about two kinds of payoff—
returns you get by using machines rather than labor for activities (investments in automation can generate benefits worth three to ten times the
cost, we estimate, much of it from better performance rather than reductions in labor costs), plus the value derived from activities that
employees carry out in the time that formerly used for work that is now automated. In the case of the marketing manager, this could mean more
time to work on new product ideas, supervise direct reports, collaborate with managers in other functional areas, or develop new strategies.

5. WHAT IF…
INTERFACE DESIGN WAS ENGINEERING DESIGN?

6. POTENTIAL BENEFITS
IMPROVED QUALITY AND RELIABILITY
GUARANTEES FOR OUTCOMES
SOLUTIONS TO HARD PROBLEMS
SCRUTINIZABLE PROCESS

7. AUGUST DVORAK
The first to use empirical findings to
drive the design of keyboard layouts
•Designed DSK (Dvorak Simplified
Keyboard)
• Demonstrated 5-10% improvement
in typing speed over Qwerty
“Typewriting Behavior” 1936
PIONEER OF RATIONAL DESIGN

8. EARLY ATTEMPTS TO OPTIMIZE INTERFACES
1983
HCI
Card, Newell, & Moran
suggested that HCI models
should drive UI design.
Led a revolution in HCI
modeling
1993
Human Factors
Donald L. Fisher: “A quiet
revolution is in progress in
the field of human factors.”
Demonstrated in the design
of control panels
1977
Optimization
Rainer Burkard defined
mathematical design of
typewriter layouts (QAP)
Theorists had no real
interest in applications

9. Design Issues, Volume 17, Number 4, 2001, pp. 44-50.
CAN A MACHINE DESIGN?
NIGEL CROSS
Department of Design and Innovation
Faculty of Technology, The Open University
Milton Keynes MK7 6AA, UK
Abstract: One strand of my research has been concerned with the computer as a design
tool; but a second strand has been concerned with design computing as a research tool for
improving our understanding of the design process. Some of this latter research is based
on the simulation of computer behavior by human beings - a reversal of the more usual
approach - and some is based on comparisons of computational models with human
design behavior. Despite recent doubts expressed by some authors, I suggest that the
question, ‘Can a machine design?’ is still a useful question to ask.
Introduction
Asking ‘Can a machine design?’ is similar to asking ‘Can a machine think?’
The answer to the latter question seems to be, ‘It all depends on what you mean
by “think”.’ Alan Turing (1950) attempted to resolve the question by his
‘Turing Test’ for artificial intelligence - if you could not distinguish, in a blind
test, between answers to your questions provided by either a human being or a
machine, then the machine could be said to be exhibiting intelligent behavior,
i.e. ‘thinking’.
In some of my research related to computers in design, I have used
something like the Turing Test in reverse - getting human beings to respond to
design tasks as though they were machines. There have been various intentions
behind this strategy. One intention has been to simulate computer systems that
do not yet exist; another has been to try to shed light on what it is that human
designers do, by interpreting their behavior as though they were computers. My
assumption throughout has been that asking ‘Can a machine design?’ is an
appropriate research strategy, not simply for trying to replace human design by
machine design, but for better understanding the cognitive processes of human
design activity. However, this assumption has been challenged recently. In this
paper I will first review some of my research, and then return to this challenge.
For computers to design, we should first
understand how designers design

10. Harold Thimbleby (1998)
proposed formal definition of
interactive devices.
•UI specifications would drive
the generation of manuals,
helps etc.
•Could not generate designs,
but compare them
Ranjitha Kumar et al. (2013) proposed
WebZeitgeist, a pattern miner that suggests
user interfaces to queries.
RECENT ATTEMPTS IN HCI
UI Specification Design Mining
top nav bar
layout query
large
text
block
large header
Figure 13. Five of the 20 search results for the three-part DQL layout query visualized on the left. The query, which executed in 2.1 minutes,
returns pages that share a common high-level layout, but exhibit different designs.
Machine Learning
Webzeitgeist also enables a new kind of design-based ma-
chine learning. For the first time, applications can stream
structured visual descriptors for page elements from a cen-
tral repository. Moreover, Webzeitgeist’s extensible architec-
ture allows new data to be collected and integrated with the
repository for supervised learning applications, for instance
comparison, Figure 15 shows nearest-neighbor results for the
top query in Figure 14 using only the vision-based GIST de-
scriptors. While these elements are visually reminiscent of
the query, they bear little structural or semantic relation to it.
DISCUSSION AND FUTURE WORK

11. COORDINATED
EFFORT IS NEEDED
THE ELEMENTS ARE THERE
We need a method that is (i) generalizable, (ii) produces
good outcomes, (iii) is compatible with design practice.

12. can computers design?
antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMIZATION
FOUNDATION / DESIGN AS SEARCH
RESULTS / BEYOND KEYBOARDS
CONCLUSION / A FUTURE
DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER

13. can computers design?
antti oulasvirta
“Math is easy; design is hard.” — Jeffrey Veen
FOUNDATION / DESIGN AS SEARCH

14. HERBERT A. SIMON
Complex problem-solving activities such
as chess can be analyzed as search in a
problem space.
This foundational concept impacted
computer science, design, and cognitive
sciences.
Problem-solving as search
“THE SCIENCES OF THE ARTIFICIAL”

15. DESIGN AS SEARCH
• The optimum is the design
vector X with the lowest value
• Constraints can be added
• E.g., g(X) < 1
Find X =
8
>><
>>:
x1
x2
· · ·
xn
9
>>=
>>;
which minimizes f(X)
Design variable A
Variable B
Objective

16. Prohibitively large design spaces
Multiple design objectives
Multiple user groups and tasks
A hierarchical menu with 50 items can be ordered in 10

17. MULTI-OBJECTIVE OPTIMIZATION
HOW COMPUTER UNDERSTANDS DESIGN
max
d2D
X
c2C
X
u2U
X
t2T
X
o2O
wcwuwtwoGc,u,t,o(d)
Find the design
that maximises
the weighted
sums of…
…the contexts
of use …the user
groups
…the user tasks
…the design objectives

18. BECAUSE THE
MATH IS HARD
DESIGN IS HARD
Interface design entails very large, high-dimensional
search spaces and complex evaluation functions.

19. can computers design?
antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMIZATION
FOUNDATION / DESIGN AS SEARCH
RESULTS / BEYOND KEYBOARDS
CONCLUSION / A FUTURE
DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER

20. can computers design?
antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMIZATION

21. AS PHYSICS IS TO ENGINEERING
OPTIMIZATION
PSYCHOLOGY IS TO INTERFACE
OPTIMIZATION…
The use of predictive models allows controlling the design process,
scrutinizing assumptions, and explaining the outcomes.

22. MODEL-BASED INTERFACE OPTIMIZATION
OVERVIEW
Designer
Generate
Evaluate
Optimizer
Model
Psychology
Neurosciences
Social sciences
Computer science
Design studies

23. Black-box
methods
(e.g., simulated
annealing)
Optimizer
Fitts’ lawBigrams
Objective function
literature. Our approach is directly usable in the available IP
solvers such as CPLEX and Gurobi.
BACKGROUND: THE LETTER ASSIGNMENT PROBLEM
Given n letters and n keyslots, the task of the letter assign-
ment problem is to minimize the average cost ck` of selecting
letter ` after k weighted by the bigram probability pk`:
min
X
k
X
`
pk` · ck` (1)
It is an instance of the Koopmans-Beckman Problem [15, 6],
which is NP-hard. As we will see later, this problem can be
modeled with a quadratic function on binary variables. Thus,
it belongs to a broad class of problems called quadratic as-
signment problem (QAP) where the goal is to minimize the
total pair-wise interaction cost [22, 24].
Except for a few papers using heuristic cost functions (e.g.,
[9]), the Fitts-bigram energy is used (see also [4]). Here, the
OVERVIEW
We formula
lem in order
and-bound.
crete optimi
lem was am
text [21]. W
present tech
A detailed r
per. The int
et al. [18].
The basic id
tions and in
Integer Prog
solution that
gers to the v
sibility Prob
Task
n!##
Open#
Save##
Save#as...#
Close#
...#
About#
n!#
A B C D E F G H I J
K L M N O P Q R
S T U V W X Y
A
A B C D E F G H I
J K L M N O P Q R
S T U V W X Y Z
A B C D E F G H I
J K L M N O P Q R
S T U V W X Y Z
Letter assignment
KEYBOARD LAYOUT OPTIMIZATION
EXAMPLE

24. Figure 12. The Metropolis keyboard (43.1 wpm)
encapsulated the circle it replaced and filled the gaps
between the circles, making more efficient use of the total
space.
The performance of this layout is 43.1 wpm2
based on the
Mayzner and Tresselt digraph table. This is over 40%
improvement over QWERTY and more than 10%
q
m
v u
p
k
z
xy
w
l
d
b
cst
i
r
o
ng
j
a
sp
Figure 9. Early stage of Metropolis random walk. The
system is at a high energy state, moving towards
lower energy state.
[Zhai et al. 2001]
t = a + b log2(D/W + 1)
Simulated annealing
EXAMPLE: THE METROPOLIS KEYBOARD

25. CODE CAN EXPRESS DESIGN KNOWLEDGE
• If-then rules (e.g., design heuristics)
• Mechanistic models (e.g., mental models theory)
• Conceptual models (e.g., task analysis)
• Regression models (e.g., Fitts’ law)
• Statistical models with structure (e.g., TAM)
• Simulations (e.g., ACT-R)
PREDICTIVE POWER IS THE PRIORITY

26. Model of Visual Search and Selection Time in Linear Menus
Gilles Bailly1
Antti Oulasvirta1
Duncan P. Brumby2
Andrew Howes3
1
Max Planck Institute for Informatics and Saarland University
2
UCL Interaction Centre, University College London
3
School of Computer Science, University of Birmingham
ABSTRACT
This paper presents a novel mathematical model for visual
search and selection time in linear menus. Assuming two
visual search strategies, serial and directed, and a pointing
sub-task, it captures the change of performance with five fac-
tors: 1) menu length, 2) menu organization, 3) target posi-
tion, 4) absence/presence of target, and 5) practice. The novel
aspect is that the model is expressed as probability density
distribution of gaze, which allows for deriving total selec-
tion time. We present novel data that replicates and extends
the Nielsen menu selection paradigm and uses eye-tracking
and mouse tracking to confirm model predictions. The same
parametrization yielded a high fit to both menu selection time
and gaze distributions. The model has the potential to im-
prove menu designs by helping designers identify more ef-
fective solutions without conducting empirical studies.
Author Keywords
Linear menus; User Performance; Mathematical predictive
models; Visual search; Eye-tracking.
ACM Classification Keywords
H.5.2. Information Interfaces and Presentation (e.g. HCI):
Miscellaneous
INTRODUCTION
Menus are used on numerous applications and systems for
presenting, organizing, and selecting commands. However,
designers reportedly struggle with menu design [5]. While
many research papers focus on novel interaction techniques
(e.g. [38]), and others on human factors (e.g. [6, 9, 32,
33, 36]), surprisingly few develop predictive models of users’
performance with menus [7, 8, 10, 12, 14, 16, 21, 28]. Pre-
dictive models are an efficient way of encapsulating scientific
knowledge. They synthesize phenomena that are typically
fragmented across several studies, and they capture subtle in-
teractions among multiple factors in an empirically verifiable
form. They can help designers to design more efficient menus
by informing choices without expensive empirical studies.
Once matured enough, models can be incorporated into in-
teractive design tools [5, 16, 25].
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full cita-
tion on the first page. Copyrights for components of this work owned by others than
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-
publish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
CHI’14, April 26–May 1, 2014, Toronto, Canada.
Copyright c 2014 ACM ISBN/14/04...$15.00.
http://dx.doi.org/10.1145/2556288.2557093
Models of menu performance can be divided into two cate-
gories: cognitive simulations and mathematical models. Sim-
ulation models such as ACT-R models [3, 8] and EPIC mod-
els [19, 21, 27] explain the progression of search by reference
to underlying cognitive processes, such as perception, atten-
tion, and memory. Mathematical models such as the SDP
model [12, 14] are nonlinear regression equations that pre-
dict selection time as a function of external factors such as
menu length, target position and practice. These models ex-
pose fewer details about the process but may be less complex
and more straightforward to apply.
Our goal is to advance the scope and validity of mathematical
models of linear menus, a widely used menu type in desk-
top applications and consumer electronics. We assume that
tasks, including both searching and pointing, are performed
using one of a number of strategies. In practice, the strat-
egy space for perceptual/motor tasks, even very simple ones,
is known to be large [23, 39] but here we explore a few key
points in the space. We focus on two searching strategies: 1)
Directed search makes use of knowledge, including knowl-
edge of locations acquired through practice, alphabetic and
semantic knowledge, to programme saccadic eye movements
that jump to the vicinity of the target location or directly to its
location. 2) Serial search starts with the first menu item and
moves from one item to the next until the target is located
or the menu is exhausted. In practice a user would combine
elements of these strategies depending on the reliability of
available of knowledge. For example, directed search might
be combined with serial search in proportion to the reliabil-
ity of location memories and the available relevance scores.
In some situations location knowledge will be highly reliable
and in others it will be the relevance scores that are more re-
liable. Finally, 3) we also explore two pointing strategies:
Find-and-point and Track-and-point.
In our current model these strategies are expressed as prob-
ability distributions of gaze. A distinct advantage of this
model formulation is that it allows deriving both distributions
for gaze points on the menu as well as selection time from
the same parametrization. These predictions can be checked
against eye tracking data.
To parametrize and evaluate the model, we collected an ex-
tensive dataset. Although the constituent processes of menu
selection have been heavily studied in psychology, the menu
selection task has been hard to study for the purposes of
modeling. Multiple factors must be controlled, the apparatus
like eye-trackers are complex, and the studies must be long
enough to gauge practice effects. Our model is informed by
Bailly et al. Proc. CHI’14
nput variables: target position (1-16),
menu organization (unordered, alphabet-
umber of previous encounters with the
el is in a mathematical form that allows
nu selection time and gaze distribution.
at the total time to select an item con-
two consecutive subtasks–search and
affected by learning (Figure 1):
ruitment of eye gaze and memory recall
tion of the target in the menu by scan-
previous simulation models [20, 21] and
strategies:
consists of top-to-bottom reading of
h practice, allows more skipping. This
a uniform distribution from the begin-
nu to the target.
ch consists of a direct attempt at moving
p of the target. At first such attempts are
user tries to guess the location, but with
they become more accurate. We model
h as a normalized Gaussian distribution
e target.
in the menu P (practice).
An original aspect is to explain where (e.g., on which item)
the eyes spend time. The total time for selecting target t is
the sum of the gaze time required for each item i of the menu.
The relationship between total time and the gaze distribution
for each component is:
T(✓) =
1
R
i=lX
i=0
Gs(i, ✓) + Gd(i, ✓) + Gp(i, ✓) (2)
where Gs, Gd, Gp estimate the numbers of gaze points (se-
rial, directed, and pointing) captured using an eye-tracker
with a sampling rate of R. In our studies, R = 50Hz.
Serial search
Serial search refers to the top-to-bottom inspection of items.
Items after the target are not visited. With practice, users not
only skip more items but also spend less time on each visited
item. For this reason, the number of gaze points on each item
in a menu is not constant but decreases as a function of rep-
etitions. The number of gaze points for i when looking for
target t is:
Gs(i, ✓) =
⇢
as ⇥ exp( bs ⇥ pi) + cs if i  t
0 otherwise
(3)
ion is that the total time to select an item con-
spent in two consecutive subtasks–search and
d both are affected by learning (Figure 1):
es the recruitment of eye gaze and memory recall
the position of the target in the menu by scan-
e follow previous simulation models [20, 21] and
o search strategies:
l search consists of top-to-bottom reading of
that, with practice, allows more skipping. This
odeled as a uniform distribution from the begin-
of the menu to the target.
cted search consists of a direct attempt at moving
yes on top of the target. At first such attempts are
om, as the user tries to guess the location, but with
exposure they become more accurate. We model
ted search as a normalized Gaussian distribution
red on the target.
T(✓) =
1
R
i=lX
i=0
Gs(i, ✓) + Gd(i, ✓) + Gp(i, ✓) (2)
where Gs, Gd, Gp estimate the numbers of gaze points (se-
rial, directed, and pointing) captured using an eye-tracker
with a sampling rate of R. In our studies, R = 50Hz.
Serial search
Serial search refers to the top-to-bottom inspection of items.
Items after the target are not visited. With practice, users not
only skip more items but also spend less time on each visited
item. For this reason, the number of gaze points on each item
in a menu is not constant but decreases as a function of rep-
etitions. The number of gaze points for i when looking for
target t is:
Gs(i, ✓) =
⇢
as ⇥ exp( bs ⇥ pi) + cs if i  t
0 otherwise
(3)
where as, bs, cs are the parameters of the learning model,
while pi is the number of previous encounters with item i.
Directed search
In directed search the user tries to glance directly at the target.
Attempts are initially more random, but the spread of gaze
distribution decreases with more repetitions. As experienced
users have memory about the general location of the target,
they can glance at the neighborhood of the target and, even-
tually, directly move the eyes to it [11]. As in serial search,
we assume that experienced users also skip more items and
spend less time reading them than novices.
We thus model this distribution of gaze with a normal distri-
bution centered on the target item t. Probability for item i in
the menu is given by its probability density function t, 2
d
(i)
modulated by the power law of learning:
Gd(i, ✓) = t, 2
d
(i) ⇥ (ad ⇥ exp( bd ⇥ pi) + cd) (4)
where d is the standard deviation reflecting the spread of the
search area. ad, bd and bd are the empirical parameters of the
learning model. Note that the sum in Eq. 2 ranges from the
first item to the end of the menu (i 2 [1, l]), but not longer,
because we assume that users only search inside the menu.
and d0 the dista
the trail strategy
T
We can thus est
that users can se
point, we revisit
t,1.1(i) that cov
the target:
Gp(i, ✓) =
where t,1.1 is a
get t with a stan
95% of the gaze
Absent items
We also provide
not in the menu
pends on menu l
(p = 0), they ins
detecting missin
Ta(✓
where aa, ba, ca
are the parameters of the learning model,
mber of previous encounters with item i.
h the user tries to glance directly at the target.
ially more random, but the spread of gaze
eases with more repetitions. As experienced
ory about the general location of the target,
at the neighborhood of the target and, even-
ove the eyes to it [11]. As in serial search,
experienced users also skip more items and
eading them than novices.
his distribution of gaze with a normal distri-
n the target item t. Probability for item i in
and d0 the distance from the cursor to the gaze when using
the trail strategy. We assume that d0 = 0.
Tp = a + b ⇥ log2(1 + ↵ ⇥ t) (8)
We can thus estimate Gp. However, to account for the fact
that users can see 2-5 items in the neighborhood of the gaze
point, we revisit the equation 8. We add a normal distribution
t,1.1(i) that covers the dispersion of gaze in the items around
the target:
Gp(i, ✓) = ap + bp ⇥ log(1 + ↵ ⇥ t) ⇥ t,1.1(i) (9)
where t,1.1 is a normal distribution centered around the tar-
get t with a standard deviation equal to 1.1. This reflects that
95% of the gazes are in the users’ (extended) fovea.
EXAMPLE: MATHEMATICAL MODEL
MODEL OF MENU SELECTION

27. OPTIMIZATION METHODS
Questions we can ask
1.The best possible design
2.All designs x% from the optimum
3.Diverse but good options to a
design
4.How far are we from the
optimum
5.The single best change to
improve a design
A POWERFUL TOOLBOX
Design variable A
Variable B
Objective

28. GUARANTEE A GREATER-
THAN-ZERO CHANCE OF
FINDING THE OPTIMUM
DESIGN
OPTIMIZATION METHODS

29. KEYBOARD LAYOUT OPTIMIZATION
We can optimise
keyboard layouts for any
terminal and any given
language. We can find
the optimum design with
mathematical guarantees
Status in 2016
CASE CLOSED?
m
K! -! ,! ?! !! :! J! V! Q!
H! U! I! E! A! R! N! T!
P! B! O! .! W! L!
;!
C!
M!
space!
Z! X!
D! S!
F! G!Y!
(c) Physical 3-row standard: Pred 78.36 wpm
three keyboard optimization tasks.
hen, T., and Kan, M.-Y. Creating a live, public short message service
+7% faster than Qwerty
+2% faster than DSK
Physical keyboard layout optimized for English

31. Antti, you‘re right.
But … You’re not
dreaming BIG enough

33. can computers design?
antti oulasvirta

34. can computers design?
antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMISATION
FOUNDATION / DESIGN AS SEARCH
RESULTS / BEYOND KEYBOARDS
CONCLUSION / A FUTURE
DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER

35. can computers design?
antti oulasvirta
RESULTS / BEYOND KEYBOARDS

36. Cohen-Or 2006Itti & Koch 1998 Rosenholtz 2007
MODELS WE USE (SELECTED)
Saliency Clutter perception Color harmony
Fitts 1954
Target selection
Kieras-Hornof 2014 Salvucci 2001
Visual search Reading
J.R. Anderson 1998 Wickens 2002
Working memory Multitasking
Chen et al. 2015; Guiard 2015
Strategy adaptation
Balinsky 2006Shipley-Kellman 1992 Wertheimer 1938
Unit perception Gestalt grouping Grid quality
PERCEPTION& AESTHETICS
Lee & Oulasvirta 2016
Internal clock HUMAN
PERFORMANCE
ATTENTION & COGNITION

37. WEB PAGE DESIGN
Heading goes here
www.firstlink.com
www.secondlink.io
www.thirdlink.org
Button Button Button
Important
Original design*
*Estimated lower bound for the size of this design space is 1090

38. SINGLE-OBJECTIVE RESULTS DO NOT MAKE SENSE
BBB
Heading goes here
www.firstlink.com
www.secondlink.io
www.thirdlink.org
ButtonButtonButton
Heading goes here
www.firstlink.com
www.secondlink.io
www.thirdlink.org
Button
Button
Heading goes here
Button
www.firstlink.com
www.secondlink.io
www.thirdlink.org
Heading goes here
www.firstlink.com
www.secondlink.io
www.thirdlink.org
B B B
Button
Button
Heading goes here
Button
www.firstlink.com
www.secondlink.io
www.thirdlink.org
Selection time only Visual search only
Clutter perception only Color harmony only Grid quality only

39. WEB PAGE DESIGN
MULTI-OBJECTIVE RESULT
Button
Button
Heading goes here
Button
www.firstlink.com
www.secondlink.io
www.thirdlink.org

40. can computers design?
antti oulasvirta
LEGEND
Optimize for
Marker size
Marker color
Aspect ratio
Transparency
Correlation estimation
Cluster detection
Outlier detection
Distribution assessment
design variables
design objectives

41. APPLICATION MENU
CASE WINDOWS PHONE
Baseline
All All except first
Meanselectiontime(ms)
2000
2500
3000
3500 Optimised
Baseline
Optimised Baseline
All All except first
Meanselectiontime(ms)
2000
2500
3000
3500 Optimised
Baseline
Optimised Baseline
All All except first
Meanselectiontime(ms)
2000
2500
3000
3500 Optimised
Baseline
Optimize for
Size
Color
xy-position
Paging
Motor performance
Visual search performance
Clutter perception
Color harmony
Users (N=16) were
significantly faster with
the optimized menu.
While they rated it to be
more cluttered, they liked
the colors more.
Evaluation

42. Title
ATTENTION-OPTIMIZED SKIM READING
We determine
dynamically the content
and number of static
highlights using principles
of visual attention that
avoid overloading
Spotlights
Optimize for
Number of objects
Duration per object
Gist comprehension

43. SCATTERPLOT OPTIMIZATION
Matplotlib default design
Optimize for
Marker size
Marker color
Aspect ratio
Transparency
Correlation estimation
Cluster detection
Outlier detection
Distribution assessment
Optimized (preliminary)

44. GAME LEVEL DESIGN
USING HUMAN PERFORMANCE MODELS
Optimize for
Pillar width
Pillar gap
Timing accuracy

45. EASY MEDIUM DIFFICULT
Real gaming performance by a colleague
Optimize for
Pillar width
Pillar gap
Timing accuracy
GAME LEVEL DESIGN
USING HUMAN PERFORMANCE MODELS

46. can computers design?
antti oulasvirta
IN-AIR GESTURE DESIGN

47. MODEL ACQUISITION
TARGET SELECTION TASK
Experimental setup
formance achievable after practice. Given a target of width
and distance D, Fitts’ law states that the MT to reach the
get is given by MT = a + b log2(D/W + 1). Fitts’ law
also been used previously to quantify performance differ-
ces in fingers, wrist, and forearm [3, 5, 25, 13, 15]. How-
r, in this work we use angular motions at joints instead
translation [12]. Considering the angular target width ↵W
d distance D, we get:
mt✓ = a✓ + b✓ log2
✓
↵D
W
+ 1
◆
(1)
acquire the a and b parameters, we conduct an experi-
nt that employs a unidimensional pointing task. We ad-
ss speed–accuracy trade-off in this task by using effective
dth ¯W and distance ¯D.
Performance
disregarded in HCI. In order to compute it for every finger,
the position of the non-instructed digit is plotted as a function
of the instructed digit’s position. The resulting trajectories
are typically linear and the slope of a line fitted to these data
points serves as a measure for the relative coactivation: the
extent to which a non-instructed finger moves relative to the
instructed finger. Given the coactivation Cij of finger i during
the movement of finger j, the individuation index of j is
Ij = 1 [(
nX
i=1
| Cij | 1)/(n 1)], (2)
where n is the number of fingers (5). Ij = 1 indicates
perfectly individuated movement, and Ij = 0 if all non-
instructed fingers move simultaneously with j. The original
study of individuation was reported for fingers, but it can be
extended to multiple joints used in multi-finger input.
EXPERIMENT: FINGER DEXTERITY
Our experimental method is based on the reciprocal selec-
tion task used in Fitts’ law studies [14]. As shown in Figure
3, users move a finger between two targets. Instead of ex-
trinsic targets (e.g. button), the target here is a joint angle.
Visual feedback is provided on a monitor with high refresh
rate. In contrast to most Fitts’ law studies, we track not only
the end-points of movements but the full motion of the hand.
This allows us to quantify three aspects of the dexterity of fin-
ger motion: performance (speed and accuracy), individuation
(unwanted motion of uninstructed fingers), and comfortable
motion ranges. In addition, the data allows us to look at the
range of individual differences.
a finger between two target angles indicated on a
motion was tracked. The color coding for joints is
section. Note that the CMC joint of the thumb is a sp
be independently moved in two directions.
in a neutral pose. Moreover, we included th
thumb which was the only interphalangeal jo
moved and tracked well. Figure 3 also sho
convention and color coding used in the rest
the thumb we use Thumb-Down and Thumb
“up-down” and “left-right movement” of the
Participants
The study was conducted with 13 participant
locations. They were 8 male and 5 female,
and with age ranging from 22 to 32 (mean 2
nical difficulties, one of the participants coul
4 of the 7 joint conditions. The experiment
2 hours per participant. Participants from on
compensated with cinema vouchers. The tri
out in a room with controlled lighting and no
Experimental Design
The experiment followed a 7⇥4 within-subj
7 DOFs and 4 ID conditions. To minimize o
joint and ID conditions were randomized for
Pre-trial practice was employed and breaks w
ter the trial for each joint.
Task, Materials, and Procedure
The task is a unidimensional target selectio
pants had to move a pointer up and down betw
Finger individuation
volved in the gesture: CiG = maxj2G Cij. Then, following
the original Equation 2, we compute the individuation index
for any multi-joint gesture as
IG = 1 [(
nX
i=1
|CiG| |G|)/n |G|), (4)
where |G| denotes the number of joints actively involved in
gesture G.
Step 3: Objective Function Formulation
Our design task is to maximize the usability U of a letter
assignment, i.e. the mapping of each character in a char-
acter set to a unique posture (gesture) of the hand. To
characterize U, we formulate a multi-term objective func-
tion for mid-air text entry called PALM. PALM addresses
four factors affecting mid-air text entry with multiple fin-
gers: Performance, Anatomical comfort (individuation),
Learnability, and Mnemonics. In addition to performance
and individuation, we formalize learnability and mnemonics
based on existing literature.
Usability U is thus defined as a weighted sum of four normal-
ized (i.e. 2 [0, 1]) terms2
. Formally, we write our usability
objective as
max U = wp
ˆP + wa
ˆA + wl
ˆL + wm
ˆM, (5)
where the positive weights wp, wa, wl, and wm set by a de-
signer sum up to 1. The remaining terms in the objective
collapsing it into a few dimensions. Practicing a complex
gesture gradually increases hierarchical organization and de-
creases reliance on feedback. This has two consequences.
First, the fewer DOFs a gesture involves, the easier it will be
to learn. For instance, gesturing with one finger is easier to
learn than a gesture using three fingers. We name the number
of involved DOFs udofs. Second, if the involved digits involve
the same end posture, it will be easier to learn because the
articulations can be represented with a single learning primi-
tive. For example, it is easier to extend all digits by 40° than
to extend some by 20° and others by 40°. We denote the num-
ber of DOFs for which a target angle is defined in a gesture as
by utargets. Our learnability score combines these two aspects:
L = 1
X
k
(0.5 ˆutargets, k + 0.5 ˆudofs, k). (7)
Mnemonics Term (M)
Studies of human memory suggest that categorization, chunk-
ing, and mnenomics help forming more durable long-term
memory traces among otherwise unrelated materials [35].
Our mnemonics score M considers the memorability of a let-
ter assignment as a whole. We call a mnemonic set a set of
similar gestures, such as gestures that all have a neighboring
finger. To identify finger mnemonics, we build on a recent
study of multi-finger chord gestures that showed a positive
effect on learning [35]. We take the mnemonic principles
presented there and extend them from three fingers to five. In
particular, we include the following mnemonics rules: neigh-
Motor control literature
Learnability

48. 48
Example design: Chorded text entry
[Proc. CHI’15]

49. We can now address interfaces
where primary emphasis on
perception, motor control, and
ergonomics. Our examples span
input devices, interaction
techniques, menu systems,
graphical user interfaces, basic
information visualizations, web
pages, mobile apps, gestural
interfaces
MILESTONE IN 2016
More UI types

50. FUNCTIONALITY SELECTION

51. can computers design?
antti oulasvirta
FUNCTIONALITY SELECTION
em independent of each other; (2) link each term to easily understandable
pulable definitions supported by literature; and (3) find a formulation that is
nough in code.
ective is to choose the set of functionality that maximizes ”goodness” G. We
in terms of four objectives called USEP: usefulness U, satisfaction S, ease-
and profitability P. These terms form the core of the objective function we
ghout our work, supplementing it with case-specific weights, objectives, and
ts when needed.
maxG = !U U + !SS + !EE + !P P (4)
2 ⌦ denote a weight given to an objective,
P
!2⌦ ! = 1.0.
ollowing, we define each term. Throughout, we denote a functionality selected
n as and the full set of selected functionality . By ”normalization”, marked
zontal bar, we refer to scaling a variable to range [0, 1].
lness
e the usefulness of a design U as the sum of usefulness u and dependency
f each selected functionality:
¯U =
1
2
(
X
2
¯u + D ) (5)
efulness scores u 2 [0, 1] and
D =
P
2
P
⌧2 ,⌧
P
i
P
j i,j
(6)
pendency scores i,j2[0,1] are directional dependency scores defined by the de-
dent of each other; (2) link each term to easily understandable
itions supported by literature; and (3) find a formulation that is
e.
hoose the set of functionality that maximizes ”goodness” G. We
our objectives called USEP: usefulness U, satisfaction S, ease-
ility P. These terms form the core of the objective function we
ork, supplementing it with case-specific weights, objectives, and
ded.
maxG = !U U + !SS + !EE + !P P (4)
weight given to an objective,
P
!2⌦ ! = 1.0.
define each term. Throughout, we denote a functionality selected
e full set of selected functionality . By ”normalization”, marked
e refer to scaling a variable to range [0, 1].
ess of a design U as the sum of usefulness u and dependency
ed functionality:
¯U =
1
2
(
X
2
¯u + D ) (5)
es u 2 [0, 1] and
D =
P
2
P
⌧2 ,⌧
P
i
P
j i,j
(6)
res i,j2[0,1] are directional dependency scores defined by the de-
on Computer-Human Interaction, Vol. 1, No. 1, Article 11, Publication date: January 1111.
Usefulness
Dependency
N
ing an objective function were to (1) include commonly consid-
as separate terms with own weights, allowing the designer to
ent of each other; (2) link each term to easily understandable
itions supported by literature; and (3) find a formulation that is
.
hoose the set of functionality that maximizes ”goodness” G. We
our objectives called USEP: usefulness U, satisfaction S, ease-
lity P. These terms form the core of the objective function we
rk, supplementing it with case-specific weights, objectives, and
ed.
maxG = !U U + !SS + !EE + !P P (4)
weight given to an objective,
P
!2⌦ ! = 1.0.
define each term. Throughout, we denote a functionality selected
e full set of selected functionality . By ”normalization”, marked
e refer to scaling a variable to range [0, 1].
ess of a design U as the sum of usefulness u and dependency
d functionality:
¯U =
1
2
(
X
2
¯u + D ) (5)
s u 2 [0, 1] and
D =
P
2
P
⌧2 ,⌧
P
i
P
j i,j
(6)
res i,j2[0,1] are directional dependency scores defined by the de-
on Computer-Human Interaction, Vol. 1, No. 1, Article 11, Publication date: January 1111.
Satisfaction
Fig. 3. Users value the loss of an important function more negatively than its inclusion in a design.
4.2. Satisfaction
We define user satisfaction as a nonlinear function that maps the presence/absence of
a functionality to a satisfaction score. This score refers to subjective experience of the
unctionality, as opposed to the usefulness score. We offer two formulations.
The first definition uses a linear sum:
S =
X
2
¯s (7)
The limitation is that the absence of a functionality is not accounted for in the score.
The second definition builds on prospect theory that predicts that the absence of an
mportant functionality is appraised more critically than its presence. This is captured
by the following mathematical function for prospect ⇡ of item :
⇡i =
⇢ p
si ifxi = 1
2
p
si ifxi = 0
(8)
which we reformulate as
⇡i = 3
p
si · xi 2
p
si (9)
Note that satisfaction can be negative if a design misses functionality users expect it
o have.
This yields a new S⇡:
S =
nX
i=1
¯⇡(i) (10)
S is used in the objective function after normalization.
4.3. Ease-of-use
Ease-of-use
⇡i =
2
p
si ifxi = 0
(8)
which we reformulate as
⇡i = 3
p
si · xi 2
p
si (9)
Note that satisfaction can be negative if a design misses functionality users expect it
to have.
This yields a new S⇡:
S =
nX
i=1
¯⇡(i) (10)
S is used in the objective function after normalization.
4.3. Ease-of-use
We provide two definitions for the ease-of-use of a design E. The former is allows
quicker computation but simplifies the effect of increasing functionality. The second
is a decreasing function of the number of selected functionality.
Our first formulation is a
E =
X
62
c (11)
where c is the estimated complexity of .
ACM Transactions on Computer-Human Interaction, Vol. 1, No. 1, Article 11, Publication date: January 1111.Profitability
Fig. 4. We assume the ease-of-use of an interface to decrease non-linearly with the number of functionality.
Here, y = 1/(x 1).
Our second formulation captures the fact that ease-of-use is compromised by includ-
ing more and more functionalities. Including very few trivially ensures high ease-of-
use. For example, designers group semantically to decrease selection time in a hierar-
chical menu [Bailly et al. 2014]. Interface techniques for command selection like search
functionality, toolbars, autocompletion, panels, and tool palettes can achieve the same
effect. Ease-of-use would increase linearly only in the case the interface was organized
randomly. We also note that some interface techniques allow constant time for a small
number of features. For example, hotkeys can make access time virtually constant,
but users typically learn only few hotkeys. While we use the reciprocal function in the
following, we recognize that the exact shape of this function depends many factors,
including the skills of the design and the interface type. The reciprocal formulation is
shown in Figure 4 and given as
E =
1
| | + 1
(12)
We can also formulate this by introducing a quadratic constraint E +
P
i Exi <= 1.
4.4. Profitability
We define the profitability of a functionality p in terms of its business value v 2 R
and costs c 2 R: p = v c . After normalization of p, we can compute P:
¯P =
X
2
p =
X
2
(v c ) (13)
Note that if c > v, profitability is negative. Sometimes including features that are
negative in profitability is justifiable in the light of other objectives.
4.5. Further elaborations
Different user groups
Uncertain values
Two or more design approaches?
ACM Transactions on Computer-Human Interaction, Vol. 1, No. 1, Article 11, Publication date: January 1111.
A
Usefulness
Satisfaction
Ease of use
Profitability
24 functionalities
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_color
Find
Font_color
Font_size
Full_screen
Handwriting_recognition
Indent
Insert_blank_page
Insert_date_and_time
Insert_horizontal_line
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_continuation
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Link_to_calendar_entry
Margins
Mark_all_text
Mark_text
Merge_cells
Move_document_to_other_view
Move_note_to_notebook
New
Number_of_copies
Number_style
Numbering
Open
Open_containing_folder
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_multiple_pages_per_sheet
Print_selection
Printer_settings
Publish_online
Put_on_homescreen
Recently_opened
Record_audio
Redo
Reload
Remove_column_from_table
Remove_row_from_table
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Synchronize_scrolling
Table_border
Text_alignment_in_table
Text_background_color
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_colorFind
Font_color
Font_size
Full_screen
Handwriting_recognition
Indent
Insert_blank_page
Insert_date_and_time
Insert_horizontal_line
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_continuation
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Link_to_calendar_entry
Margins
Mark_all_text
Mark_text
Merge_cells
Move_document_to_other_view
Move_note_to_notebook
New
Number_of_copies
Number_style
Numbering
Open
Open_containing_folder
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_multiple_pages_per_sheet
Print_selection
Printer_settings
Publish_online
Recently_opened
Record_audio
Redo
Reload
Remove_column_from_table
Remove_row_from_table
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Synchronize_scrolling
Table_border
Text_alignment_in_table
Text_background_color
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
File creation and organization
Link and share
Printing
Table
Structural elements
Input
Text style options
Page layout
Controls
View
F
Usefulness
Satisfaction
Ease of use
Profitability
82 functionalities
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_color
Find
Font_color
Font_size
Full_screen
Handwriting_recognition
Indent
Insert_blank_page
Insert_date_and_time
Insert_horizontal_line
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_continuation
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Link_to_calendar_entry
Margins
Mark_all_text
Mark_text
Merge_cells
Move_document_to_other_view
Move_note_to_notebook
New
Number_of_copies
Number_style
Numbering
Open
Open_containing_folder
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_multiple_pages_per_sheet
Print_selection
Printer_settings
Publish_online
Put_on_homescreen
Recently_opened
Record_audio
Redo
Reload
Remove_column_from_table
Remove_row_from_table
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Synchronize_scrolling
Table_border
Text_alignment_in_table
Text_background_color
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_colorFind
Font_color
Font_size
Full_screen
Handwriting_recognition
Indent
Insert_blank_page
Insert_date_and_time
Insert_horizontal_line
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_continuation
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Link_to_calendar_entry
Margins
Mark_all_text
Mark_text
Merge_cells
Move_document_to_other_view
Move_note_to_notebook
New
Number_of_copies
Number_style
Numbering
Open
Open_containing_folder
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_multiple_pages_per_sheet
Print_selection
Printer_settings
Publish_online
Recently_opened
Record_audio
Redo
Reload
Remove_column_from_table
Remove_row_from_table
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Synchronize_scrolling
Table_border
Text_alignment_in_table
Text_background_color
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
File creation and organization
Link and share
Printing
Table
Structural elements
Input
Text style options
Page layout
Controls
View
…
k Diverse designs
R
Usefulness
Satisfaction
Ease of use
Profitability
106 functionalities
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete_element
Delete_file
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_color
Find
Font_color
Font_size
Full_screen
Handwriting_recognition
Indent
Insert_blank_page
Insert_date_and_time
Insert_horizontal_line
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_continuation
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Link_to_calendar_entry
Margins
Mark_all_text
Mark_text
Merge_cells
Move_document_to_other_view
Move_note_to_notebook
New
Number_of_copies
Number_style
Numbering
Open
Open_containing_folder
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_multiple_pages_per_sheet
Print_selection
Printer_settings
Publish_online
Put_on_homescreen
Recently_opened
Record_audio
Redo
Reload
Remove_column_from_table
Remove_row_from_table
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Synchronize_scrolling
Table_border
Text_alignment_in_table
Text_background_color
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Zoom
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete_element
Delete_file
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_colorFind
Font_color
Font_size
Full_screen
Handwriting_recognition
Indent
Insert_blank_page
Insert_date_and_time
Insert_horizontal_line
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_continuation
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Link_to_calendar_entry
Margins
Mark_all_text
Mark_text
Merge_cells
Move_document_to_other_view
Move_note_to_notebook
New
Number_of_copies
Number_style
Numbering
Open
Open_containing_folder
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_multiple_pages_per_sheet
Print_selection
Printer_settings
Publish_online
Recently_opened
Record_audio
Redo
Reload
Remove_column_from_table
Remove_row_from_table
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Synchronize_scrolling
Table_border
Text_alignment_in_table
Text_background_color
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Zoom
File creation and organization
Link and share
Printing
Table
Structural elements
Input
Text style options
Page layout
Controls
View
Robust design

52. EMPIRICAL EVALUATION
Professional designers found
the optimized functionality sets
as good or better than the ones
they designed themselves.
Result
R
Usefulness
Satisfaction
Ease of use
Profitability
106 functionalities
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete_element
Delete_file
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_color
Find
Font_color
Font_size
Full_screen
Handwriting_recognition
Indent
Insert_blank_page
Insert_date_and_time
Insert_horizontal_line
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_continuation
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Link_to_calendar_entry
Margins
Mark_all_text
Mark_text
Merge_cells
Move_document_to_other_view
Move_note_to_notebook
New
Number_of_copies
Number_style
Numbering
Open
Open_containing_folder
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_multiple_pages_per_sheet
Print_selection
Printer_settings
Publish_online
Put_on_homescreen
Recently_opened
Record_audio
Redo
Reload
Remove_column_from_table
Remove_row_from_table
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Synchronize_scrolling
Table_border
Text_alignment_in_table
Text_background_color
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Zoom
Add_column_to_table
Add_row_to_table
Align_text
Bullet_size
Bullet_type
Bullets
Cell_color
Cell_margins
Change_encoding
Change_number_size
Change_start_number
Character_spacing
Choose_printer
Close
Close_all_windows
Column_size
Compare_documents
Copy
Create_notebook
Create_table
Cut
Delete_element
Delete_file
Delete_notebook
Draw_freehand
Draw_shape
Exit
Export
Fill_colorFind
Font_color
Font_size
Full_screen
Handwriting_recognition
Indent
Insert_blank_page
Insert_date_and_time
Insert_horizontal_line
Insert_link
Insert_picture
Insert_video
Line_color
Line_number_color
Line_number_continuation
Line_number_font
Line_number_size
Line_numbers
Line_spacing
Line_style
Line_wrap
Link_to_calendar_entry
Margins
Mark_all_text
Mark_text
Merge_cells
Move_document_to_other_view
Move_note_to_notebook
New
Number_of_copies
Number_style
Numbering
Open
Open_containing_folder
Orientation
Page_break
Page_size
Paragraph_spacing
Paste
Print_current_page
Print_custom_range
Print_document
Print_multiple_pages_per_sheet
Print_selection
Printer_settings
Publish_online
Recently_opened
Record_audio
Redo
Reload
Remove_column_from_table
Remove_row_from_table
Rename
Rename_notebook
Replace
Row_size
Save
Save_all_windows
Search_printer
Share
Show_grid
Show_ruler
Sort_elements
Spell_check
Split_cell
Split_view
Synchronize_scrolling
Table_border
Text_alignment_in_table
Text_background_color
Text_columns
Text_font
Text_style
Type_text
Undo
Voice_recognition
Zoom
File creation and organization
Link and share
Printing
Table
Structural elements
Input
Text style options
Page layout
Controls
View
Example optimized functionality for a
note taking application

53. can computers design?
antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMIZATION
FOUNDATION / DESIGN AS SEARCH
RESULTS / BEYOND KEYBOARDS
CONCLUSION / A FUTURE
DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER

54. can computers design?
antti oulasvirta
DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER
“tools might enforce user interfaces that were highly
usable, rather than today’s stance that tools should be
neutral…” - Brad Myers et al. (2000)

55. DesignEvaluate
INITIALIZING INTERFACE OPTIMIZATION
designer
computer
scientist
social
scientist
Design Task
Optimizer
Predictive
Models

57. DesignEvaluate
DESIGN TOOLS HIDE ALGORITHMIC COMPLEXITY
designer
Optimizer
Design Task
Heuristics
Models
Interactive
optimization
tool
Defines
Steers

58. FULL CONTROL CONCEPT
Optimize for
Menu structure
Position of item
Selection time
Full interactive control
of design objectives,
input data. Computer
suggestions never
override designer
decisions, so you can
still design as normally.
MenuOptimizer
Grouping of items
Shortcuts Familiarity Novice/expert performance

59. “SKETCHPLORER”
ZERO-EFFORT INTERACTIVE OPTIMIZATION
Hands-on demo of
Sketchplorer at CHI’16
in San Jose; May 2-7

60. ZERO-EFFORT CONTROL CONCEPT
Optimize for
Item positions
Item colors
Motor performanceItem sizes
Alignment Grid quality
No need for the
designer to control
anything. The
computer recognises
the design task and
provides suggestions
accordingly.
Sketchplorer
Clutter perception
Color harmony
Visual search

61. can computers design?
antti oulasvirta
METHOD / MODEL-BASED INTERFACE OPTIMIZATION
FOUNDATION / DESIGN AS SEARCH
RESULTS / BEYOND KEYBOARDS
CONCLUSION / A FUTURE
DESIGN TOOLS / SKETCHING WITH AN OPTIMIZER

62. can computers design?
antti oulasvirta
CONCLUSION / A FUTURE

63. AS DESIGN FROM FIRST PRINCIPLES
COMPUTATIONAL INTERFACE DESIGN
Predictive models (principles of interaction) allow
algorithms to identify usable interfaces in multi-
dimensional design spaces. All inputs are scrutinizable and
empirically verifiable, thus contributing to accumulation of
design knowledge.

64. To improve the
positive effect that
optimization can have
on design quality, we
need to better
understand creativity
in human design
process.
To extend the scope of
optimizable problems,
we are working on
predictive models of
learnability, user
experience, aesthetics,
social cognition, etc.
To extend applications
in interaction design,
we must study
designers and formulate
their activities in code.
RESEARCH CHALLENGES
Modeling Design Tasks Creativity

65. Design
Superiority
An organization
emerges that
can present
guarantees for
their designs
Bulk Design
Automated
A design service
emerges that
use interactive
optimization to
significantly
improve cost-
efficiency in a
popular
category
Computer
Outperforms
Human
Computer is
shown to reliably
outperform
human designers
in controlled
studies with end-
users
Self-Designing
Interfaces
Log data is used
to parametrize
optimizers that
can optimally
adapt the
interface without
human
intervention.

66. COMPUTERS CAN design.
Funding: European Research Council Staring Grant (contract 637991); Academy of Finland
research project COMPUTED; AKA-JST research project with Kochi University of Technology;
Max Planck Institute for Informatics; Aalto University Graduate Schools; Saarland University
Thanks to my brilliant colleagues: Kumaripaba Athukorala, Myroslav Bachynskyi, Duncan
Brumby, Xiuli Chen, Gilles Bailly, Anna Feit, Giulio Jacucci, Eve Hoggan, Kasper Hornbæk,
Andrew Howes, Jussi Jokinen, Andreas Karrenbauer, Janin Koch, Per Ola Kristensson, Byungjoo
Lee, Janne Lindqvist, Perttu Lähteenlahti, Luana Micallef, Arttu Modig, Jörg Müller, Sayan
Sarcar, Rod Murray-Smith, Mathieu Nancel, Gregorio Palmas, Xiangshi Ren, Olli Savisaari,
Srinath Sridhar, Jürgen Steimle, Christian Theobalt, Jilles Vreeken, Tino Weinkauf, Daryl Weir
Links: userinterfaces.aalto.fi, computationalinteraction.org
Antti Oulasvirta
Associate Professor
userinterfaces.aalto.fi
Hands-on demo of
Sketchplorer at CHI’16
in San Jose; May 2-7