Card Sort - UXRA Education Series_2022-06-16l.pptx

Jason Telner, Ph.D., Colin Crehan, Mark Wise, Ph.D.
Card Sorting:
There is so much more to learn than you think you already know
UXPA Conference 2022
San Diego, USA
June 20, 2022

Presentation title
IBM | Project/team name
2 UXPA Card Sort | IBM CIO Design

Presentation title
A good system shortens the road to
the goal.
- Orison Swett Marden

Presentation title
Card sorting is a research method used
to help define, design, or evaluate the
information architecture of a site.

Presentation title
Web Site Navigation
Content Categorization
Menu Structure

Card Sort
Used to define the organization of
content to help design menus,
navigation, groups of similar
content, and site information
architecture.
What it is good for
Overview
A card sort is typically conducted
early early in the design process to
inform how to group content
within a page/screen, and across a
website or application.
When to use
Participants sort a set of “cards”
that represent content and group
them into categories that make
logical sense to them.
The results are collated across
participants to define categories
that should make the most sense
to the most people.
How it is conducted
Web based tools like Optimal Sort,
Userlytics, Alchemer, or UserZoom
can be used to collect data and
analyze results.
… OR Index cards, Post-it Notes,
Markers, and Rubber Bands … but
the tools make analysis easier.
Tools and materials

Memory Organization
Information categorization allows
humans to organize things, objects,
ideas that exist around them to
their understanding of the world.
The hippocampus is
the hub of the brain
system that supports
memory organization.

Organizing Information
Aristotle was the first to attempt to classify
animals in his Historia Animalium (History of
Animals)
He grouped the types of creatures according to
their similarities:
• animals with blood and animals without
blood,
• animals that live on water and animals that
live on land.
Historia
Animalium
- Aristotle
~350 BC

Information Organization
To Immanual Kant, a category is a characteristic of the
appearance of any object in general, before it has been
experienced (a priori).
Following Aristotle, Kant uses the term 'categories' to
describe the "pure concepts of the understanding,
which apply to objects of intuition in general a priori…”.
"They are concepts of an object in general, by means of
which its intuition is regarded as determined with
regard to one of the logical functions for judgments."
1781

Neuropsychological Testing
The ability to generate categories is used in
neuropsychology to test elements of cognitive
function and mental capacity.
Wisconsin Card Sorting Test (1948) presents
participants with several stimulus cards. They are
told to match the cards, but not the characteristic to
match on.
• Participants are told whether a particular match is
right or wrong.
• Measures competence in abstract reasoning, and
the ability to change problem-solving strategies
when needed.
Wisconsin Card SortingTest
Shape
Color
Quantity
No
match
1948

Magical Number Seven
George Miller (1956) investigated limitations in
human information processing and determined
that most adults can store between 5 and 9 items
in their working memory.
However, humans can remember more than nine
pieces of information, if the items are grouped
(chunked) into categories.
“My problem is that I have been persecuted by an integer… This
number assumes a variety of disguises, being sometimes a little
larger and sometimes a little smaller than usual, but never
changing so much as to be unrecognizable.”
George A. Miller (1956). “The Magical Number Seven, Plus or Minus Two:
Some Limits on Our Capacity for Processing Information”
1956

Memory Coding and Chunking of
Data
During his studies on memory recall, Cohen (1966)
observed that participants “readily detect the
categorized nature of the word lists, store some
coded representation of each category in
memory, and attempt to retrieve this coded
information during recall.”
Humans naturally categorize information into
chunks, according to common properties or
characteristics.
Train
Bus
Glider
Car
Canoe
Tank
Helicopter
Sailboat
Hot air balloon
Motorcycle
Submarine
Land vehicles
Train
Bus
Car
Tank
Motorcycle
Air vehicles
Hot air balloon
Helicopter
Glider
Plane
Water vehicles
Canoe
Sailboat
Submarine
1966

Individual Differences
A study by Losche et al. (1991) had participants
group 60 graphical images into categories.
Found a difference between categories graphic
artists and other participants.
In general, participants will not always create the
same categories, nor will they put the same items
items into the same categories.
Need to test a representative sample of end users.
1991

X
Card Sort
UXPA Card Sort | IBM CIO Design

X
Card Sort
Prepare
Execute
Analyze

X
Card Sort
Prepare
1. Pick a Method
2. Build the Card Deck
3. Refine the Card Deck
4. Write Invite + Instructions
5. Debrief Questions
6. Recruit Users
7. Prep & Test the Sort
Execute
1. User Groups Cards
2. User Labels Groups (open)
3. Debrief each User
4. (opt: Iterate & Consolidate)
Analyze
1. Review and Analyze
2. Propose the IA
3. User Validate the IA

X
Card Sort
Where to Start?
Pick a Method

Open Card Sort
Typically, the right place to start fresh – open!
User’s Groups + Labels - User will create card-
groups, and also label each group they form.
Closed Card Sort
Often used to validate an IA after an open sort
Tester’s Groups + Labels - User will place cards
inside your predetermined, pre-labeled groups.

Open Card Sort
Closed Card Sort
? ? ? ?
…

Open Card Sort
Closed Card Sort
? ? ? ?
…
X Y Z

Open Card Sort
Discover patterns and generate ideas about how to
how to organize information
Closed Card Sort
Reveals the extent of agreement. Verify and
validate categories + labels + add new items.
? ? ? ?
…
X Y Z

Open Card Sort
Closed Card Sort
START Validated
X Y Z
? ? ? ?
…

Open Card Sort
Closed Card Sort
Hybrid
Card Sort
A closed sort ... with
an option to add new groups
X Y Z ?
START Validated
X Y Z
? ? ? ?
…

Hybrid – Pros & Cons

Open Card Sort
Revolutionary
… start new / start over
Generate new Information Architecture ideas
Qualitative analysis with a quantitative boost
Closed Card Sort
Evolutionary
… validate / update an existing structure
Test an existing Information Architecture
? ? ? ?
…
X Y Z

Open Card Sort
Revolutionary
Closed Card Sort
Evolutionary
? ? ? ?
…
X Y Z

Open Card Sort
Revolutionary
Tree Test
Evolutionary
Quantitative
… sometimes called a ”Reverse Card Sort”
? ? ? ?
… X
Y
Z
Tree

Tree Test

X
Card Sort
Daisy Chain
Combine Methods

X
Y
Z
Tree
? ? ? ?
…
Open
X Y Z
Closed
Mix & Match Methods to get what you need

X
Y
Z
Tree
? ? ? ?
…
Open
X Y Z
Closed
OR
New Site / IA
Validated
Validated
Start
Fresh

X
Card Sort
? ? ? ?
…
Open
X
D
E
Tree
Q
Y
M
Tree
P
E
Z
Tree
New Site / IA
Start
Fresh

X
Card Sort
X
Y
Z
Tree
? ? ? ?
…
Open
X
D
E
Tree
Q
Y
M
Tree
P
E
Z
Tree
New Site / IA
Validated
Start
Fresh

X
Card Sort
X
Y
Z
Tree
Q
Y
Z
Tree
Existing Site / IA
Current
Site
Validated

X
Card Sort
X
Y
Z
Tree
? ? ? ?
…
Open
W
D
E
Tree
Q
Y
Z
Tree
Existing Site / IA
Current
Site
START
OVER
Validated

X
Card Sort
X
Y
Z
Tree
? ? ? ?
…
Open
W
D
E
Tree
Q
Y
Z
Tree
Existing Site / IA
X Y Z
Closed
Current
Site
START
OVER
Validated
Validated

Build AND Refine
The Card Deck

Card Sort
How many cards?

How many cards?
20 30 50
100

How many cards?
20 30 50
100
IA Value

How many cards?
20 30 50
100
Less than 20
cards is not
typically
insightful
IA Value

How many cards?
20 30 50
100
Less than 20
cards is not
typically
insightful
30 to 50 cards is often
the sweet-spot … a lot
of data, but not too
fatiguing
IA Value

How many cards?
20 30 50
100
Less than 20
cards is not
typically
insightful
fatiguing
IA Value
More than 50
cards can be
risky. Tired
users don’t give
quality insights.

How many cards?
20 30 50
100
Less than 20
cards is not
typically
insightful
fatiguing
100 cards is
doable … but only
in limited
circumstances that
reduce user
fatigue
IA Value
More than 50
cards can be
risky. Tired
users don’t give
quality insights.

How many cards?
20 30 50
100
Less than 20
cards is not
typically
insightful
fatiguing
100 cards is
doable … but
only in limited
circumstances
that reduce user
fatigue
IA Value
More than 50
cards can be
risky. Tired
users don’t give
quality insights.
Don’t go
over 100
cards

How many cards?
20 30 50
100
Less than 20
cards is not
typically
insightful
fatiguing
100 cards is
doable … but
only in limited
circumstances
that reduce user
fatigue
Don’t go
over 100
cards
More than 50
cards can be
risky. Tired
users don’t give
quality insights.
Bounded by User Fatigue

How many cards?
20 30 50
100
Less than 20
cards is not
typically
insightful
fatiguing
100 cards is
doable … but
only in limited
circumstances
that reduce user
fatigue
Don’t go
over 100
cards
More than 50
cards can be
risky. Tired
users don’t give
quality insights.

How many cards?
20 30 50
100
Less than 20
cards is not
typically
insightful
fatiguing
100 cards is
doable … but
only in limited
circumstances
that reduce user
fatigue
IA Value
More than 50
cards can be
risky. Tired
users don’t give
quality insights.
Don’t go
over 100
cards

How many cards?
20 30 50
100
Less than 20
cards is not
typically
insightful
fatiguing
100 cards is
doable … but
only in limited
circumstances
that reduce user
fatigue
Don’t go
over 100
cards
More than 50
cards can be
risky. Tired
users don’t give
quality insights.
IA Value

Find a natural split
Too Many Cards? … Last Resort: Divide into 2 Card Sorts

More Fatigue … Less Cards
• Sentences take longer to read & digest
• Complex & Unfamiliar concepts add fatigue
• Time: over 30 minutes will tax attention
• If Unmoderated, users will trail off sooner
• Boring & Uninteresting topics will lead to users
abandoning the sort early.
• 12+ Categories expected
• Tool Complexity yeah … usability matters
Less Fatigue … More Cards
• 1-3 Word Titles: short & clear (use hover help for
examples and clarifying descriptions)
• Easily Understood: simple or an area of the users’
expertise.
• Time: closer to 15 minutes is a good length
• If Moderated, users will try give it extra effort
• Interesting topics will keep users’ attention
• 7 +/- 2 Categories expected
Too Many Cards? … Much better option: Reduce the Fatigue

Test Users’ Mental Model

Intro / Briefing the user

X
Card Sort
Adobe
Captivate
Adobe Captivate is an authoring
tool for creating eLearning
content with interactive
elements, which you can then
publish to desktops and mobile
devices.

X
Card Sort
Adobe
Captivate
devices.
The Card Title is the most
important element
… and the only
mandatory piece

X
Card Sort
Adobe
Captivate
devices.
An Optional Description
can help clarify, and help
keep the Title short
The Card Title is the
most important element
… and the only
mandatory piece

X
Card Sort
Adobe
Captivate
devices.
An Optional Description
can help clarify, and help
keep the Title short
An Optional Icon
or Pic looks cool
… but can be
dangerous,
distracting users
to sort by visual
style - Be careful
The Card Title is the
most important element
… and the only
mandatory piece

Card Labels & Terms
Can we do it all? (in my experience, not usually)
If you can test every item in a catalog or every navigation
item on your site … with its actual wording – great!
… more often, you will need to make choices:
• Wording choices
• Conceptual choices
• Representation choices
Take your time to do this right … it will greatly affect
everything that follows
Kitchen sink
too?

Card Labels & Terms … What’s wrong with this deck?
1. Gala Apples
2. Oranges
3. Broccoli
4. Lightlife
5. Produce
6. Lasagnea
7. Bacon
8. Cherimoya
9. Grab’n’Go Sandwiches
10.Ginger Ale
11.Bread
12.Apple Pies
13.Cereal
14.Kellogg’s Cereal Bars,
Assorted Family Pack
15.Canned Soup
16.Salad Bar
17.Beans
18.Breakfast
19.Kellogg's Frosted Flakes
20.Vegetarian Prepared Meals
21.Apple, Honeydew
22.Cakes
23.Grab’n’Go Hot Lunch
24.Popcorn
25.Generic Brands
26.Fruit Filling
27.Orange Fanta
28.Vegetables (frozen)
29.Paper Products (e.g., Toilet
Paper, Paper Towels,
Napkins, etc.)
30.Fresh Salsa
31.Dole Bananas
32.Canned Ravioli
33.Oatmeal
34.On Sale
35.Stuff to serve for a large
group of guests at a dinner
party
36.Pastries
37.Pepsi
38.Fruit Rollups
39.Lipton’s Instant Soup
40.Grab’n’Go Ravioli
41.Bagels
42.Chef Boy’R’Dee
43.Plant-based Bacon
44.Flowers
45.Dole Pineapple Chunks
46.Ramen Noodles
47.Bird’s Eye Stir Fry Mix
48.Pie Crusts
49.Tomatoes
50.Dole Fruit-in-a-Cup

Card Labels & Terms … get the Concepts & Wording done right
Concepts
1. Representative sampling – a few items from groups that
you expect, with at least one prototypical example
2. Test the edges – extra cards to represent edge cases that
are non-obvious
3. Right-sized … each card’s topic should not be too broad
OR overly focused
4. Bias - Beware word recognition … cards that all start with
the same word may lead to thoughtless grouping
5. Loners - No need to test a single card that is profoundly
different from all others
6. Absence - Look again for what is missing.
Wording
1. Long sentences & lists on
one card
2. Rare, unknown terms
3. Brand Names, unless they
help describe
4. Awkward language &
misspellings
5. Key attributes must ALL be
conveyed … then let the user
decide
6. Descriptions, use as needed

Practice updating cards

Don’t Forget
Randomize
the card order

Manual
Easy, tactile, natural – but extra work to analyze
Automated
Usability of the tool can impact fatigue
vs

Manual Cards … capture the decks with a picture AND rubber bands

Moderated
In Person or Remote
Pro 1 - Observed: You can ask users to speak
aloud, and get a deeper understanding of WHY
sort … and if they misunderstand a label
Pro 2 - Interactive: You can revisit and debrief in
person … and go deeper
Pro 3 - Time: Users will spend more time sorting
Con 1 - Synchronous: 1:1 meetings mean this will
take more time
Unmoderated
Remote
Pro 1 - Asynchronous: LOTS more users can
perform the sort and give you a lot of data, fast
Con 1 - Insight: Harder to observe the users’
reasoning, or to follow-up with additional
debriefing questions
Con 2 - Time: Users will abandon the sort
sooner
vs

Post Sort - Revisit
Double Check
As you sort, the categories become clearer …
moment to check your work:
1 – Any cards you want to move?
2 – Any categories seem too large or small?
3 – Are the category names good?
Post Sort - Debrief
Ask for extra insight
As you sort, the categories become clearer
1 – Which items were especially easy to sort?
2 – Which items were difficult to sort?
3 – Which items seemed to belong in two or
more groups?
4 – Why were some items left unsorted?

How it’s organized … How it’s labeled

X
Taxonomies
Folksonomies

Tips

Common Groupings + Common Pairs

• X
X
Presentation title

X
Card Sort
Analyzing the Data

Results
Frequency table
• Summarizes the frequency with with each item was
associated with a given category
• Alchemer only presents results in tabular format
Dendrogram
• Visualization of the clustering of similarly organized
cards
• Can adjust minimum strength of the relationships
between items to create more/fewer groupings
• UserZoom displays results using a dendrogram and
other results (table)

X
Card Sort

Reading Dendrograms … Goldilocks
< too many categories … Fractured
All in one … too few categories >

X
Card Sort
https://app.optimalworkshop.com/optimalsort/bananacom/
bananacom-demo-results-open/shared-results

X
Card Sort

App Categories … Sort 50 Representative Tools for Employees
App Name
Red Hat
CIO Hybrid Cloud
FileZilla FTP
Box@IBM
Slack
Mural@IBM
Box Notes
Adobe Photoshop CC
InVision Enterprise
Carbon Design System
GitHub Enterprise
New Relic
Eclipse
Devices@IBM
eAMT Lite Asset Management
Capitol Request Tool (WWCT)
Orca
Jira@IBM
Microsoft PowerPoint
w3 Publisher
Buy@IBM (BonD)
Adobe Acrobat Reader
AccessHub
7-Zip
App Name
Your Career & Learning
Lighthouse
IBM Leadership Academy
IBM Cognos Analysis for Microsoft Excel (CAFE)
BI@IBM Portal
Microsoft Project Standard Edition
1Password@IBM
Cisco Anyconnect Secure Mobility Client
Code42 CrashPlan
IBM Cybersecurity
Update Password
Checkpoint
BCG: Business Conduct Guidelines
Concerns & Appeals
IBM Recognition Center
Jobs: Global Opportunity Marketplace
Professional Marketplace
Travel@IBM: Travel & Expenses
Workday@IBM
AskHR
CV Wizard
IBM Cogni-Pay
Manager Essentials
SalesTools@IBM
BlueMine
Client Success Central (CSC)
Old Category
Cloud
Cloud
Cloud
Collaboration
Collaboration
Collaboration
Collaboration
Design
Design
Design
Development
Development
Development
Devices
Devices
Devices
Devices
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Old Category
Career & Learning
Career & Learning
Career & Learning
Data & Analytics
Data & Analytics
Data & Analytics
Help & Security
Help & Security
Help & Security
Help & Security
Help & Security
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
Management
Management
Sales
Sales
Sales

App Categories - Card Sort Dendrogram 81 users

Client Sales & Marketing
Data & Analytics
IT Performance
Cloud Platform
File Storage & Sharing
Collaboration & Teaming
Role: Developer
Accessibility
Role: Design
HR: Management
HR: Employee
HR: Career & Learning
Security, Access & Backup
Assets & Devices
Purchasing & Registration
App Categories - Card Sort Dendrogram 81 users

App Categories
New Categories
Accessibility
Assets & Devices
Cloud Platform
Data & Analytics
HR: Employee
HR: Management
IT Performance
Productivity
Role: Design
Role: Developer
16 potential
w3U-Apps
Categories

App Categories
New Categories
Accessibility
IT Performance
New
New
New
New
Not a strong category … but I think
this will be important for the PwD
audience
Very strong category … helps
clean up awkward prior categorizations
Very weak category … consider
merging with Cloud Platform and/or
Role: Developer
Good category … potentially could
merge with Assets & Devices, but
would be hard to name

Apps: Category Names
New Categories
Assets & Devices
Cloud Platform
Data & Analytics
HR: Employee
HR: Management
Productivity
Role: Design
Role: Developer

Apps: Category Names
New Categories
Assets & Devices
Cloud Platform
Data & Analytics
HR: Employee
HR: Management
Productivity
Role: Design
Role: Developer
Do these extra words help make
categories clearer?
… if not, drop them
I think these cat/subcategory
markers will help users scan the
list (especially if in alpha-order)
We felt we needed to group these
3 items together. Unsure is “Access”
naturally falls under “Security”, etc.

Apps: Category Order
Note 2 – there are enough categories
that I would consider keeping them
in alphabetical order for scanability
(order of use can be disorienting).

Apps that were difficult to sort
Note 2 – even with our descriptions,
not all users will know the Apps well enough
to make an expert categorization.

w3U-Apps that were difficult to sort >15%

Apps … Do we agree with user re-Categorizations?
App Name
Red Hat
CIO Hybrid Cloud
FileZilla FTP
Box@IBM
Slack
Mural@IBM
Box Notes
Adobe Photoshop CC
InVision Enterprise
GitHub Enterprise
New Relic
Eclipse
Devices@IBM
eAMT Lite Asset Management
Capitol Request Tool (WWCT)
Orca
Jira@IBM
w3 Publisher
Buy@IBM (BonD)
AccessHub
7-Zip
App Name
Your Career & Learning
Lighthouse
IBM Leadership Academy
IBM Cognos Analysis for Microsoft Excel (CAFE)
BI@IBM Portal
Microsoft Project Standard Edition
1Password@IBM
Cisco Anyconnect Secure Mobility Client
Code42 CrashPlan
IBM Cybersecurity
Update Password
Checkpoint
BCG: Business Conduct Guidelines
Concerns & Appeals
IBM Recognition Center
Jobs: Global Opportunity Marketplace
Professional Marketplace
Travel@IBM: Travel & Expenses
Workday@IBM
AskHR
CV Wizard
IBM Cogni-Pay
Manager Essentials
SalesTools@IBM
BlueMine
Client Success Central (CSC)
Old Category
Cloud
Cloud
Cloud
Collaboration
Collaboration
Collaboration
Collaboration
Design
Design
Design
Development
Development
Development
Devices
Devices
Devices
Devices
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Productivity
Old Category
Career & Learning
Career & Learning
Career & Learning
Data & Analytics
Data & Analytics
Data & Analytics
Help & Security
Help & Security
Help & Security
Help & Security
Help & Security
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
HR & Employee
Management
Management
Sales
Sales
Sales
New Category
Cloud Platform
Cloud Platform
Productivity
Role: Design
Role: Design
Role: Developer
IT Performance
Role: Developer
Assets & Devices
Assets & Devices
Accessibility
Role: Developer
Productivity
Productivity
New Category
Data Analytics
Data Analytics
Productivity
HR: Employee
HR: Employee
HR: Employee
HR: Employee
HR: Employee
HR: Employee
HR: Employee
HR: Management
HR: Management

w3U-Apps in multiple Categories
Note 3 – some Apps can (and SHOULD)
be put in more than one category

w3U-Apps in multiple Categories
Need to crawl through the data & comments to make recommendations
>15%

Don’t be afraid to modify research
activities to better answer your specific
research questions
On Demand Workplace example
• Designing the first user configurable w3 homepage
• There were over two dozen “porlets” that contained
• Research question: “What is the best default layout of
portlets that would satisfy most users’ needs?”
Modified card sort
• Created a blank homepage canvas
• Created “cards” for each portlet
• Participants arranged according to their preference
• The average position of each portlet was calculated to
decide its default location on the page

Card Sort
Reverse card sorting
• Validating an existing structure of categories and sub-
categories
• Users are given tasks (e.g., locate the Mango card) and
asked to complete them navigating a collection of
• Evaluates the categories and structure in isolation,
mitigating the effects of navigational aids, visual design,
other factors.
Modified-Delphi card sorting
• the first participant does a full card sort of organizing
arranging items
• The next participant iterates on the first participant's
then the third participant iterates on the second's
and so on.
• The idea is that with each iteration the card sort gets
refined with fewer participants and consensus is built
sooner.

Survey platform When to use it? Setup of survey Number of
categories
Layout Analysis, reporting and
data
Optimal
Sort
All types of surveys. Usability tests
and unmoderated).
Card sort feature is more advanced and
automated.
Great for closed sorts with large number of
items or open sorts if expect less than 12
categories.
Easier to use
Little stats knowledge needed and no SPSS
as creates dendrograms
Both open and closed
sorts.
Easy to setup.
Automatic fields for
descriptions and
Automatic post-sort
questions (difficulty &
more than 1 category).
Unlimited for both
open and closed
sorts.
Cannot specify number
of columns in sort task.
No layout limitations,
page scrolls up nicely
items remain in view for
participant.
Dendrograms,
participant summary,
card and category
frequency summaries,
similarity grid (% of
respondents classified
item under each
category).
Can export to Excel and
SPSS.
User
Zoom
and unmoderated).
Card sort feature is more advanced and
automated.
Great for closed sorts with large number of
items or open sorts if expect less than 12
categories.
Easier to use
Little stats knowledge needed and no SPSS
as creates dendrograms
sorts.
Easy to setup.
Automatic fields for
descriptions and
Automatic post-sort
questions (difficulty &
more than 1 category).
Max of 12 for
open sort.
No limit for closed
sort.
participant.
Dendrograms,
participant summary,
card and category
frequency summaries,
similarity grid (% of
respondents classified
item under each
category).
Can export to Excel and
SPSS.

Survey platform When to use it? Setup of survey Number of
categories
Layout Analysis, reporting and
data
Userlytics
and unmoderated).
Useful for open and closed card sorts
Records participants and screen as video
they do the task
sorts.
-Easy to setup
-Does not appear to
support image uploads
-Must manually enter
item descriptions.
Must create post-sort
questions yourself
Unlimited for both
open and closed
sorts.
participant.
-Frequency table of
items to categories
-Category report that
explains best option for
each item, tells items
did not quite fit but
correlation of where
could go (warning
for items did not fit
anywhere with low
correlations).
-AI algorithm
recommends where
item could fit best
Alchemer
All types of surveys, however card sort
more rudimentary and requires more
manual entries.
Useful for open sort if expect > 12 categories
however display issues.
Good knowledge of stats and data (cluster
analysis).
sorts
Can customize font
and styles.
Must manually enter
descriptions.
Must create post-sort
questions yourself
(difficulty & more than
category).
Unlimited for both
open and closed
sorts.
Can customize number
columns in sort task.
Layout limitations when
sorting many items i.e. >
30, participant must
down the page due to
layout limits and items
get out of view for
participant.
Frequency table only
no dendrograms.
Can export to Excel
SPSS.
+

X
Card Sort
https://app.optimalworkshop.com/optimalsort/bananacom/
bananacom-demo-results-open/shared-results

X
Card Sort

Optimal Sort card sort exercise –setup
Card Sort
UXPA 2022, San Diego
Using Optimal Sort create an open sort survey for the following 30 items to determine the best
categories for the CNN.com website.
• Latest US news
• Crime and Justice
• Extreme weather
• Africa
• US election
• Biden Presidency
• Media
• Fitness
• Markets
• Tech
• Social Commentary
• Sleep
• Food
• Relationships
• Screen
• Stars
• Culture
• Space and Science
• Investing
• Retirement
• Work transformed
• Success
• Covid
• Real estate
• Innovative cities
• Energy
• Money
• Cars
• Homes
• Risk takers

Optimal Sort Card sort exercise –Dendrogram for items
Card Sort
Agreement
100% 0%
Latest US news
US Election
Biden Presidency
Crime and Justice
Extreme weather
Covid
Sleep
Food
Relationships
Fitness
Work transformed
Success
Risk takers
Retirement
Money
Markets
Investing
Real estate
Homes
Cars
Tech
Space and Science
Energy
Innovative cities
Social Commentary
Culture
Africa
Media
Screen
Stars

Optimal Sort Card sort exercise –Dendrogram for categories
Card Sort
Agreement
100% 0%
US news
US politics
National News
Local news
Culture
International News
Environment
Earth
Health and Wellness
Lifestyle
Life
Career
Jobs
Personal opinion
Consumer
Home products
Science and Tech
Innovation
Technology
Finance
Money
Investing
Entertainment
TV and Film

Alchemer Open Card Sort Setup
Select open card
sort from
question type.
Enter question,
items to sort and
their font size
and style.
Enter items to
sort and their
font size and
style.
Card Sort

Can customize
layout- number
of columns and
ordering.
Card Sort

Can ask
participants about
difficulty and items
that were difficult
to sort.
Can ask
participants about
items they wanted
to sort into more
than one category.
Card Sort

Alchemer Open Card Sort Preview
Welcome
page
Card Sort

Participant asked to
sort the items into as
many categories as
they want.
Recommended is 30-
50 items per open
sort.
Card Sort

Participant drags
items from the left
side to create a new
category then can
type in the name.
Participant drag
additional items that
fit that category
below the other
items.
Pros: participant
can create as
many categories
as they want.
Cons: Participant
must scroll down
the page due to
layout limits and
items can get out
of view.
Card Sort

Pros: participant
can create as
many categories
as they want.
Cons: Participant
might have to scroll
down the page due
to layout limits and
items can get out
of view.
Card Sort

Alchemer Card Sort Analysis of Results and Dendrograms
Frequency table from
open card sort, must
combine categories that
participants named into
meaningful category
names, this involves
domain domain and
looking at the table.
Alchemer only
provides a frequency
table of items by
category and no
Dendrograms. Must
create Dendrograms
yourself using
hierarchical cluster
analysis.
Card Sort

Copy frequency
data in SPSS and
run hierarchical
cluster analysis to
create Dendrogram
plot.
Card Sort

Category names are
labels and sorted
items are variables.
Select statistics,
plots, method for
further selections.
Card Sort

Statistics Plots Methods
Card Sort

This shows the
distance of the items,
the closer the
distance the stronger
the association or
grouping.
Collaboration
& teaming
Role:
Design Assets &
Devices
Cloud
platform
File
storing &
sharing
Domain
knowledge helps
to know that
Orca shown on
its own can pair
with Github as
developer also
shown on its
own.
Card Sort

Proximity matrix
also shows
distances between
items that cluster
together and can
help tweak clusters.
Proximity Matrix
Case
Matrix File Input
Red Hat
CIO Hybrid
Cloud
FileZilla FTP -
FTP file
transfer client Box@IBM Slack
Red Hat
CIO Hybrid Cloud
FileZilla FTP - FTP file
transfer client
Box@IBM
Slack
Mural@IBM
Box Notes
Adobe Photoshop CC
InVision Enterprise
GitHub Enterprise
New Relic
Eclipse
Devices@IBM
eAMT Lite Asset
Management
Capitol Request Tool
(WWCT)
Orca
Jira@IBM
w3 Publisher
Buy@IBM (BonD)
AccessHub
7-Zip
Microsoft Project
Standard Edition
.000 18.000 33.000 66.000 64.000 62.000
18.000 .000 33.000 60.000 60.000 56.000
33.000 33.000 .000 33.000 33.000 33.000
66.000 60.000 33.000 .000 20.000 18.000
64.000 60.000 33.000 20.000 .000 18.000
62.000 56.000 33.000 18.000 18.000 .000
60.000 60.000 33.000 20.000 16.000 22.000
56.000 62.000 33.000 58.000 54.000 60.000
46.000 46.000 33.000 54.000 56.000 52.000
44.000 50.000 33.000 64.000 64.000 62.000
50.000 48.000 33.000 48.000 44.000 46.000
36.000 32.000 33.000 62.000 64.000 60.000
38.000 38.000 33.000 60.000 58.000 58.000
62.000 54.000 33.000 56.000 58.000 60.000
60.000 52.000 33.000 60.000 56.000 62.000
56.000 52.000 33.000 60.000 62.000 62.000
50.000 50.000 33.000 56.000 58.000 52.000
56.000 54.000 33.000 42.000 38.000 34.000
58.000 64.000 33.000 50.000 50.000 52.000
62.000 48.000 33.000 36.000 32.000 38.000
58.000 54.000 33.000 56.000 58.000 60.000
58.000 60.000 33.000 54.000 54.000 54.000
50.000 40.000 33.000 56.000 58.000 58.000
54.000 58.000 33.000 58.000 60.000 62.000
56.000 62.000 33.000 58.000 58.000 58.000
Card Sort

Alchemer Open Card Sort Set 1 data –Dendrogram open categories

Can manually create
question asking about
items that were difficult
to sort –gives insight
into renaming
items/categories
Verbatims help
understand the reason
an item was difficult to
sort
Card Sort

Verbatims help
understand the reason
an item could be placed
into more than one
category
Can manually create
question asking about
items that could be
placed into more than
one category–gives
insight into renaming
items/categories
Card Sort

Alchemer Closed Card Sort Setup
Enter question
Enter Category
names
Enter items
Card Sort

Participant
drags items
from left into
predefined
categories.
Card Sort

Drawback is the
layout and scrolling
down the page that
may be required for
the participant if large
number of items
placed into single
category.
Card Sort

Alchemer only
provides a frequency
table of items by
category and no
Dendrograms. Must
create Dendrograms
yourself using
hierarchical cluster
analysis.
Card Sort

Alchemer card sort exercise –setup
Card Sort
Using Alchemer create an open sort survey for the following 30 items to determine the best
categories for the Wells Fargo banking website.
• Checking
• Credit cards
• Investing
• Savings
• Business Resources
• Payroll Services
• Wealth Management
• Commercial Capitol
• Middle Market Banking
• Commercial Real Estate
• Global Treasury Management
• Global Services
• Auto Loans
• Savings and CDs
• Education and Tools
• Home Loans
• Merchant Services
• Lines and Loans
• Insights and Reports
• Insights and Economics
• Everyday checking
• Prime Checking
• Refinance your mortgage
• Online trading
• Money management
• Way2Save Savings
• Equities
• Asset backed finance
• Business line of credit
• Payment processing options

Alchemer Card sort exercise –Dendrogram for items
Card Sort
Agreement
100% 0%
Checking
Everyday Checking
Prime checking
Savings
Savings and CDs
Way2Save Savings
Credit cards
Auto loans
Home loans
Refinance mortgage
Education and tools
Investing
Online trading
Wealth management
Merchant Services
Lines and Loans
Payroll Services
Business Resources
Business Line of Credit
Payment processing options
Commercial Capitol
Middle Market Banking
Equities
Asset Backed Finance
Commercial Real Estate
Global treasury Management
Global Services
Insights and Reports
Insights and Economics

Alchemer Card sort exercise –Dendrogram for categories
Card Sort
Agreement
100% 0%
Personal
Personal banking
Everyday banking
Daily banking
My banking
Investing
Wealth management
Trading
Small business
Retailers
Businesses
Large business banking
Large business finance
Commercial banking
Commercial finance
Non personal banking
Corporate banking
Investment banking
Corporate finance
Real estate
Global services
Global banking
Insights and reports
Economic reports and insights

Userlytics Open Card Sort Setup
User must
select
create new
study
User must
select
unmoderated
study
User must give name
to study, a starting
webpage, max session
time, and study
language
User must select
participant device,
recording method,
participant source, number
of participants, other
recruiting demographics
Card Sort

User must
select card
sort.
User must
enter in
card names
in card tab.
User must
select open
sort.
User can
preview their
study in
preview
mode.
User can
randomize
the cards
User can edit
instructional
text
Card Sort

To test the study in
the participant mode,
user must install a
browser extension
(for the video
recording and web
link).
User can
test the card
sort in
participant
mode.
Card Sort

Participant is
first prompted
that screener
questions may
be presented
Participant is
shown the
webpage or
application that
is being tested
from the link
provided
Participant must
click get started
link to begin card
sort and start
recording.
Participant
must click the
box to create
new category
Participant must type
in category name then
click add and drag
items from left side
into the category
Card Sort

Userlytics Closed Card Sort Setup
User must
input card
names in card
tab.
User can
preview
the card
sort.
User can
preview the
card sort
Card Sort

User must
select card
sort
User must
input category
names in
category tab.
User can
add new
category
User can
randomize
category
names
Card Sort

User can test
the card sort
in participant
mode.
Participant is
shown the
webpage or
application that
is being tested
from the link
provided
Participant must
click get started
link to begin card
sort and start
recording.
Participant drag items
from left side into the
appropriate category.
Card Sort

Userlytics Analysis of Results
Card Sort
Find the metrics
option from the
Userlytics
dashboard
Click on the card
sorting task within
your test to view the
detailed metrics from
participant’s results

Card Sort
Card report shows
how participants
sorted each card or
item, gives category
% item falls into –
this item is clearly
better in the ”drinks
"cards category.
Three types of
analysis of
participant results –
Cards, Categories
and
Recommendations

Card Sort
Category gives
information about
any categories
participants created
and the % of cards
listed under each
category.
Lets you know what cards
were placed in each
category by the majority of
participants and which cards
were placed in each
category at least once.

Card Sort
Clicking on any of these
metrics will expand and give
greater detail on the % of
participants that placed
specific cards in each
category

Card Sort
“View as Table” shows the number of
participants that placed each card
within a given category. Red color
indicates all participants placed card in
same category, colors appear lighter as
fewer participants place cards within
specific category.

Card Sort
Recommendations gives
recommendations on
categories and cards that
should be considered
removed or revisited.
Algorithm creates warnings
for specific cards and
categories that may need to
be removed or renamed as
have minimal data to
support them.

Userlytics card sort exercise –setup
Card Sort
Using Userlytics create an open sort survey for the following 30 items to determine the best
categories for the eBay website.
• Women’s winter gloves
• Men’s ties
• Refurbished Macbook Pro
• Speed boat
• iPhone 7
• Fitbit
• Cabbage patch kids doll
• Superman comic book
• Cookware
• Sun tan lotion
• GPS device
• Caliper
• Patio furniture
• Snowmobile parts
• Drone
• Men’s belt
• Motorcycle
• HDMI cord
• Body fat scale
• Star Wars Han Solo figurine
• Blender
• Women’s handbag
• Skin care for women
• Vitamins
• Electric toothbrush
• Hedge trimmer
• Scrabble board game
• Dyson vacuum cleaner
• Chain saw
• Men’s wallet

Women’s winter gloves
Men’s ties
Men’s belt
Women’s handbag
Men’s wallet
Skin care for women
Sun tan lotion
Vitamins
Patio furniture
Hedge trimmer
Snowmobile parts
Chain saw
Motorcycle
Speed boat
Dyson vacuum cleaner
Electric toothbrush
Blender
Cookware
Refurbished Macbook Pro
iPhone 7
HDMI cord
Fitbit
GPS device
Caliper
Body fat scale
Cabbage patch kid’s doll
Star Wars Han Solo figurine
Scrabble board game
Superman comic book
Drone
Userlytics Card sort exercise –Dendrogram for items
Card Sort
Agreement
100% 0%

Userlytics Card sort exercise –Dendrogram for categories
Card Sort
Agreement
100% 0%
Clothes
Men’s clothes
Women’s clothes
Accessories
Men’s accessories
Women’s accessories
Health and beauty
Lotion
Outdoor furniture
Home products
Kitchen products
Motorized vehicles
Transport
Outdoor and leisure
Equipment
Home equipment
Fitness
Health and fitness
Gadgets
Mobile devices
Technology
Computers and accessories
Electronic products
Games
Board games
Children’s toys
Toys and games

User Zoom Open Card Sort Setup
Set up is
straight forward
and wizard like.
Choose
create new
project.
Choose
unmoderated.
Choose
card sort.
Choose
both
desktop &
tablet.
Card Sort

Name project
and provide
description.
Card Sort

Can invite own
participants,
specify number
and get survey
link.
Card Sort

Enter in welcome
page, review
legal consent,
and enter in any
other survey
questions such as
demographics
etc.
Enter in task
description
Choose
range of
categories,
max is 12.
Select ask for
category
description and
names.
Best to order
randomly
Choose open
categories
Card Sort

Insert each item on
separate line, click
create.
Can add in
description and
image easily.
Card Sort

User Zoom Open Card Sort Preview
Welcome page,
privacy consent
and
demographic
questions.
Card Sort

Instructions are
very detailed
and clear.
Card Sort

Drag items from
left to grey boxes
to create category
or place into a
category.
Hovering cursor
over the “i” gives
item description.
Type inside
category name field
to give category a
name.
Card Sort

User Zoom Open Card Sort Set 1 data –Dendrograms from 33 respondents and 168
cases for items
File
sharing/storage
Productivity
software
Cloud software
Design software
Asset
management/purchase
Collaboration/project
management
Card Sort

User Zoom Open Card Sort Set 1 data –Dendrogram open categories

User Zoom Open Card Sort Set 1 data -Difficulty
Card Sort

User Zoom Open Card Sort Set 1 data –More than one category
Card Sort

User Zoom Closed Card Sort Setup
Enter in welcome
page, review
legal consent,
and enter in any
other survey
questions such as
demographics
etc.
Select closed
categories
Best to order
randomly
First start by
adding in
category
names.
Card Sort

Add category
names
Add item
names
Card Sort

Add
descriptions of
categories.
Add
descriptions of
items.
Add images for
items.

User Zoom Closed Card Sort Preview
Task
instructions are
very clear.
Card Sort

Drag items
from left into
categories on
right.
Scrolling items
prevents items
from going out
of view and can
accommodate a
lot of items.
Hover over “i”
to get more
description of
item.
Card Sort

Post-task
questions are
automatically
generated.
Card Sort

The researcher can adjust
the threshold of
agreement between
items to create clusters
depending on how many
they want.
User Zoom Card Sort Analysis of Results and Dendrograms
File storing
and sharing
Cloud
This scale shows the
% of agreement
between the items,
the higher % the less
they form the same
cluster.
Client sales
and
marketing
Collaboration
and teaming
Role: Design
HR
management
Data and
analytics
Role developer

User Zoom Card Sort Analysis of Results and Dendrograms
The similarity matrix
further helps
establish clusters
between items, the
darker color shows
more clustering.
Card Sort

Thank You
User Research
IBM | w3 Unification

Card Sort
Open card sort
• Participants create their own labels for categories
• Used to discover patterns in how participants
which in turn helps generate ideas for organizing
information
• Can be an open number of buckets or a fixed
Closed card sorting
• Set of predetermined category names.
• Assign the cards to the fixed categories
• Reveals the extent of agreement on which cards
under each category
• Typically used to verify and validate category
add new items into an existing taxonomy
Apple
Asparagus
Cantaloupe
Carrot
Eggplant
Broccoli
Cheese
Orange
Pear
Potato
Mango
Zucchini
Eggs
Breakfast Lunch
Dinner Snack

Card sorting is used in defining and designing the information
architecture, workflows, menu structure, or web site navigation
paths.
A card sort is often used when designing a navigation structure for
an application or environment that offers a variety of content and
functions (i.e., website).
The way the items are organized should make sense to the target
audience.
If an accepted and standardized taxonomy exists for a subject
(biological taxonomy, Dewey Decimal), it would be natural to apply
that taxonomy.
Information Architecture
Presentation title

No existing taxonomy is available to organize the items.
The variety of items to organize is too large and/or diverse to
into conventional categories.
Similarities among the items make them difficult to divide
clearly into categories.
User groups differ in how they view the similarities among
the appropriate groupings of items.
When to use card sorting
Presentation title

Tools
Moderated/In person
• Benefit of discussing choices to understand the logic
the categories
• Use web-based tools (UserZoom, Alchemer) or physical
assets (cards)
Unmoderated/Remote
• Web based tools (UserZoom, Alchemer)
• Opportunity for larger number of participants in less
• Lose the context and logic for participants’
decisions

Variations on the Card Sort
The occupational card sort is a technique used by
career counselors the people with present or
future vocational choices. Sort jobs cards into
three piles: Might Choose, Probably Would Not
Not Choose, or Undecided.
Like a rating scale, however instead of adding a
a rating to an object, you move the object to the
rating/bucket.
Similarly, a 1967 study was designed to
understand individual differences in reactions to
taking LSD. Participants rated their level of
enjoyment for each of 156 common LSD
experiences (written on cards). They added a card
to one of five groups according to how much they
they did, or did not, like the experience.
1967

Neuropsychological Testing
California Verbal Learning Test (1987)
measures episodic verbal learning and
Participants are read a list of 16 words, each
belonging to four semantic categories (tools,
fruits, clothing, spices and herbs).
People with advanced abstract reasoning
categorize the words as a memory
1987

(Open) Card Sort
Participants sort “cards” and place them in
categories that make sense to them.
1. A representative user receives a randomized
set of cards with terms (and sometime
definitions)
2. The participant groups the terms in according
to what is logical and meaningful
3. The participant then labels the category
4. This process is repeated this for several group
of test participants
5. The results are analyzed to uncover and
patterns that emerge
Apple
Asparagus
Cantaloupe
Carrot
Eggplant
Broccoli
Cheese
Orange
Pear
Potato
Mango
Zucchini
Eggs
Fruits Vegetables
Foods I don’t like Misc
Card Sort

X
Card Sort
X
Y
Z
Tree
X
Y
Z
Tree
? ? ? ?
…
Open
X Y Z
Closed
X Y Z
Closed

X
Card Sort

Card Sort - UXRA Education Series_2022-06-16l.pptx

Recommended

Recommended

More Related Content

Similar to Card Sort - UXRA Education Series_2022-06-16l.pptx

Similar to Card Sort - UXRA Education Series_2022-06-16l.pptx (20)

Recently uploaded

Recently uploaded (9)

Card Sort - UXRA Education Series_2022-06-16l.pptx