The document summarizes different methods for validating information architecture (IA), including card sorting techniques. It then presents a case study where multiple validation methods were used to evaluate the IA of a healthcare website. An open card sort revealed how users naturally grouped content and identified difficult items. A closed card sort showed better alignment with the proposed IA structure. The combination of methods provided valuable insights to improve the site's organization and findability.

1. www.nicecompany.comCOMPANY NAME
Caron Garstka
UXPA 2018
Card Sorting
and
Mixed Method
IA Validation

2. www.nicecompany.comCOMPANY NAME
A little about me
Information Architecture (IA): A quick review
IA validation methods
Mixed method IA validation
Case Study
01.
02.
03.
04.
05.
C ON TEN TS
Conclusion

3. www.nicecompany.comCOMPANY NAME
A little about me
01

4. www.nicecompany.comCOMPANY NAME
A l i ttl e a bo u t m e …
Who I am…
• Caron Garstka
• Senior UX Researcher at Express Scripts
• Masters of Science in Information from University of Texas at Austin
What I do…
• Conduct research with patients who have chronic, complex, or life-threatening
conditions
• Help shape their digital experiences using their feedback
My goal is to make patient lives easier

5. www.nicecompany.comCOMPANY NAME
Information Architecture (IA): A quick
review
02

6. • It is the creation of a structure for a website, intranet, application, or other
project
• Allows users to understand where they are, and where the desired
information is in relation to their position
• Results in the creation of site maps, hierarchies, categorizations, navigation,
and metadata
W h a t i s In fo r m a ti o n Ar c h i te c tu r e ( IA) ?

7. Information architecture is about helping people understand their
surroundings and find what they’re looking for, in the real world
as well as online.
– Information Architecture Institute
W h a t i s In fo r m a ti o n Ar c h i te c tu r e ( IA) ?

8. A Ph a se d Appr o a c h to IA D e ve l o pm e n t
There are multiple steps to creating and maintaining an IA
Research Strategy Design Implementation Administration
Card Sorting
lives here!

9. A Ba l a n c e d IA Re se a r c h Appr o a c h
Documents
Data types
Content objects
Metadata
Volume
Existing structure
Business goals
Funding
Politics
Culture
Technology
Resources
Constraints
Audience
Tasks
Needs
Information seeking behavior
Experience
Vocabularies
C o n t e n t C o n t e x t U s e r s

10. How is it categorized?
How is it displayed?
Where should it be stored?
What information is relevant at a
given time?
Is the information necessary?
What are they looking for?
What’s important to them?
What are they familiar with?
C o n t e n t C o n t e x t U s e r s
A Ba l a n c e d IA Re se a r c h Appr o a c h

11. A Ba l a n c e d IA Re se a r c h Appr o a c h
Heuristic evaluation
Metadata and content analysis
Content mapping
Benchmarking
Background research
Presentations and meetings
Stakeholder interviews
Technology assessments
Search log and clickstream analysis
Use cases and personas
Contextual inquiry
User interviews and user testing
C o n t e n t C o n t e x t U s e r s

12. IA
Context
UsersContent
A Ba l a n c e d IA Re se a r c h Appr o a c h

13. www.nicecompany.comCOMPANY NAME
IA Validation Methods
03

14. Content audits
Establishing starting point
Surveys
Gathering opinions
Field studies, diaries, contextual
inquiry, interviews, focus groups
Learning about users
Web traffic statistics, clickstreams
Online behavior
Be fo r e do i n g IA va l i da ti o n …
Do the following methods before starting validation research.

15. IA Va l i da ti o n R e s e a r c h M e th o ds
Tree Testing
Looks at IA as working hierarchy
Closed Card Sorting
Looks at viability of defined categories
Click Testing
Looks at first-level IA in context of UI
Hybrid Card Sorting
Combination of open and closed
Skews open or closed
Open Card Sorting
Looks at user-generated groupings and names
Delphi Card Sorting
Looks at user-generated groupings and names

16. IA Va l i da ti o n R e s e a r c h M e th o ds
Tree Testing
Looks at IA as working hierarchy
Closed Card Sorting
Looks at viability of defined categories
Click Testing
Looks at first-level IA in context of UI
Hybrid Card Sorting
Combination of open and closed
Skews open or closed
Open Card Sorting
Looks at user-generated groupings and names
Delphi Card Sorting
Looks at user-generated groupings and names
• Users sort items into groups and give groups a name; typically done
prior to an architecture being put in place
• Allows the researcher to document a user’s mental model and inform
the IA

17. IA Va l i da ti o n R e s e a r c h M e th o ds
Tree Testing
Looks at IA as working hierarchy
Closed Card Sorting
Looks at viability of defined categories
Click Testing
Looks at first-level IA in context of UI
Hybrid Card Sorting
Combination of open and closed
Skews open or closed
Open Card Sorting
Looks at user-generated groupings and names
Delphi Card Sorting
Looks at user-generated groupings and names
• Users sort items into defined categories
• Allows the researcher to document a user’s mental model and
understand how well the proposed IA matches their model

18. IA Va l i da ti o n R e s e a r c h M e th o ds
Tree Testing
Looks at IA as working hierarchy
Closed Card Sorting
Looks at viability of defined categories
Click Testing
Looks at first-level IA in context of UI
Hybrid Card Sorting
Combination of open and closed
Skews open or closed
Open Card Sorting
Looks at user-generated groupings and names
Delphi Card Sorting
Looks at user-generated groupings and names
• Users sort items into researcher-defined categories but have the
option to create their own
• Allows the researcher to document a user’s mental model and allows
users to participate in building the IA

19. IA Va l i da ti o n R e s e a r c h M e th o ds
Tree Testing
Looks at IA as working hierarchy
Closed Card Sorting
Looks at viability of defined categories
Click Testing
Looks at first-level IA in context of UI
Hybrid Card Sorting
Combination of open and closed
Skews open or closed
Open Card Sorting
Looks at user-generated groupings and names
Delphi Card Sorting
Looks at user-generated groupings and names
• Users work one after another to complete card sort
• Users modify as needed and pass modifications to next user
• Allows the researcher to rapidly arrive at an organizational model

20. IA Va l i da ti o n R e s e a r c h M e th o ds
Tree Testing
Looks at IA as working hierarchy
Closed Card Sorting
Looks at viability of defined categories
Click Testing
Looks at first-level IA in context of UI
Hybrid Card Sorting
Combination of open and closed
Skews open or closed
Open Card Sorting
Looks at user-generated groupings and names
Delphi Card Sorting
Looks at user-generated groupings and names
• Users navigate a hierarchy using only top-level link names.
• Allows the researcher to see how findable items are within the
hierarchy

21. I A V a l i d a t i o n R e s e a rc h Me t h o d s
Tree Testing
Looks at IA as working hierarchy
Closed Card Sorting
Looks at viability of defined categories
Click Testing
Looks at first-level IA in context of UI
Hybrid Card Sorting
Combination of open and closed
Skews open or closed
Open Card Sorting
Looks at user-generated groupings and names
Delphi Card Sorting
Looks at user-generated groupings and names
• Users locate items within a static design
• Allows the researcher to see if the top level IA is easily findable within
the context of the design

22. C ompari son of IA Research Methods
Method Research Type Activity Learn About Project Stage
Open card sorting Generative Organizing
Mental models
Organization
Labeling
Relationships between items
Early user research
Closed card sorting Evaluative Organizing
Mental models
Organization
Labeling
Information architecture design
Hybrid card sorting
Generative
Evaluative
Organizing
Mental models
Organization
Labeling
Relationships between items
Early user research
Delphi card sorting Generative Organizing
Mental models
Organization
Labeling
Relationships between items
Early user research
Reverse card sorting
(Tree testing)
Evaluative Finding
Organization
Labeling
Findability
Information architecture design
Click testing
(or usability testing with low-
fidelity prototypes)
Evaluative Finding
Organization
Labeling
Findability
Navigation design
Interface design
Interface design

23. Pr a c ti c a l T r i a n gu l a ti o n
[Triangulation is an] attempt to map out, or explain
more fully, the richness and complexity of human
behavior by studying it from more than one
standpoint.
- Cohen and Manion (1986)

24. www.nicecompany.comCOMPANY NAME
Case Study: Accredo.com
04

25. www.nicecompany.comCOMPANY NAME
Accredo.com
Accredo.com
1. Stakeholders wanted to validate a proposed IA on the Accredo.com
website
2. Conduct a closed card sort to catch any glaring issues
The Perceived Need

26. www.nicecompany.comCOMPANY NAME
The Real Need
Expanded scope…
1. How well the proposed IA matches the user’s mental model
2. How findable items are within the proposed IA
3. How UI helps or obscures IA
4. Areas for improvement
5. Validate the proposed patient IA
6. Learn how potential users group content
7. Evaluate the findability of topics within the website structure
8. Evaluate the homepage design and whether potential users can
fine the expected entry point for specific information or
functionality within the design
And there wasn’t much time…
Accredo.com
Accredo.com

27. www.nicecompany.comCOMPANY NAME
• Site caters to multiple audiences
• Users are distributed
• Site IA is different pre and post login
Focus on one side of the experience first.
Conduct study remotely, asynchronously, and without
moderation.
Allowed for fast data capture without having to
moderate each session.
Accredo.com
Accredo.com
Other Challenges

28. Appr o a c h
Tree Testing
Closed Card Sorting
Click Testing
Hybrid Card SortingOpen Card Sorting
Delphi Card Sorting
One method alone would not get at what the team needed.

29. Appr o a c h
Used UserZoom to conduct the study.
Tree Test
N= 43
Median Completion Time: 25 min
Click Test
N= 44
Median Completion Time: 20 min
Open/Closed Card Sort
N= 87

30. Appr o a c h
• Get at users’ mental model
• Note alternative categories, item
placement, and category names
• Pin point hard to place items
Step 01
Open Card Sort
• See how users group items
within given categories
• See how well groupings
match the proposed IA
• Note alternative categories,
item placement, and
category names
• Pin point hard to place items
Step 02
Closed Card Sort
• Look at where users drill into
the tree
• Note findability of key items
Tree Test
Step 03A
• Look at where users drill into
the design
• Note expectations of finding
items in design
Click Test
Step 03B

31. A p p ro a c h
Tree Test
N= 43
Median Completion Time: 25 min
Click Test
N= 44
Median Completion Time: 20 min
Open/Closed Card Sort
N= 87

32. C ard Sort Setup
1. Selected content that could be grouped
2. Selected content that was at the same level
3. Selected content that was representative
4. Selected content that did not lead the activity
5. Selected content that participants would understand
Maintained consistency across the open and closed card sorts.

33. Ope n C a r d So r t
• Presented participants with a list of Accredo proposed IA items and asked them to sort
those items into groups
• Participants sorted items into groups that made sense to them
• Participants created their own names for the categories the created
• Some labels were modified for the card sort to add clarity
UserZoom interface for open card sort

34. Ope n C a r d So r t F i n di n gs
• Dendrogram: A tree diagram of
hierarchical clustering to learn
how users group items.
• Dendorgram scale: 0 to 1; the
lowest level of clusters with the
highest agreement.
• Euclidian distance the average
distance between items.
Interpreting dendrogram
clusters is an art.
Data Output: Dendrogram Overview

35. Ope n C a r d So r t F i n di n gs
Interpretations
Cluster 2: Had a runt:
“Communication Information for
digital support team”
Cluster 1: Fairly cohesive,
aligned with the hypothesized
IA, and participants came up
with similar names
Cluster 3: High agreement for
items related to prescriptions, not
matching hypothesized IA, but
logically made sense

36. Ope n C a r d So r t F i n di n gs
Cluster 4: Cohesive group related
to a Welcome or introductory
section
Cluster 5: Mixed assortment of
forms and administration items.
These items had strong
agreement but did not match
IA.
Interpretations

37. Ope n C a r d So r t F i n di n gs
Finding: Participants grouped items according to Disease FAQ, Prescriptions and Refills, and Account
functions
Participants were less certain on how to handle “Communication information for digital support team”

38. Ope n C a r d So r t F i n di n gs
Difficult Items to Sort
Items Participants Wanted to Sort into
Multiple Categories
Post hoc questions about the sorting experience provides additional insights

39. Ope n C a r d So r t F i n di n gs
Difficult Items to Sort
Items Participants Wanted to Sort into
Multiple Categories
Post hoc questions about the sorting experience provides additional insights
Finding: “Communication information for digital support team” and “Register” were both highly ranked for difficulty to sort
and participants thought the items could belong in multiple categories.
Perhaps they would be easier to sort if given a structure.
Sounds like a job for the Closed Card Sort!

40. C l osed C ard Sort
• Participants were first presented with a list of Accredo IA items and were asked to sort
those items into pre-named groups
UserZoom interface for closed card sort

41. C l osed C ard Sort
Wanted to see how well participants’ sorts aligned with proposed closed groups.
Home Getting Started Conditions Find Medication Contact Us
Manage your account
Create an Accredo account
information
Symptoms Search to view Accredo drugs
Communication information for
digital support team
Download Accredo mobile app
Authorization to use and disclose
health information PDF
Disorder overview
Phone numbers for disorder-
specific assistance
Order status
Authorization to use and disclose
health information PDF
Find disease-specific support Billing call information
Register
Patient consent agreement and
acknowledgement document
Specific disease/disorder care
team
Phone number to refill medicine
How to begin with Accredo Welcome to Accredo information Financial resources for a disorder Fax number for new prescription
Refill an Rx Filling Rx for the first time
Medicines for a particular
disease
Secure email form
New patient refill process Payment mail-in address
Patient forms
What is a specialty pharmacy?
video
Refill procedure for existing
patients

42. C l osed C ard Sort Fi ndi ngs
Methodology
• For this analysis I chose to
use the average linkage
dendrogram for analysis.
• I set the agreement
minimum to 50% - this is the
minimum percentage of
agreement the selected
items that participants
included into the same
category.

43. C l osed C ard Sort Fi ndi ngs
Finding: Participants seemed to have an easier time grouping “Communication information for digital support
team” – this was the runt in the open sort – grouping it in with other “contact” related items.
Interestingly, “Medicines for a particular disease” and “Search to view Accredo drugs” hung together, where as
in the open card sort they did not.

44. C l osed C ard Sort Fi ndi ngs
Items Home Getting Started Conditions Find a Medication Contact Us Highest %
Register #% #% #% #% #% #%
Refill an Rx #% #% #% #% #% #%
Order status #% #% #% #% #% #%
How to begin with Accredo #% #% #% #% #% #%
Welcome to Accredo information #% #% #% #% #% #%
New patient refill process #% #% #% #% #% #%
Refill procedure for existing patients #% #% #% #% #% #%
Manage your account #% #% #% #% #% #%
Disorder overview #% #% #% #% #% #%
Symptoms #% #% #% #% #% #%
Medicines for a particular disease #% #% #% #% #% #%
Financial resources for a disorder #% #% #% #% #% #%
Find disease-specific support #% #% #% #% #% #%
Specific disease/disorder care team #% #% #% #% #% #%
Search to view Accredo drugs #% #% #% #% #% #%
What is a specialty pharmacy? video #% #% #% #% #% #%
Authorization to use and disclose health information
PDF
#% #% #% #% #% #%
Patient consent agreement and acknowledgement
document
#% #% #% #% #% #%
Phone number to refill medicine #% #% #% #% #% #%
Fax number for new prescription #% #% #% #% #% #%
Patient forms #% #% #% #% #% #%
Filling Rx for the first time #% #% #% #% #% #%
Create an Accredo account information #% #% #% #% #% #%
Phone numbers for disorder-specific assistance #% #% #% #% #% #%
Secure email form #% #% #% #% #% #%
Payment mail-in address #% #% #% #% #% #%
Billing call information #% #% #% #% #% #%
Communication information for digital support team #% #% #% #% #% #%
Download Accredo mobile app #% #% #% #% #% #%

45. C l osed C ard Sort Fi ndi ngs
Items Home Getting Started Conditions Find a Medication Contact Us Highest %
Register #% #% #% #% #% #%
Refill an Rx #% #% #% #% #% #%
Order status #% #% #% #% #% #%
How to begin with Accredo #% #% #% #% #% #%
Welcome to Accredo information #% #% #% #% #% #%
New patient refill process #% #% #% #% #% #%
Refill procedure for existing patients #% #% #% #% #% #%
Manage your account #% #% #% #% #% #%
Disorder overview #% #% #% #% #% #%
Symptoms #% #% #% #% #% #%
Medicines for a particular disease #% #% #% #% #% #%
Financial resources for a disorder #% #% #% #% #% #%
Find disease-specific support #% #% #% #% #% #%
Specific disease/disorder care team #% #% #% #% #% #%
Search to view Accredo drugs #% #% #% #% #% #%
What is a specialty pharmacy? video #% #% #% #% #% #%
Authorization to use and disclose health information
PDF
#% #% #% #% #% #%
Patient consent agreement and acknowledgement
document
#% #% #% #% #% #%
Phone number to refill medicine #% #% #% #% #% #%
Fax number for new prescription #% #% #% #% #% #%
Patient forms #% #% #% #% #% #%
Filling Rx for the first time #% #% #% #% #% #%
Create an Accredo account information #% #% #% #% #% #%
Phone numbers for disorder-specific assistance #% #% #% #% #% #%
Secure email form #% #% #% #% #% #%
Payment mail-in address #% #% #% #% #% #%
Billing call information #% #% #% #% #% #%
Communication information for digital support team #% #% #% #% #% #%
Download Accredo mobile app #% #% #% #% #% #%
Finding: “Communication information for digital support team” and “Register” were both highly ranked for difficulty
sorting and belonging in multiple categories in the open card sort.
However, once given a structure, participants had higher agreement for placement.
“Patient forms” has a split grouping for where participants think it belongs.

46. C l osed C ard Sort Fi ndi ngs
Difficult Items to Sort
Items Participants Wanted to Sort into
Multiple Categories
Post hoc questions about the sorting experience provides additional insights

47. C l osed C ard Sort Fi ndi ngs
Difficult to Sort
Patient consent agreement and acknowledgement document
What is a specialty pharmacy? video
Authorization to use and disclose health information PDF
Patient forms
Refill an Rx
Communication information for digital support team
Download Accredo mobile app
Billing call information
Fax number for new prescription
New patient refill process
Payment mail-in address
Search to view Accredo drugs
Create an Accredo account information
Phone number to refill medicine
Refill procedure for existing patients
Financial resources for a disorder
Find disease-specific support
How to begin with Accredo
Order status
Disorder overview
Manage your account
Medicines for a particular disease
Register
Secure email form
Welcome to Accredo information
Filling Rx for the first time
Phone numbers for disorder-specific assistance
Specific disease/disorder care team
Top
“Difficult to Sort”
Items
Difficult to Sort
Finding: “Communication information for digital support team” and “Register” were both highly ranked for participants thinking
they were difficult to sort in the open card sort. Both dropped overall in the closed card sort.
But now Refilling Rx is one of the top 5 most difficult to sort.

48. Items Participants felt belonged in
Multiple Categories
Communication information for digital support team
Order status
Patient consent agreement and acknowledgement document
Create an Accredo account information
Phone number to refill medicine
Authorization to use and disclose health information PDF
Billing call information
Fax number for new prescription
New patient refill process
Patient forms
Find disease-specific support
Manage your account
Refill procedure for existing patients
What is a specialty pharmacy? video
Financial resources for a disorder
Payment mail-in address
Refill an Rx
Register
Search to view Accredo drugs
Specific disease/disorder care team
Welcome to Accredo information
Filling Rx for the first time
Medicines for a particular disease
Phone numbers for disorder-specific assistance
Secure email form
Disorder overview
Download Accredo mobile app
How to begin with Accredo
Symptoms
Top
“Multiple Categories”
Items
Multiple Categories
Finding: Even though Refill Rx was top five most difficult to sort, it was not as highly ranked for belonging in multiple
categories.
Order status has also jumped up in the list. This would be a prime candidate to look at in the tree test as well to see where
people start looking for it.
C l osed C ard Sort Fi ndi ngs

49. C l osed C ard Sort Fi ndi ngs
Home Getting Started Conditions Find Medication Contact Us
Hypothesis Correct Hypothesis Correct Hypothesis Correct Hypothesis Correct Hypothesis Correct
Manage your account Green
Create an Accredo
account information
Green Symptoms Green
Search to view
Accredo drugs
Green
Communication
information for digital
support team
Green
Download Accredo
mobile app
Yellow
Authorization to use
and disclose health
information PDF
Yellow Disorder overview Green
Phone numbers for
disorder-specific
assistance
Green
Order status Red
Authorization to use
and disclose health
information PDF
Yellow
Find disease-specific
support
Green Billing call information Green
Register Red
Patient consent
agreement and
acknowledgement
document
Yellow
Specific
disease/disorder care
team
Yellow
Phone number to refill
medicine
Yellow
How to begin with
Accredo
Red
Welcome to Accredo
information
Yellow
Financial resources for
a disorder
Red
Fax number for new
prescription
Yellow
Refill an Rx Red
Filling Rx for the first
time
Yellow
Medicines for a
particular disease
Red Secure email form Yellow
New patient refill
process
Red
Payment mail-in
address
Yellow
Patient forms Red
What is a specialty
pharmacy? video
Red
Refill procedure for
existing patients
Red

50. C l osed C ard Sort Fi ndi ngs
Home Getting Started Conditions Find Medication Contact Us
Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct
Manage your account Green
Create an Accredo
account information
Green Symptoms Green
Search to view
Accredo drugs
Green
Communication
information for digital
support team
Green
Download Accredo
mobile app
Yellow
Authorization to use
and disclose health
information PDF
Yellow Disorder overview Green
Phone numbers for
disorder-specific
assistance
Green
Order status Red
Authorization to use
and disclose health
information PDF
Yellow
Find disease-specific
support
Green Billing call information Green
Register Red
Patient consent
agreement and
acknowledgement
document
Yellow
Specific
disease/disorder care
team
Yellow
Phone number to refill
medicine
Yellow
How to begin with
Accredo
Red
Welcome to Accredo
information
Yellow
Financial resources for
a disorder
Red
Fax number for new
prescription
Yellow
Refill an Rx Red
Filling Rx for the first
time
Yellow
Medicines for a
particular disease
Red Secure email form Yellow
New patient refill
process
Red
Payment mail-in
address
Yellow
Patient forms Red
What is a specialty
pharmacy? video
Red
Refill procedure for
existing patients
Red
Since the hypothesis works for some items but not others, we need to know how easily users can find these items given the
current hierarchy.
Card sorting is exploratory, tree testing is confirmatory.
Time to look at the tree test…

51. T r e e T e s t Se tu p
The setup for the tree test activity needed to align with the card sort and then be carried over into the
click test.
1. Select items that may be difficult to sort, items you want more data on, or items we had additional
questions about
2. Select easy and hard items based on location in the tree
3. Create tasks for those items

52. • Participants were presented with the top-level navigation
• Participants then were asked to perform tasks using the tree
• When a participant believed they found the desired information, they would select it and continue. If they
were unsure where to find the information, they could select “I don’t know”.
T r e e T e s t

53. Clicks
Task Home
Getting
Started
Conditions
Find
Medication
Contact Us
Overall
Success
Success Path Time N
Sign up for Accredo Specialty Pharmacy #% #% #% #% #% #%
Home > Register #%; Create an Account > Register Now #%; Filling for the
First Time > Call patient services to get started #%; Home > How to get
started #%; Welcome > Register Now #%
0:00 #
Reorder a drug your doctor prescribed you #% #% #% #% #% #% Home > Refill an Rx #%; Contact Us > Call patient services – Refills and
Orders #%; Refilling Prescriptions > Phone number to refill prescription #%
0:00 #
Check where your drug is in the fulfillment and delivery process #% #% #% #% #% #%
Contact Us > Call patient services – Refills and Orders #%; Home > Order
Status #%
0:00 #
Find information about the process for getting set up with Accredo
Specialty Pharmacy
#% #% #% #% #% #%
Home > How to get started #%; Create an Account > Create an Accredo
account information #%; Welcome > Welcome to Accredo! Information #%
0:00 #
Learn about the purpose of a specialty pharmacy #% #% #% #% #% #%
Welcome > “What is a specialty pharmacy?” video #%; Welcome > What
Accredo provides the patient #%
0:00 #
Learn how to order your drug with Accredo Specialty Pharmacy as a
new patient
#% #% #% #% #% #%
Filling for the First Time > How to fill your prescription with Accredo for the first
time #%; Home > How to get started #%; Filling for the First Time > Call patient
services to get started #%
0:00 #
Learn how to order your drug with Accredo Specialty Pharmacy as a
returning patient
#% #% #% #% #% #%
Refilling Prescriptions > Existing Patient – How to refill your prescription #%;
Contact Us > Call patient services – Refills and Orders #%; Home > Refill an
Rx #%
0:00 #
Find general information about Bladder Cancer #% #% #% #% #% #% Bladder Cancer > Overview #% 0:00 #
Find out the signs of having Epilepsy #% #% #% #% #% #% Epilepsy > Symptoms #% 0:00 #
Learn about drug treatments associated with Hemophilia B #% #% #% #% #% #%
Hemophilia B > Medications #%; Find a Medication > Search Medications
Offered #%
0:00 #
Learn more about manufacturer and community programs that help
reduce the drug costs for Growth Hormone Replacement
#% #% #% #% #% #% Growth Hormone Replacement > Financing Your Care #% 0:00 #
Learn more about Leukemia assistance programs and advocacy
organizations
#% #% #% #% #% #% Leukemia > Find Support #% 0:00 #
Learn more about Hepatitis C's Accredo care team #% #% #% #% #% #% Hepatitis C > Accredo Team #% 0:00 #
Find a drug by it's manufacturer's name #% #% #% #% #% #% Find a Medication > Search Medications Offered #% 0:00 #
Find number to call to ask about a balance on your account #% #% #% #% #% #% Contact Us > Call patient services – Billing and Payments #% 0:00 #
Find the mailing address to send in money to pay off an account
balance
#% #% #% #% #% #% Contact Us > Mail your payments address #% 0:00 #
Find phone number for a specialized condition-assistance team #% #% #% #% #% #% Contact Us > Call patient services – Condition-specific assistance #% 0:00 #
Reach out to the Digital Support Team to get help with the website #% #% #% #% #% #% Contact Us > Call patient services – Digital Support Team #% 0:00 #
Find the information for sending Accredo Specialty Pharmacy a fax #% #% #% #% #% #% Contact Us > Send us a fax – General Patient Fax #% 0:00 #
Send Accredo Specialty Pharmacy an email from the website #% #% #% #% #% #% Contact Us > Send us a secure online message – Email us #% 0:00 #
T r e e T e s t F i n di n gs

54. T r e e T e s t F i n di n gs
Tasks
Overall
Success
Success First
Attempt
Success
Multiple
Attempts
Non Success I don't know
Percent Time Percent Time Percent Time Percent Time Percent Time
Sign up for Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Reorder a drug your doctor prescribed you #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Check where your drug is in the fulfillment and delivery process #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find information about the process for getting set up with Accredo Specialty
Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn about the purpose of a specialty pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn how to order your drug with Accredo Specialty Pharmacy as a returning
patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find general information about Bladder Cancer #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find out the signs of having Epilepsy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn about drug treatments associated with Hemophilia B #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn more about manufacturer and community programs that help reduce the
drug costs for Growth Hormone Replacement #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn more about Leukemia assistance programs and advocacy organizations #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn more about Hepatitis C's Accredo care team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find a drug by it's manufacturer's name #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find number to call to ask about a balance on your account #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find the mailing address to send in money to pay off an account balance #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find phone number for a specialized condition-assistance team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Reach out to the Digital Support Team to get help with the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find the information for sending Accredo Specialty Pharmacy a fax #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Send Accredo Specialty Pharmacy an email from the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00

55. T r e e T e s t F i n di n gs
Tasks
Overall
Success
Success First
Attempt
Success
Multiple
Attempts
Non Success I don't know
Percent Time Percent Time Percent Time Percent Time Percent Time
Sign up for Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Reorder a drug your doctor prescribed you #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Check where your drug is in the fulfillment and delivery process #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find information about the process for getting set up with Accredo Specialty
Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn about the purpose of a specialty pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn how to order your drug with Accredo Specialty Pharmacy as a returning
patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find general information about Bladder Cancer #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find out the signs of having Epilepsy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn about drug treatments associated with Hemophilia B #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn more about manufacturer and community programs that help reduce the
drug costs for Growth Hormone Replacement #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn more about Leukemia assistance programs and advocacy organizations #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Learn more about Hepatitis C's Accredo care team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find a drug by it's manufacturer's name #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find number to call to ask about a balance on your account #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find the mailing address to send in money to pay off an account balance #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find phone number for a specialized condition-assistance team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Reach out to the Digital Support Team to get help with the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Find the information for sending Accredo Specialty Pharmacy a fax #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Send Accredo Specialty Pharmacy an email from the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
Finding: “Check where your drug is in the fulfillment and delivery process” (Order Status)
placement in the card sorts did not match our hypothesis.
Participants also looked in a variety of categories trying to find this item in the tree test
which made for longer search times.
Where would they look on a homepage image? Time for the click test.

56. C l i ck Test Setup
The setup for the click test activity needed to align with the tree test so the same
tasks were used.
1. Select items that may be difficult to sort, items you want more data on, or
items we had additional questions about.
2. Selected easy and hard items based on location in the tree.
3. Create tasks for those items.

57. C l i ck Test
• Participants were presented with static image of home screen
• Participants then were asked to perform tasks and click where they think they would find information

58. C l i ck Test Fi ndi ngs
“Check where your drug is in the
fulfillment and delivery process”
(Order Status) placement did not
match IA hypothesis. Participants
also looked in a variety of
categories trying to find this item in
the tree test which made for longer
search times.
Where would they look on a
homepage image?

59. www.nicecompany.comCOMPANY NAME
Conclusion
05

60. An a l ysi s Appr o a c h
Step 01
Open Card Sort Tree Test
Step 03A
Click Test
Step 03BStep 02
Closed Card Sort
To recap how we approached analysis…

61. An a l ysi s Appr o a c h
Step 01
Open Card Sort Tree Test
Step 03A
Click Test
Step 03B
• Examine the groups
• Analyze card placement
• Analyze labels
• Analyze organizational schemes
• Analyze how accurately participants have grouped content
• Analyze participant comments
• Analyze a closed card sort
• Look at items classified as difficult to sort or belonging in multiple
categories
Step 02
Closed Card Sort
To recap how we approached analysis…

62. An a l ysi s Appr o a c h
Step 01
Open Card Sort Click Test
Step 03B
• Analyze success rate
• Analyze directness of success
• Analyze time on task
• Look at card sort items classified as difficult to sort or belonging in
multiple categories – how do participants approach looking for those
items?
• Look at runts or categories with two items – are they more easily found
in the tree? Where are users looking to find them?
Step 02
Closed Card Sort Tree Test
Step 03A
To recap how we approached analysis…

63. An a l ysi s Appr o a c h
Step 01
Open Card Sort Click Test
Step 03BStep 02
Closed Card Sort Tree Test
Step 03A
• Compare where users drill into the tree v. where they click into the UI
• Look at initial entry points for the tree and first click data for the click
test – are there similarities?
• Note if success changes from the tree test to the click test
To recap how we approached analysis…

64. T i ps fo r M i x e d M e th o d IA Stu dy Se tu p
• Select items and tasks that are crucial or controversial
• Ordering of method matters
• Be consistent across the study
• Participant fatigue is a thing –scope your work!

65. T i ps fo r M i x e d M e th o d IA An a l y s i s
1. Determine if new categories should be created or
consolidated using the mental model put forth in open
card sort
2. Look to confirm or deny hypotheses about categories
using the closed card sort
3. See what issues of findability there are in the tree test
4. Note where people would place items in the UI in the
click study

66. • Allows for data comparisons – useful in understanding
convergence and divergence of findings
• Collects rich, comprehensive data – provides a more
complete story than a single method would alone
• Combining two or more of these methods worth the
cost.
• Time consuming to create
• Participants can easily grow tired
• Data might be messy
• Can be difficult to successfully integrate data
Some challenges with mixed method
IA validation…
Adva n ta ge sC hal l enges
Some advantages of mixed method
IA validation…

67. C oncl usi on
By spending the time constructing a more robust study design, you can create
a holistic view of how the user is thinking and behaving toward your IA.

68. www.nicecompany.comCOMPANY NAME
GET IN T OU C H
cegarstka@express-scripts.com
caron.garstka@gmail.com
E-mail
Thanks!

69. www.nicecompany.comCOMPANY NAME
Questions!