Presentation given to the SLA at Drexel University covering the reasons for taxonomy testing, Delphi-method card sorting, remote card sorting, usability testing, and search analysis. A list of resources and online validation tools is included.

1. Taxonomy Validation
Dave Cooksey
Founder & Principal
DUSLA | Drexel University, Philadelphia
October 14, 2009

2. Today’s Presentation
‣ About Dave
‣ Testing & Taxonomy | Metadata
‣ Taxonomy Validation Techniques
‣ Delphi Card Sorting
‣ Remote Card Sorting
‣ Usability Testing
‣ Search Analysis
‣ Discussion
‣ Resources

3. Dave Cooksey
Founder & Principal
saturdave UX Consulting
Affiliations
‣ Chair of PhillyCHI, the Philadelphia region’s chapter of the ACM SIGCHI
‣ Information Architecture Institute (IAI)
‣ American Society for Information Science & Technology (ASIS&T)
Education
‣ M.S. Information Systems, Drexel University
‣ M.A. Political Science, University of Illinois
Previous Work Experience
‣ GSI Commerce, Inc.: User Experience Lead; Information Architect
‣ LandAmerica Financial Group: (Web) Application Developer

4. Testing & Taxonomy | Metadata

5. Taxonomy
‣ Is an expert activity
‣ Comes from a privileged view
‣ Is more closely aligned to business & technology goals than user goals
Taxonomies & Metadata
‣ By their very nature, hold implicit meanings
‣ Hierarchy & relationships
‣ Labeling & naming convention
‣ Are interpreted from the user’s experience, context, & goals

6. The experience, context, & goals of
users are rarely our own!

7. Taxonomy Validation
‣ Ensures information organization is
user-centric
‣ Vets untried ideas with real users
You’ve been warned: Princeton Record Exchange.
But it also...
‣ Justifies the cost of taxonomy
research & analysis
‣ Reassures the business that project
focus is on success
‣ Helps alleviate organizational conflict

8. Formative Testing Techniques
‣ Explores the concepts that users employ to describe the real world
‣ Focuses on the way folks think about something (mental models)
‣ Encourages open discussion & brainstorming
‣ Informs your design
Summative Testing Techniques
‣ Examines the way people use an existing design
‣ Focuses on the way folks do something (work flow)
‣ Encourages feedback & critique
‣ Informs design strategy

9. Testing Requires Neutrality
‣ When testing, you’re not a taxonomist anymore - you’re a fly on the wall
‣ Good test moderation is hard and takes practice
How to Moderate Effectively...
‣ Reassure the participant that you will not be upset by comments or critique:
“I’m not the designer so I won’t be offended by what you say.”
‣ Avoid leading questions - try to say as little as possible
‣ Don’t ask a participant if they would like something, a feature, category,
label - she will almost always say, “yes”
‣ When asked specific questions of how things work, avoid explaining and
simply say something like, “I don’t know. What do you think?”

10. Consider Test Structure & Effects
‣ Environmental effects: in-lab, in the mall, moderated session online
‣ Precedent effects: order of activities, transparency of previous results
‣ Compensation effects: stipend up front, afterwards, or none at all
‣ Moderation effects: interview during, after, or none at all
‣ Methodological effects: think aloud, observation, artifact creation
‣ Medium effect: in-person, on the phone, online

11. How you structure the test will affect the
results so pay attention to details.

12. Taxonomy Validation Techniques

13. A Few Taxonomy Validation Techniques
‣ Delphi Card Sorting
‣ Remote Card Sorting
‣ Usability Testing
‣ Search Analysis

14. A Few Taxonomy Validation Techniques
Technique Method Focus Use
Delphi Card Sorting Direct Concepts Test baseline taxonomy
Remote Card Sorting Direct Concepts
Test categorization &
labels
Usability Testing Direct Interactions
Test navigation, search,
browsing
Search Analysis Indirect Patterns
Adjust taxonomy terms &
priority

15. Delphi Card Sorting
Qualitative | Small Sample
♥ Based on the Delphi Method (RAND Corporation)
♥ Developed by Professor Kathryn Summers & Celeste Lyn Paul at University
of Baltimore
♥ Hierarchy is laid out in cards - seeded or not
♥ Test for both categories & individual items
♥ Participants modify the structure one at a time
♥ Runs for 15 - 20 participants until hierarchy “stabilizes”
♥ Watch and learn - Interview for detail
♥ Can also present navigation schema, images for labeling, filters | facets

16. Step-by-Step Delphi Card Sorting
1. Decide to seed deck or let first user create seed
2. Think of other pertinent test points: navigation, facets, images for labeling
3. Put categories on cards (You or first participant)
4. Put representative items on cards | script (You)
5. Interview participant
a. Explain exercise
b. Participant organizes and comments
c. Watch and interview
6. Repeat with participants until you are satisfied
7. Analyze data

17. Performing Delphi Card Sorting
1. User starts with deck from previous participant
2. Participant modifies card deck
a. Removing category - turning the card over
b. Changing labels - creating new card & place over old one
c. Moving category - turn card over & create new card in new place
3. Participant examines changes made to card deck
a. Looks at existing cards & placement
b. Turns cards over to see previous changes to labels & placement
4. Moderator interviews informally during session

19. Different than Traditional Card Sorting
♥ Targets group understanding, not individual understandings
♥ Participants pick up where previous participant left off
♥ Assumes some level of “expertise”
♥ The end result is the “final” hierarchy
♥ Round Robin approach reduces analysis time
♥ Informs the “black box” (Thanks, Michael)

20. Different than Traditional Card Sorting
♥ Delphi Method designed for use with groups of experts
♥ Each expert contributes her knowledge to building a consensus view
♥ Outliers are removed through iteration
Consider...
♥ What kind of users are you testing?
♥ Do they have domain expertise?
♥ Do you have multiple user types? Then multiple studies perhaps?
♥ Do you have a general population (e.g. public website)?

21. Tips for Delphi Card Sorting
♥ Plan for lots of table space for cards
♥ Allow for plenty of time between participants
♥ Create a “starter” sheet to capture notes
♥ Take pictures of table after each participant
♥ Detail is great, but keep your eye on patterns
♥ Video recording is essential
♥ Use questions to explore what the user is doing
♥ Employ multiple methods to solicit feedback

22. Tips for Remote Card Sorting
Quantitative | Large Sample
‣ Online tools allow testing categories & labels
‣ Either open or closed
‣ Closed - place items in hierarchy - tests structure
‣ Open - create categories for items - gathers many perspectives
‣ Recruit via intercept or email
‣ Easily run for large sample sizes
‣ Captures national sample
‣ Good for identifying patterns
‣ Use with in-person card sorting for balance

23. Step-by-Step Remote Card Sorting
1. Decide on open or closed card sort
2. Decide population target (online, email, other)
3. Determine sample size needed
4. Choose software package
5. Focus each sort to 15 - 20 items
6. Put items | categories on cards
7. Run study
8. Analyze the data

26. Tips for Remote Card Sorting
‣ Run a few qualitative interviews to vet if possible
‣ Scale the test appropriately - run several smaller studies if needed
‣ Pick important categories | items to test
‣ Significant to the business but hard to classify
‣ Stereotypical cases that justify taxonomy guidelines
‣ Give yourself time to dig into the raw data
‣ Read the juicy, juicy comments!
‣ Deliverables depend on software used
‣ Your satisfaction with the tool depends on software used

27. Usability Testing
Qualitative | Small Sample | Task-based
‣ Piggyback on system testing
‣ From paper prototyping to existing systems
‣ Great for testing navigation step-by-step (as opposed to birds eye view)
‣ Helps test other important aspects of taxonomy: facets, tagging, non-
hierarchical relationships
‣ Great for examining the contextual interaction of multiple taxonomy |
metadata points
‣ Demonstrates explicitly the value of taxonomy
‣ Good for reassuring the business
‣ Way easy

28. Tips for Usability Testing
‣ If someone else is testing, simply explain your goals & offer tasks
‣ Keep your tasks simple
‣ Pick tasks for the business (most significant) & for you (test the taxonomy)
‣ Give multiple tasks so moderator can pick for the situation
‣ Watch the live sessions and ask follow ups
‣ Take the opportunity to show to other stakeholders how the data is driving the
experience
‣ Offer to create an overview for the study organizer

29. Search Analysis
Quantitative | Interpretive | Focused on Patterns
‣ Uses existing reporting
‣ In your hands becomes a barometer of user satisfaction, requests, &
complaints
‣ Easily shows you
‣ What folks want from the system (search results)
‣ What folks need and are not getting (no search results)
‣ The vocabulary folks use to describe things
‣ Great for pointing out opportunities for business & technology
‣ Easy but time consuming

30. Step-by-Step Search Analysis
1. Find the owner of analytics and ask for advice
2. Determine what’s available to you: search results, no search results, etc.
3. Use spreadsheet to filter and order results in order to highlight patterns
4. Identify: like terms & misspellings, adjectives, frequent terms both returned &
not
5. Create summary
6. Discuss summary with business to get contextual detail
7. Create recommendations brief: changes to data, process, & functionality

31. Tips for Search Analysis
‣ Know the data: what do the numbers really mean?
‣ Know the system
‣ Keep your interpretations simple
‣ Plan extra time for analysis - it’s time consuming
‣ When delivering results, create 2 views: overview & detail
‣ Pass results to teams that can benefit: business, search, technology,
marketing
‣ Works best when used with other methods / techniques

32. Using multiple methods will increase the
impact of validation testing.

33. Discussion

34. Resources

35. Recruitment
Ethnio: http://ethnio.com
Sample Size Calculator
Creative Research Systems Calculator: http://www.surveysystem.com/sscalc.htm
Online Card Sorting Tools
The Mind Canvas: http://themindcanvas.com
OptimalSort: http://optimalsort.com
C-Inspector: http://www.c-inspector.com
Socratic CardSortsm: http://www.sotech.com/
WebCat (NIST): http://zing.ncsl.nist.gov/WebTools/WebCAT/overview.html
WebSort: http://websort.net/
STC’s Usability & User Experience List of Card Sorting Tools:
http://www.stcsig.org/usability/topics/cardsorting.html
Others?
Validation Tools

36. The Delphi Method for Card Sorting
“A Modified Delphi Approach to a New Card Sorting Methodology” by Celeste
Lyn Paul:
http://www.upassoc.org/upa_publications/jus/2008november/paul1.html
“A Delphi Approach to Card Sorting” by Celeste Lyn Paul (PDF):
http://iasummit.org/proceedings/2007/files/Delphi_Card_Sorting.pdf
For Your Reading Pleasure

37. Taxonomy Validation
Pre-conference Workshop
Joseph Busch of Taxonomy Strategies & Dave Cooksey of saturdave
Gilbane Conference Boston
Content, Collaboration & Customers
Tuesday, December 1, 2009
http://gilbaneboston.com/index.html

38. That’s All, Folks!
Dave Cooksey
saturdave UX Consulting
713 Pine Street 1R
Philadelphia, PA 19106
Twitter: @saturdave
dave@saturdave.com
© 2009 saturdave