1. Defining and Evaluating Usability Criteria for
Content Management Systems
Saurabh Kudesia
Customer Documentation Manager
Nokia Siemens Networks, Bangalore
August 2011
1 © Nokia Siemens Networks 2011 Presented at STC India Usability Conference 2011, Bangalore, India

2. About the Speaker
• Customer Documentation Manager, Nokia Siemens Networks, Bangalore
• Co-founder, former editor-in-chief, KnowGenesis International Journal for Technical
Communication (IJTC)
• Former associate editor of Directives, a newsletter published by the (STC) Management SIG
• Reviewer: ISO/IEC 25612 standards for International Standards Organization (ISO)
• Bachelor of Electronics and a Certified Scrum Master (CSM)
• Alumnus of Symbiosis Institute (Pune) and IIM-B
• Part of Author-it DevHub Community
2 © Nokia Siemens Networks 2011

3. In this Session
Generic Content
Management Usability
Usability Testing
Framework (CMF) Measurement
Tactics for CMS
and Usability Methodology
Perspectives
3 © Nokia Siemens Networks 2011

4. Content Management is Universal
Micro Macro
The information viewpoint applies at different levels
4 © Nokia Siemens Networks 2011

5. Content Management Perspectives
Distributes Business Value Business Perspective
Balances Organizational Forces Organizational Perspective
Content Combines Content-related disciplines Content Perspective
Collects/Manages/Publishes/Distributes Information Process Perspective
Technical Infrastructure Technical Perspective
5 © Nokia Siemens Networks 2011

6. Content Management Framework (CMF) Content/Process View
Collection System Management System Publication System
Print Syndication
Electronic Web
Impacts publication Impacts component Impacts access structure
designs designs designs
6 © Nokia Siemens Networks 2011

7. Content Management Framework (CMF) Technical View
Access Interface Workflow Auditing
Formats/Standards
Evaluation Analysis
Quality Format Conversion
Assurance Audit Data
Access Attributes
Version Tracking
R&D Relational Links Implementation Design Usage/Performance Statistics
Auditors
Tools Development
Issue Categorization/Tracking
Automatic Advanced
Support Teams Notifications Search
Security
Customers Feedback/Issue Reporting
Configuration and Administration
7 © Nokia Siemens Networks 2011

8. Usability Mea Culpa (or “That’s Not Architectural”)
“Making a system’s user interface clear and easy to use is primarily a matter of getting
the details of a user’s interaction correct…but these details are not architectural”
- Bass, Clements, and Kazman [Software Architecture in Practice, first edition]
8 © Nokia Siemens Networks 2011

9. The ‘Planes’ of User Experience
Surface brings everything together visually: What will the finished
Surface product look like?
Skeleton makes structure concrete: What components will enable
Skeleton people to use this site?
Structure gives shape to scope: How will the pieces of the system fit
Structure together and behave?
Scope transforms strategy into requirements: What features will the
Scope system need to include?
Strategy is where it all begins: What do we want to get our of the
Strategy system? What do our users want?
9 © Nokia Siemens Networks 2011

10. The ‘Planes’ of User Experience
Discrete ‘Planes’ of User Experience Concurrent ‘Planes’ of User Experience
Requiring work on each plane to finish before work on the next can start A better approach is to have work on each plane finish before work on the
leads to unsatisfactory results for the designers and the users. next can be finish.
Effort
Effort
Strategy Scope Structure Skeleton Surface Strategy Scope Structure Skeleton Surface
Time Time
10 © Nokia Siemens Networks 2011

11. Usability Tactics
Runtime Tactics Design time Tactics
Separate User Support User Support System
Localize changes to User Interface
Interface Initiative Initiative
code
Cancel User Model
Undo System Model
Aggregate Task Model
Runtime Tactics Design time Tactics
• Maintain a model of the task • Localize changes to user interface code
• Maintain a model of the system • Use supported software architecture patterns like Model-View-Controller
• Maintain a model of the user (MVC), Presentation-Abstraction-Control (PAC)
11 © Nokia Siemens Networks 2011

12. Quality Attributes Stimuli
Availability Modifiability Performance Security
Unexpected Event Functionality Periodic Display Services
Non occurrence of expected event Platform (Portability) Stochastic Modify Services
Quality attribute Sporadic Modify Information
Capacity (Scalability) Modify Access
Reduce availability to system services
12 © Nokia Siemens Networks 2011

13. Usability General Scenarios
Stimulus Response
Artifact
Source Environment Response Measure
Source Stimulus Artifact Response Response Measure
End Users Learn system features System Support “learn system features” Task Time
Use system efficiently Support “use system efficiently Number of errors
Minimize error impact Minimize error impact Number of problems solved
Adapt system Environment Adapt system User satisfaction
Feel comfortable Feel comfortable Gain of user knowledge
Run time Proportion of Successful operation
Configure time Amount of time/data lost
13 © Nokia Siemens Networks 2011

14. Usability Measurement Methodology
Identification of Measurement and Result analysis and
Dimensional scale, Selecting the Validation of Scale estimation of
Sub constructs and Measurement model based on usability
item pool measurement model
14 © Nokia Siemens Networks 2011

15. Usability Measurement is Fuzzy
15 © Nokia Siemens Networks 2011

16. Dimensional Scales, Sub Constructs and item pool
Simplicity Familiarity Consistency Visual Feedback Responsiveness Fault Tolerance Scalability
1. Visual Efficiency and Consistency 3. Interface-Task Performance
Consistent Interface Design labeling Simple structure to accomplish a task
Consistent Interface Response UI screens don’t include too many elements
Screen design inline with visual design principles
Understandable Interface language
4. Interface Familiarity
Interface resemble user’s prior experience
6. Efficient/Flexible Access to Content
Interface behaves similar to what user is accustomed to
Easy interface design for accessing content
No specialized training required to perform functions
Shortcuts to access frequently used features 5. Interface Operation Performance
Multiple options to perform the same action
CMS loads fast Prompt CMS responses to users’ prompts
CMS has a flexible use Interface design changes are localized
Less user interaction required to complete a task
7. Efficiency of Navigation 2. Error Handling and Functional Efficiency
Easy access to critical navigational buttons Easy to understand error messages
Navigation remains unaffected with increase in content volume Explicit error messages
No problems with CMS functions
16 © Nokia Siemens Networks 2011

17. Measurement Models
Model 1: Strictly Uni- Model 2: Group-Factor Model 3: Essentially Model 4: Multidimensional
dimensional Model Unidimensional Structural Model
E1 X1
E2 X2
E3 X3 C
: :
Ek Xk
17 © Nokia Siemens Networks 2011

18. Fit Indices of Models
Covariance Matrix
Sub 1 Sub 2 Sub 3 Sub 4 Sub 5 Sub 6 Sub 7 0.57 Sub 1
Sub 1 1.00 0.66
Sub 2 0.59 1.00
0.17 Sub 2
0.91
Sub 3 0.01 -0.01 1.00
0.57 1.00
Sub 4 0.37 0.52 -0.01 1.00 0.68 Sub 4
Upper 1
Sub 5 0.01 -0.01 0.64 -0.01 1.00 0.93
Sub 6 0.61 0.84 -0.01 0.53 -0.01 1.00 0.14 Sub 6 0.66
Sub 7 0.43 0.60 -0.01 0.38 -0.01 0.62 1.00
-0.02
Upper 1 (ζ) 0.66 0.91 -0.02 0.57 -0.01 0.93 0.66
0.56 Sub 7
Goodness-of-fitness Lack-of-fitness
Upper 2 (ζ) -0.01 -0.02 0.86 -0.01 0.74 -0.02 -0.01
Models NNFI CFI RMSEA SRMR
0.25 0.86
Model 1 0.85 0.86 0.134 0.121 Sub 3
Model 2 0.92 0.93 0.075 0.073 Upper 2
0.74
1.00
Model 3 0.92 0.92 0.078 0.075 Sub 5
0.46
Model 4 0.92 0.93 0.075 0.062
18 © Nokia Siemens Networks 2011

19. Important Guidelines
• Don’t mix content presentation and architectural design; both are perceived as separate
constructs by participants
• Usability scale is multidimensional and should be treated accordingly in benchmarking
usability of CMS
• Consider context dependencies of User’s perceptions and functional use of CMS into usability
calculation
• Generalizability and transferability of ideas across contexts, extensive collaboration is
essential for the multidimensionality of usability scales
19 © Nokia Siemens Networks 2011

20. For further discussion:
Phone: +91-9620208216
Email: saurabhkudesia@gmail.com
20 © Nokia Siemens Networks 2011