1. Fitness for Use: Present or Future?
Insights on U3lity as a Primary Dimension of
Quality in e‐Learning
IASK - Teaching and Learning 2009 Conference – Porto, December 9, 2009
Rosário Cação
António Dias de Figueiredo
1

2. Agenda
1. Research Ques3ons
2. Case Study
3. Conclusions
2

3. 1. RESEARCH QUESTIONS
3

4. 1. Research Ques;ons
Starting points:
• In educational contexts, quality is related to the ability to
satisfy assessed needs.
• Those needs reflect ends-in-view (Dewey, 1939): expected
changes or uses.
• Quality is fitness for use (Juran, 1951).
• Utility reactions can be used as an indicator of the
possibility of use (Ruona, Leimbach, Holton, & Bates,
2002).
Research questions:
• Are perceptions of utility relevant dimensions of
quality in e-learning courses?
• Are future uses more valued than immediate uses?
4

5. 2. CASE STUDY
5

6. 2. Case Study
www.evolui.com
A Portuguese provider of asynchronous e-learning for
professional training, with ten years of experience in the
consumer e-learning market.
• Has 50.000 clients from 26 countries.
• Offers more than 160 short-term courses.
• The courses take from 1 to 9 weeks to be completed.
The subjects of the study were participants in an online
training course that they had paid.
6

7. 2. Case Study
Figure: Three ways of mixing quan3ta3ve and qualita3ve data
7
Source: Creswell & Clark, 2007, p. 7

8. 2. Case Study
Qualita3ve study Quan3ta3ve studies
• Respondents: training professionals
• Pedagogical courses
• Researcher as trainer
• Participant observation and interviewing
• August 2007 – June 2008
• Collected: 2398 messages posted by 210 trainees
• Used: 104 messages from 44 customers – the
theoretical saturation point (Glaser & Strauss, 1967)
8

9. 2. Case Study
• Learning transfer
• Increase of performance
• Return of investment
• Behavior changes
• Future utility
• Practical uses
• Professional utility
• Motivation for future
learning
• Use deadline
• Motivation
• Trainees
• Knowledge
• Transfer of knowledge
• Organizational impact
Node to be explored: U3lity
• Applicability
• Use Short‐term vs. long‐term u3lity
• Changes in behavior
9

10. 2. Case Study
Qualita3ve study Quan3ta3ve studies
• Satisfaction survey
(in the end of the course)
• Single-item survey with 15 variables in a 1-10
Likert scale
• March 2008 – February 2009
• 2741 answers
• 145 courses
• 1085 unique customers
• Reliability (internal consistency) - Cronbach’s
alpha: .963 (very good)
10

11. 2. Case Study
U;li;es Future vs. immediate Types of courses Gender differences
Future utility is higher than immediate utility.
Quality
Future:
Mo;va;on Mean: 8,39
Median: 9
Special case
Immediate:
Mean: 8,2
Median: 9
Wilcoxon Test on the Equality of the Means of the Utilities
Ho: Immediate utility is equal to future utility
H1: immediate utility is different from future utility
With a 95% confidence future utility is perceived as higher than
immediate utility (p value = 0). 11

12. 2. Case Study
U;li;es Future vs. immediate Types of courses Gender differences
Quality
Regular vocational
Certification courses
e-learning courses
Mo;va;on
Special case • Both utilities were high in both kinds of
courses.
• Future utility is higher in certification courses
but immediate utility is lower (Mann-Whitney
test).
12

13. 2. Case Study
U;li;es Future vs. immediate Types of courses Gender differences
• Women have higher perceptions of both utilities
Quality • Future utility is higher than immediate utility for both
groups
• The difference between utilities is wider among male users
Mo;va;on
Special case Women
Men
p value = 0
13

14. 2. Case Study
U;li;es General percep;ons Regression Factor analysis
• Quality perceptions have a positive Kurtose, a
Quality negative skewness and an average of 8.35 out of
10.
Mo;va;on • Female users have a higher perception of quality
but an equal perception of quality-price relation.
Special case • Quality is highly correlated with global
satisfaction (.8), immediate utility (.67), and
future utility (.86).
These variables must be relevant to explain quality,
even in a linear regression model.
14

15. 2. Case Study
U;li;es General percep;ons Regression Factor analysis
stepwise method, R2 = .85
Quality Y = a + b1 X1 + b2 X2 + b3 X3 + b4 X4 + b5 X5 + b6 X6 + b7 X7 +
b8 X8 + b9 X9 + b10 X10 + b11 X11
Mo;va;on Y = Global quality
X1 = Global satisfaction
X2 = Future utility
Special case X3 = Training contents
X4 = Quality-price relation
X5 = Competence, kindness and promptness of the staff
X6 = Fulfillment of expectations
X7 = The trainer's expertise
X8 = The platform and its functions
X9 = Initial motivation
Excluded variables:
X10 = Final motivation • Fulfillment of training objectives
X11 = Immediate utility • The contribution of the forum for
the learning process
• The dynamics and help provided
by the tutor 15

16. 2. Case Study
U;li;es General percep;ons Regression Factor analysis
Y = .3 + .125 X1 + .169 X2 + .152 X3 + .133 X4 + .83 X5 + .122
Quality X6 + .7 X7 + .057 X8 - .045 X9 + .069 X10 + .044 X11
Y = Global quality
X1 = Global satisfaction
Mo;va;on • Alone, satisfaction explains X2 = Future utility
X3 = Training contents
71.6% of the variability of quality X4 = Quality-price relation
X5 = Competence, kindness and
Special case promptness of the staff
• Future utility is the second X6 = Fulfillment of expectations
most important contributor X7 = The trainer's expertise
X8 = The platform and its functions
X9 = Initial motivation
X10 = Final motivation
• Immediate utility and final X11 = Immediate utility
motivation provide low
contributions to explain quality.
• Quality is a long-term attitude
16

17. 2. Case Study
U;li;es General percep;ons Regression Factor analysis
Quality factors:
Quality
Mo;va;on
Special case
17

18. 2. Case Study
U;li;es General interest Course dura;on
Does quality depends on who pays the bill?
Quality (Mann-Whitney tests)
• All variables have higher ratings when the
Mo;va;on
course is paid by the trainee.
Special case • Initial motivation has a wider distribution when
the course is paid by the company where the
trainee works.
• When the trainee does not pay anything to attend
the course, she has a lower perception on quality-
price relation, most probably due to a lower
quality perception.
18

19. 2. Case Study
U;li;es General interest Course dura;on
Why Look Deeper into Motivation?
Quality
If a person cannot foresee the consequences of his act (…),
it is impossible for him to guide his act intelligently.
(Dewey, 1916, p. 24)
Mo;va;on
• Initial motivation helps to explain quality (regression).
Special case • Both initial and final motivation are considered in
the attitude factor (factor analysis).
• Correlation initial-final motivation: .5
• Final motivation helps to explain global satisfaction
(fulfillment of expectations is the most contributor)
(R2 = .873).
19

20. 2. Case Study
U;li;es General interest Course dura;on
The evolution of final motivation
Quality
The final motivation of courses which length is two weeks is
Mo;va;on different from the final motivation of shorter and longer
courses.
Special case
Final motivation is optimized with two week courses.
Those courses also maximize global satisfaction, immediate,
and future utility … and quality perceptions.
20

21. 2. Case Study
The special case of the courses in initial
U;li;es
certification of trainers
Promoter: EVOLUI.COM
Quality p-value Decision Conclusion
Factor 1: Training Process
8. The trainer’s expertise .137 Not Reject May be equal
Mo;va;on 9. The contribution of the forum to the learning process .214 Not Reject May be equal
10. The dynamics and help provided by the trainer in the .103 Not Reject May be equal
forum
11. Competence, kindness and promptness of the staff .000 Reject Are different
Special case
6. Platform and its functions .774 Not Reject May be equal
Factor 2: Training Attitudes
1. Global satisfaction .001 Reject Are different
2. Expectations fulfilment .000 Reject Are different
4. Final motivation .000 Reject Are different
5. Fulfilment of training objectives .041 Reject Are different
Factor 3: Training Utility
12. Immediate utility .000 Reject Are different
13. Future utility .000 Reject Are different
21

22. CONCLUSIONS
22

23. Conclusions
• Future use is the second most important dimension of
quality, following sa3sfac3on.
– Quality can be defined as fitness for future use.
• Whenever training reac3ons are studied, immediate and
future u3lity percep3on and ini3al and final mo3va3ons
should also be addressed, besides affec3ve reac3ons.
• The dura3on of e‐learning‐based professional training
courses is an important factor to be taken into account.
– Two week courses op3mize the percep3ons of quality and final
mo3va3on, in the case we studied.
23

24. Slides available at www.slideshare.net/rosariocacao
Fitness for Use: Present or Future?
Insights on U3lity as a Primary Dimension of
Quality in e‐Learning
Teaching and Learning 2009 – Oporto, December 9, 2009
Rosário Cação (mrac@dei.uc.pt)
António Dias de Figueiredo (adf@dei.uc.pt)
24