lecture 7

1. 7.
Measurement &
Questionnaires
Design
Dr. Nguyen Quynh Mai

2. Measurement
Give examples:
• What are some things
that are easy to
measure?
• What are some things
that are difficult to
measure?

3. http://www.mountwashington.org

4. Measurement
• Quantitative measurement involves assigning numbers to
attributes or constructs in order to create variables so that
we can better understand the relationships among
variables
• With qualitative inquiry involves assigning words to attributes
or concepts in order to draw out themes related to human
perceptions and meaning

5. Measurement
• If our studies do not allow us to measure variation in the
dependent variable (Y) as related to variation in our X
variables, then we cannot do any scientific testing.
1. We measure whether certain variables are meaningful –
individually significant.
2. We measure the variation in our variables.
3. We also measure the significance and explanatory power of our
models and the relationships between variables.
4. If it can be quantified, then you should do so.

6. Qualities of Variables
• Exhaustive -- Should include all possible answerable responses.
(Schooling: No Schooling, Elementary, Middle, HS, College)
• Mutually exclusive -- No respondent should be able to have two
attributes simultaneously (e.g. Female Male ).

7. What Is Level of Measurement?
The relationship of the values that are assigned to the
attributes for a variable
1 2 3
Relationship
Values
Attributes
Variable
Low Medium High
Development

8. 8
Types of Scales
Ordinal
interval
Ratio
Nominal

9. 9
Levels of Measurement
Ordinal
interval
Ratio
Nominal
Classification
Order
Classification
Order
Classification Distance
Natural Origin
Order
Classification Distance

10. 10
Nominal Scales
• Nominal or categorical data is data that comprises of categories
that cannot be rank ordered – each category is just different
• Mutually exclusive and collectively exhaustive categories
• Exhibits the classification characteristic only. Nominal data reflect
qualitative differences rather than quantitative ones.
• Variables that have only two responses i.e. Yes or No, are known as
dichotomies
• Examples: What is your gender?
(please tick)
Male
Female
Did you enjoy the film?
(please tick)
Yes
No

11. 11
Ordinal Scales
• Characteristics of nominal scale plus
an indication of order
• Implies statement of greater than
and less than
• Ex: Doneness of meat (well, medium
well, medium rare, rare)

12. • Both interval and ratio data are examples of scale data.
• Scale data:
 data is in numeric format (£50, £100, £150)
 data that can be measured on a continuous scale
 the distance between each can be observed and as a result
measured
 the data can be placed in rank order.
Interval and ratio data

13. 13
Interval Scales
• Interval data measured on a continuous scale and has
no true zero point
• Characteristics of nominal and ordinal scales plus the
concept of equality of interval.
• Equal distance exists between numbers
• Examples:
• Time – moves along a continuous measure or seconds,
minutes and so on and is without a zero point of time.
• Temperature – moves along a continuous measure of
degrees and is without a true zero.

14. 14
Ratio Scales
• Characteristics of previous
scales plus an absolute zero
point
• Examples
– Weight
– Height
– Number of children
– Age

15. Measurement Hierarchy
NOMINAL
ORDINAL
INTERVAL
RATIO
WEAKEST
STRONGEST

16. Sources of Error
Respondent
InstrumentMeasurer
Situation

17. Evaluating Measurement Tools
Criteria
Validity
Practicality Reliability

18. 18
Reliability & Validity

19. 19
Validity Determinants
Content
ConstructCriterion

20. 20
Increasing Content Validity
Content
Literature
Search
Expert
Interviews
Group
Interviews
Question
Database
Etc.

21. 21
Increasing Construct Validity
New measure of trust
Known measure of trust
Empathy
Credibility

22. 22
Judging Criterion Validity
Relevance
Freedom from bias
Reliability
Availability
Criterion

23. 23
Reliability Estimates
Stability
Internal
Consistency
Equivalence

24. Reliability
• Reliability is defined as the extent to which a questionnaire, test, observation
or any measurement procedure produces the same results on repeated trials
• Equivalence refers to the amount of agreement between two or more
instruments that are administered at nearly the same point in time.
Equivalence is measured through a parallel forms procedure in which one
administers alternative forms of the same measure to either the same group
or different group of respondents.
• Stability occurs when the same or similar scores are obtained with repeated
testing with the same group of respondents. Stability is assessed through a
test-retest procedure that involves administering the same measurement
instrument to the same individuals under the same conditions after some
period of time.
• Internal consistency concerns the extent to which items on the test or
instrument are measuring the same thing. If the individual items are highly
correlated with each other you can be highly confident in the reliability of the
entire scale
(Miler, Western International University)
24

25. Equivalence test
• Reliability is established by using similar/alternate forms (Forms A
& B) that measure the same trait/knowledge.
• Two forms are created by splitting the questions on the test
randomly before administration of the forms. One group of
students gets form "A" first, then "B". Another group takes the "B"
form of the test first, then the "A" version. The scores on both are
then correlated, producing a correlation or reliability coefficient.
25

26. Stability test: Test-Retest
• If you use a test with a student in the morning and then
administer it again in the afternoon, would you expect
about the same results?
• Which would have the highest test-retest reliability
coefficient (correlation)?
– retesting a youngster on an achievement test within the
same week
– retesting in different years
26

27. Internal consistency
• The internal consistency reliability of survey instruments is a measure of
reliability of different survey items intended to measure the same
characteristic.
• For example, there are 5 different questions (items) related to anxiety
level. Each question implies a response with 5 possible values on a Likert
scale (1 – 5). Responses from a group of respondents have been
obtained. In reality, answers to different questions vary for each
particular respondent, although the items are intended to measure the
same aspect or quantity. The stronger the correlation, the greater the
internal consistency reliability of this survey instrument.
• In statistic, they use Cronbach alpha to measure reliability
27

28. 28
Practicality
Economy InterpretabilityConvenience

29. MEASUREMENT
SCALES

30. 30
Nature of Attitudes: example of measurement
Cognitive
I think oatmeal is healthier
than corn flakes for breakfast.
Affective/
attitude
Behavioral
I hate corn flakes.
I intend to eat more oatmeal
for breakfast.

31. 31
Improving Predictability
Reference
groups
Multiple
measures
Factors
Strong
Specific
Basis
Direct

32. 32
Selecting a Measurement Scale
Research objectives Response types
Data properties
Number of
dimensions
Forced or unforced
choices
Balanced or
unbalanced
Rater errors
Number of
scale points

33. 33
Response Types
Rating scale
Ranking scale
Categorization
Sorting
Number of Dimensions
Unidimensional
Multi-dimensional

34. Balanced or Unbalanced
Very bad
Bad
Neither good nor
bad
Good
Very good
Poor
Fair
Good
Very good
Excellent
How good an actress is Angelina Jolie?

35. Forced or Unforced Choices
Very bad
Bad
Neither good nor
bad
Good
Very good
Very bad
Bad
Neither good nor bad
Good
Very good
No opinion
Don’t know
How good an actress is Angelina Jolie?

36. Number of Scale Points
Very bad
Bad
Neither good nor bad
Good
Very good
Very bad
Somewhat bad
A little bad
Neither good nor bad
A little good
Somewhat good
Very good
How good an actress is Angelina Jolie?

37. Simple Category Scale
I plan to purchase a Samsung laptop in the
12 months.
 Yes
 No

38. Multiple-Choice, Single-Response
Scale
What newspaper do you read
most often for financial news?
 East City Gazette
 West City Tribune
 Regional newspaper
 National newspaper
 Other (specify:_____________)

39. Multiple-Choice, Multiple-Response
Scale
What sources did you use when designing your new
home? Please check all that apply.
 Online planning services
 Magazines
 Independent contractor/builder
 Designer
 Architect
 Other (specify:_____________)

40. Likert Scale
The Internet is superior to traditional libraries for
comprehensive searches.
 Strongly disagree
 Disagree
 Neither agree nor disagree
 Agree
 Strongly agree

41. Semantic Differential

42. SD Scale for Analyzing Actor Candidates

43. Numerical Scale

44. Multiple Rating List Scales

45. Ranking Scales
• Paired-comparison scale
• Forced ranking scale
• Comparative scale

46. Paired-Comparison Scale

47. Forced Ranking Scale

48. Questionnaires
Design
Dr. Nguyen Quynh Mai

49. Overall Flowchart for Instrument Design

50. Flowchart for Instrument Design: Phase 1

51. Strategic Concerns in Instrument Design
What type of scale is needed?
What communication approach will be used?
Should the questions be structured?
Should the questioning be disguised?

52. Flowchart for Instrument Design: Phase 2

53. Question Categories and Structure
Administrative
Classification
Target

54. Question Content
Should this question be asked?
Is the question of proper scope and coverage?
Can the participant adequately
answer this question as asked?
Will the participant willingly
answer this question as asked?

55. Question Wording
Criteria
Shared
vocabulary Single
meaning
Misleading
assumptions
Adequate
alternatives
Personalized
Biased

56. Response Strategy
Factors
Objectives
of the study
Participant’s
level of
information
Degree to which
participants have
thought through topic
Ease and clarity with
which participant
communicates
Participant’s
motivation to
share

57. Free-Response Strategy
What factors influenced your enrollment in Metro U?
____________________________________________
____________________________________________

58. Dichotomous Response Strategy
Did you attend the “A Day at College”
program at IU?
 Yes
 No

59. Multiple Choice Response Strategy
Which one of the following factors was most influential
in your decision to attend Metro U?
 Good academic standing
 Specific program of study desired
 Enjoyable campus life
 Many friends from home
 High quality of faculty

60. Checklist Response Strategy
Which of the following factors influenced your
decision to enroll in Metro U? (Check all that
apply.)
 Tuition cost
 Specific program of study desired
 Parents’ preferences
 Opinion of brother or sister
 Many friends from home attend
 High quality of faculty

61. Rating Response Strategy
Strongly
influential
Somewhat
influential
Not at all
influential
Good academic reputation   
Enjoyable campus life   
Many friends   
High quality faculty   
Semester calendar   

62. Ranking
Please rank-order your top three factors from the following
list based on their influence in encouraging you to apply to
Metro U. Use 1 to indicate the most encouraging factor, 2
the next most encouraging factor, etc.
_____ Opportunity to play collegiate sports
_____ Closeness to home
_____ Enjoyable campus life
_____ Good academic reputation
_____ High quality of faculty

63. Summary of Scale Types
Type Restrictions Scale
Items
Scale
points
Data Type
Rating Scales
Simple Category
Scale
Needs mutually exclusive choices One or
more
2 Nominal
Multiple Choice
Single-Response
Scale
Needs mutually exclusive choices;
may use exhaustive list or ‘other’
many 2 Nominal
Multiple Choice
Multiple-Response
Scale (checklist)
Needs mutually exclusive choices;
needs exhaustive list or ‘other’
many 2 Nominal
Likert Scale Needs definitive positive or
negative statements with which to
agree/disagree
One or
more
5 Ordinal
Likert-type Scale Needs definitive positive or
negative statements with which to
agree/disagree
One or
more
7 or 9 Ordinal

64. Summary of Scale Types
Type Restrictions Scale
Items
Scale
points
Data Type
Semantic
Differential
Scale
Needs words that are opposites to
anchor the graphic space.
One or
more
7 Ordinal
Numerical
Scale
Needs concepts with standardized or
defined meanings; needs numbers
anchor the end-points or points along
the scale; score is a measurement of
graphical space from one anchor.
One or
many
3-10 Ordinal or
Interval
Multiple Rating
List Scale
Needs words that are opposites to
anchor the end-points on the verbal
scale
Up to 10 5-7 Ordinal
Fixed Sum
Scale
Participant needs ability to calculate
total to some fixed number, often 100.
Two or
more
none Interval or
Ratio
Stapel Scale Needs verbal labels that are
operationally defined or standard.
One or
more
10 Ordinal or
Interval

65. Summary of Scale Types
Type Restrictions Scale
Items
Scale
points
Data Type
Ranking Scales
Graphic Rating
Scale
Needs visual images that can be
interpreted as positive or negative
anchors; score is a measurement of
graphical space from one anchor.
One or
more
none Ordinal
(Interval, or
Ratio)
Paired
Comparison
Scale
Number is controlled by participant’s
stamina and interest.
Up to 10 2 Ordinal
Forced
Ranking Scale
Needs mutually exclusive choices. Up to 10 many Ordinal or
Interval
Comparative
Scale
Can use verbal or graphical scale. Up to 10 Ordinal

66. Internet Survey Scale Options

67. Internet Survey Scale Options

68. Internet Survey Scale Options

69. Flowchart for Instrument Design: Phase 3

70. Guidelines for Question Sequencing
Interesting topics early
Classification questions later
Sensitive questions later
Simple items early
Transition between topics
Reference changes limited

71. Components of Questionnaires
Introduction
Transition
Instructions for ….
Conclusion
a. Terminating
b. Participant discontinue
c. Skip
d. Disposition instruction

72. Tips for a good questionnaires
1. Keep it short and simple
2. Start with a brief introduction explaining the purpose of the
questionnaire
3. Ask yourself what you will do with the information from each
question
4. Put easier questions first
5. Ask just one thing in a question
6. Leave difficult or sensitive questions towards the end
7. Try to be exhaustive when offering answer choices
8. Avoid bias in language

73. Tips for a good questionnaires (Cont)
9. Avoid jargon (special words that are used by a particular
profession group and are difficult for others to understand)
10. Do not use "emotional language" or leading questions
11. Present Disagree/Agree choices in that order (disagree to agree);
Same with Excellent to Poor, and Positive to Negative
12. User higher numbers to signify a more positive answer in rating
scales
13. Be consistent with your layout
14. Separate related questions
15. Have Don’t Know or Not Applicable as an option for most
questions

74. Tips for a good questionnaires (Cont)
16. Have Other or None for questions with a list of options
17. Include Other Comments at the end
18. Keep the number of open-ended questions to a minimum
19. Make sure questions are relevance with scale
20. Pilot test the questionnaire