The document provides information on designing questionnaires and surveys. It discusses determining research questions, different types of surveys, principles of good survey design, question formatting and layout, advantages and disadvantages of questionnaires, improving response rates, question types, designing effective questions, and data analysis. Key points covered include framing the research question, descriptive vs. analytic vs. exploratory vs. confirmatory surveys, keeping surveys concise, credible, and minimizing burden on respondents.
1. Research Methodology Workshop 22 May 2013
Hampshire Teaching School Alliances
Wan Ching Yee
Questionnaire design & analysis
2. The research question frames the rest
• … the concepts you are researching
• … the relevant research literature
• … design considerations
• … samples
• … methodologies
• … the conclusions that you can draw!
3. Types of survey
• Descriptive – produce relevant data
• Analytic – examine relationships
• Exploratory – patterns explored without prior theory
• Confirmatory – model or causal relationships are
tested
4. What are the principles of good survey
design?
• Realistic and feasible
• KISS
• Select only those variables that are most likely to
affect outcomes
• Minimise cost and effort to respondents
• Credibility is a factor
5. 5
Presentation & Layout:
• Clear presentation e.g use of fonts
• Open questions used as little as possible
• Clear instructions about how to respond e.g. give
examples
• Vertical or horizontal closed answers?
• Keep question and answers together
6. 6
Why use questionnaires?
• Cheap to administer
• Quick to administer
• Absence of interviewer effects
• No interviewer variability
• Convenience for respondents
7. 7
Disadvantages of questionnaires
• Cannot prompt
• Cannot probe
• Cannot ask too many questions that are not salient
to respondents
• Do not know who answers
• Not appropriate for some kinds of respondents
• Greater risk of missing data
• Lower response rates
8. 8
Improving response rates
• Good (personalising?) covering letter
• Accompanied by SAE if postal questionnaire
• Follow up non-respondents
• Shorter questionnaires
• Clear instructions and attractive layout
• Avoid „bulky‟ questionnaire
• Begin with questions which are likely to be of
interest to the respondent
• Incentives
9. 9
Types of questions
• Personal factual questions
• Factual questions about others
• Questions about attitudes
• Questions about beliefs and values
• Questions about knowledge
• Branching question for more detailed questions
10. 10
General rules for designing questions
• Always bear in mind your research questions
(these are at a different level to your questions on
your questionnaire!)
• What do you want to know?
• How would you answer it?
• Aim for clarity, simplicity & neutrality
11. 11
Improve the quality of your questions
Try to avoid:
• Long complex questions
• Double negatives
• Jargon
• Abbreviations
• Culture specific terms
• Words with double meaning
• Leading questions
• Emotionally loaded words
12. 12
Improve the quality of your questions
May want to consider:
Including a response category:
• Don't know
• Not applicable
• Do not want to answer
13. 13
Data gathering
Question types: Closed and open ended text or
open-ended numeric
• Closed: Yes/No, Male/Female
+ Easy to convert to numerical format if using
statistical packages
- Limited choices
• Open ended: What is your major source of
anxiety at the moment?
+ Respondents have more freedom to respond in
their own way
- Responses need to be summarised (more time)
15. 15
Formatting
• Group together questions on major subject areas
• Begin with emotionally neutral questions which
ideally, are easy and pleasant to answer
• More sensitive or difficult questions in the middle
• Personal characteristics eg income, at the end
16. 16
Piloting and pre-testing questions
• Open questions used to generate fixed-choice
answers
• Identify questions that don‟t work e.g skipped,
unclear
• Identify clarity of instructions
• Use questions employed by other researchers?
• http://www.natcen.ac.uk
• http://surveynet.ac.uk/sqb/surveys/introduction.asp
17. 17
Steps in questionnaire design
• Make a list of variables
• Assemble file of questions or instruments for
measuring each variable e.g available in books, on
web
• Compose a draft
• Shorten the set of instruments
• Pretest
• Validate
• Consider your sample – how will you access them?
And how will you recruit them?
18. 18
Different types of data
Nominal or categorical
Examples Sex, preference type of coffee, colour
Characteristics No sense of order
Can be stored as a word, text or numerical
code
Summarise Frequency or percentage
Cannot use Mean or average value
Display Pie chart
Column chart
Bar chart
19. 19
Different types of data
Ordinal
Examples Satisfaction, rank, fanciness
Characteristics Have a meaningful order but the intervals
between the values in the scale may not
be equal
Summarise Frequency or percentage
Cannot use Mean or average value?
Display NOT as a pie chart
Column chart
Bar chart
20. 20
Different types of data
Interval/Ratio
Examples Number of customers, weight, age, size
Characteristics Includes things that can be measured
rather than classified or ordered
Data can be discrete whole numbers (5
customers) or continuous with fractional
numbers (1.2 miles)
Summarise Mean
Median
Standard deviation
Display Histogram
Bar chart
Line chart (data that occur over time)
21. 21
Data gathering and processing
• Data gathering –
Quantification/measurement
• Data modification –grouping/categorising,
ordering, taxonomies
Displaying – looking for trends
Comparing – with each other, with the
population
Relationships between variables
Use of statistics
23. 23
Grouping
Sex 1=Males
2=Females
Age 1= 21-25 years
2= 26-30 years
3= 31-35 years
4= above 36 years
5= No answer given
Optimising scale (items 1-4) 1 (strongly agree)
2 (agree)
3 (disagree)
4 (strongly disagree)
Closed questions
24. Grouping
Open ended questions
What is your major source of anxiety at the moment?
Scan through questionnaires and try to find common
themes.
Anxiety 1=Grade
2=Pass the course
3=Lack of time
25. 25
Displaying data
• Basic diagrams can be created with word processing
programmes such as Windows
• Most useful diagrams are typically tables, pie charts,
bar charts, trend lines
• Diagrams are best if simple
• Diagrams do not tell you about the importance of the
size of the quantities involved
31. 31
Analysis
• Hypotheses may require comparisons, e.g. difference
between groups, difference between treatments,
relation between scores
• There is always some variation
• Possibility of measurement error, sample error, etc
• Hence we use statistics for an assessment of how
likely it is that the difference/relation found is not
worth thinking about
32. 32
Probability
• Statistical analyses assess probability that found
result is unlikely to be result of random variation in
population
• Convention is that result needs to be such that it
would not occur by chance more than 1 time in 20
(5%) or 1 time in 100 (1%) - or even less likely –
before we work on interpreting it
33. 33
Enhancing chances of getting Significance
• Carefully define your prediction (effectively bet on
outcome, don’t just look for difference say which
direction the difference should be)
• Use good reliable measures
• Select participants properly
• Use a large enough sample to be able to detect the
result you predict
34. Enhancing chances of making a real
significant discovery
Never ever „cherry-pick‟ the data that suit you
Never ever „cherry-pick‟ the results that suit you
Replicate, replicate, replicate
Test, test, test
35. 35
The core point
Scientific research needs to be seen for
what it truly is; a way of preventing me
from deceiving myself in regard to my
creatively formed subjective hunches
which have developed out of the
relationship between me and my material.
Carl Rogers 1961
36. Paired task
Considering the examples of “what not to do” from the
start of the session:
• Using the research question on your proforma,
start to construct a questionnaire to help you
answer this question. Think about the different
types of questions you could use, and how best to
phrase them.
• Pilot your questionnaire with another pair. What
improvements could you make?
37. Useful references
• Dillman, D.A. (1978) Mail and telephone surveys: The total
design method. New York: John Wiley and Sons.
• Oppenheim, A.N. (1992) Questionnaire design, interviewing
and attitude measurement. London: Pinter
• Sheatsley, P.B. (1983) Questionnaire construction and item
writing. In Ross, P.H., Wright, J.D. and Anderson, A.B.(Eds.),
Handbook of Survey Research. New York: Academic Press.
• Sudman, S. and Bradburn, N.M. (1989) Asking questions: A
practical guide to questionnaire design. San Francisco:
Jossey Bass.