Presentation by Stephen Hole in collaboration with Christopher House (University of Wales: Trinity Saint David, Swansea) and Gavin Bunting (University of Wales) at the Research-Teaching Practice in Wales Conference, 10th September 2013, at the University of Wales, Gregynog Hall. Slidecast edited by Professor Simon Haslett.
On National Teacher Day, meet the 2024-25 Kenan Fellows
It’s time to ‘Face’ the truth. Is Facebook’s Survey Monkey a legitimate research and pedagogical tool?
1. It’s time to ‘Face’ the truth. Is Facebook’s
Survey Monkey a legitimate research and
pedagogical tool?
Christopher House, Gavin Bunting
and Stephen Hole
.
1
2. Survey Monkey
• Survey Monkey is:
–
–
–
–
–
Well known;
Free for small surveys;
Can niche non spatial data;
Special interest groups;
A growing mechanism of choice for undergraduate
research.
2
3. Sources Surveyed
• Faculty of Applied Design & Engineering:
– PhD Students (some with 1,000 + replies);
– MSc & BSc class exercises and dissertations;
– HEI staff awareness survey.
• School of Applied Computing:
– Approximately 80 undergraduate projects and
dissertations per year for the last 5 years;
– 20% Dissertations;
– 80% Projects;
• Approximately 50% of students will use a survey to
gather primary data;
• Numbers are growing year on year.
3
4. Surveys: Setting the Scene (1)
• Surveys are probably the most commonly used
data gathering instrument world wide.
• Students’ perceive survey based research:
– To be straight-forward;
– An easy means of gathering primary data.
• Students' often fail to consider:
– Response rate;
– Reliability;
– Validity.
4
5. Surveys: Setting the Scene (2)
• They do not expect to encounter
problems and often produce surveys
that do not:
–
–
–
–
–
Take into account survey design;
Ask the right questions;
Gather appropriate samples;
Provide data that can be meaningfully analysed;
Take into account relevant statistical analysis
techniques.
5
6. Surveys: Setting the Scene (3)
• Badly designed surveys when administered:
– Render the analysed results:
• At best inconclusive;
• At worst producing information that results in flawed
decision making.
• Due to the time frame of an undergraduate
dissertation there is limited opportunity to
redesign and administer a further study.
6
7. Survey Process Activities
• The survey process typically includes the
following activities:
–
–
–
–
–
–
–
–
–
–
Setting specific, measurable objectives;
Planning and scheduling the survey;
Ensuring that appropriate resources are available;
Designing the survey;
Preparing the data collection instrument;
Validating the instrument;
Selecting the participants;
Administering and scoring the instrument;
Analysing the data;
Reporting the results (Pfleeger and Kitchenam, 2001)
7
8. Student Survey Dilemma (1)
• Not looking before they leap:
– Data gathering equates to a questionnaire;
– A questionnaire equates to Survey Monkey;
– Sample population equates to Facebook,
Twitter and other Social Media contacts;
– Data Analysis equates to Bar Charts, Pie
Charts, Line Graphs etc.
8
9. Student Survey Dilemma (2)
• Not looking before they leap results in:
–
–
–
–
–
Poor quality data;
Poor analysis;
Weak conclusions;
Inappropriate recommendations;
Low grading.
9
10. Question & Data Types
• Open Format;
• Closed Format:
–
–
–
–
–
–
–
Dichotomous;
Range;
Radio Button;
Rank Order Scaling;
Multiple Choice;
Likert Scale;
Numeric.
• Question type will determine data type:
– Survey Monkey constrains the options and
flexibility of survey design.
10
11. Statistical Methods
• Parametric
– Numeric data that is from a normal (bell shaped)
distribution.
• Non-parametric
– Data is ordinal in nature and does not fit a normal
distribution;
– Data is reliant on some form of ranking and is not
numeric in nature;
– Likert Scales generate ordinal (non-parametric
data).
11
13. Recognising Data Types
• Students may not understand the
difference between parametric and nonparametric data and as a result:
– Use a parametric statistical test on ordinal
data;
– Draw inaccurate results from their analysis.
13
14. Student Checklist (1)
• Survey Monkey is a legitimate research and
pedagogical tool provided the following checklist is
used:
– Questionnaire must be a suitable data collection
method;
– Administration by use of a web interface will not skew
the sample population;
– Sample population must be identifiable:
• Not an invite to all friends on Facebook, Twitter, etc;
– Quality design principles are incorporated into the
questionnaire construction;
14
15. Student Checklist (2)
– Ethical approval has been granted;
– No questionnaire to be administered without
supervisors approval;
– Questions meet the objectives of the research
hypothesis;
– Collected data can be suitably analysed;
– Statistical measures used in the analysis match the
type of data being collected:
» Parametric for normal bell-shaped data;
» Non-parametric for ordinal data.
15
16. Student Checklist (3)
– Select appropriate statistical analysis
software:
•
•
•
•
•
SPSS;
Microsoft Excel;
Weka Data Mining Tool;
SQL Server Data Mining Tool;
Etc.
– Understand the output generated by the
software with sufficient background
reading.
16
17. P& K Enhanced Survey Guidelines
• The survey process typically includes has the
following activities:
– Setting specific, measurable objectives:
• Administration by use of a web interface will not
skew the sample population.
– Planning and scheduling the survey:
• Consider innovative completion drivers.
– Ensuring that appropriate resources are available:
• Less significant in SM surveys.
– Designing the survey:
• Question type/data.
– Preparing the data collection instrument:
• Consider technical synchronisation of software
packages.
17
18. P& K Enhanced Survey Guidelines
– Validating the instrument:
• Vary depending on level of academic qualification.
– Selecting the participants:
• Carefully following methodological practice.
– Administering and scoring the instrument:
• Controlled by SM.
–
–
–
–
Analysing the data;
Reporting the results;
Ethical approval has been granted;
Supervisory approval has been granted.
18