Slides accompanying a 90-min SEDA workshop on 15th Nov 2012. Credit to Tunde Varga-Atkins, Jaye McIsaac and Ian Willis, University of Liverpool. It is the first time we have introduced our new, combined approach for gathering student feedback on teaching. The method can also be used in other contexts such as curriculum review or development. The Nominal Group Technique is akin to focus groups, but with more structure and an immediate, quantitative output. Our approach has been to combine two stages: stage 1, focus group, followed by stage 2: nominal group. We have found this an effective approach at the University of Liverpool.

1. Evaluation techniques of teaching:
focus groups and the
Nominal Group Technique
Tünde Varga-Atkins, eLearning Unit
University of Liverpool
15 November 2012
SEDA Conference
Aston Business School, Birmingham

2. OUTLINE
• Introduction
• Demonstration of the Nominal Group
Technique
• Benefits and disadvantages of NGT
• A new, combined two-staged
approach: Nominal Focus Group
• Reflections in own context

3. OUTCOMES
• Familiar with the NGT and its stages.
• Aware of the benefits and potential
challenges of NGT.
• Contrast focus groups & NGT.
• Consider a combined approach of
FG&NGT.
• Reflect on the evaluation technique in
own context.

4. CONTEXT
• Curriculum development
• Student engagement
• Our research-informed
experiences as educational
developers for last 4-5 years.
• Hopefully a useful evaluation
technique.
• Works in other contexts: staff,
or any setting requiring
group-decision making.
• Method selected to suit
purpose!

5. LISTENING TO THE EXPERIENCE?
survey face-to-face groups
 Wood-peckers?
 Wolves?
 Dinosaurs? “I found it extremely
…. helpful to have not just an
idea what is going on,
but to hear what is said
“with the questionnaire, by students.”(staff)
you never know if [you]
give the right question
out.”(staff)

6. LISTENING TO THE EXPERIENCE?
survey face-to-face groups
 Wood-peckers?
 Wolves?
 Dinosaurs?
….
Please rate how useful
was the wood-pecker‟s
song in relation to your
journey:
10-Very useful. 0-Not useful at all.
Gary Robson - Flickr

7. PRIOR EXPERIENCES WITH NGT?
face-to-face
focus groups
Nominal groups
interviews
quantitative qualitative
surveys
Delphi technique
at-a-distance

8. WHAT IS
NOMINAL GROUP TECHNIQUE?
• Structured group activity
• One given topic
• Facilitates group
decision-making
• Immediate results
• Quantitative element
• Reduces researcher/participant
bias
Delbecq & Van de Ven (1971)

9. NGT: STAGES
Example question: „What would you change in your
course?‟
1) Individual 2) Clarification and 3) Ranking responses
responses consolidation
Duration = 1-1.5 hours
„Nominally‟ group < individual

10. AND NOW: LET’S HAVE A GO...
THE QUESTION IS:
In your current role as
educational developer
what is one
key challenging issue
you are facing?
[purpose: identifying top 3 key ones to tackle together]
[normally we would ask participants to write 2-3 – but
shortening the task here due to time constraints]

11. STAGE 1: CHALLENGING ISSUE(S)
1 2 3
.
4
5 6

12. STAGE 2 CONSOLIDATION: CHALLENGING ISSUES
Same answers
4
2
Same answers
Same answer
3 6
5
1
.

13. STAGE 3 RANKING:
YOUR TOP 3 KEY CHALLENGES
that you want the group/SEDA etc. to tackle…
Item no. Item description
3 points 1
2 points 2
1 points 4
[normally top 5 but for brevity
only doing 3]

14. THE TOP FIVE KEY CHALLENGES AS
EDDEV-ER:
1 1
2
2
3
3
4
4 5
5

15. SUMMARY & BRIEF QUESTIONS
• Nominal Group Technique
• 3 stages
• 1-2 key questions explored.
• Focus on individual work.
• Group consensus.
• Quantitative outcome.
• Scalable: results from more groups can be
integrated.

16. BENEFITS AND CHALLENGES
• Paperchase exercise
• 3 minutes for each flipchart:
dis/advanta
ges of NGT
NGT vs focus NGT vs
groups surveys

17. GUIDE TO NGT & PROJECT REPORT
See References
On slideshare.net , search for Nominal Group
Technique

18. Context NGT is more useful for: NGT is less useful for:
Research Evaluation and decision-making Researching general
purpose learner experiences
Topic focus When you have one single topic When you have more
to explore topics or a complex topic
to explore
Likely “What changes would you make “What are your
research to your programme/curriculum?” experiences with your
questions “What would help you improve programme so far?”
the quality of feedback on this “What are your
course?” experiences with the
quality of feedback on this
course?”
Participants Participants with different power If power relations are not
relations within the same group; an issue in the group.
when consulting various
stakeholders groups within same
research (e.g. from students

19. A COMBINED, TWO-STAGED APPROACH:
‘NOMINAL FOCUS GROUP’
Why?
What is it?
Jaye McIsaac, Educational Development, University of Liverpool
Video at: https://stream.liv.ac.uk/mntbvv9d

20. A COMBINED, TWO-STAGED APPROACH:
‘NOMINAL FOCUS GROUP’
Stage 1: focus group
Stage 2: nominal, ranking „bit‟

21. STUDENT EVALUATION ON
EVALUATION (SURVEY, N=13)
This is the Worlde cloud of the student
survey responses on the group
process.

22. SURVEY: EFFECT OF SESSION & F2F
“felt you could be honest “All of the students agreed and
and discuss openly about appeared to be facing very similar
your opinions” issues to myself in terms of
feedback”
“Was nice to see if the
University cared about the
Visual attribution
problems we are all of responses is for
having.” illustrative
purposes only
(survey was
anonymous)

23. VIEWS ON STAGE 1: FOCUS GROUP
“can speak much more info “good to hear other students‟
than writing down on a opinions to help expand my own”
post-it.”
“able to agree/disagree
with other people‟s “Discussing made me
experiences and share remember problems in Visual attribution
your own to enforce or previous years.”
of responses is for
illustrative
refute their opinion.” purposes only
(survey was
anonymous)

24. VIEWS ON STAGE 2: ‘NOMINAL’ BIT
“outlined the main
problems with feedback “[gave] more time to
and made it clear what is think about answers.”
needed to improve.”
“if individuals didn‟t contribute
much in the open discussion, their
views were still taken in to account Visual attribution
[in stage 2]” of responses is for
illustrative
purposes only
(survey was
anonymous)

25. SURVEY: VIEWS ON COMBINATION
“The open discussion
allowed many ideas to be
put forward, where as the
second part of the session
allowed a summary of all
of the views that were
discussed.”
“The open discussion
helped to get me thinking
of my own experience of Visual attribution
feedback, whilst writing my of responses is for
opinion on the post-it note illustrative
helped get my opinion purposes only
(survey was
across..” anonymous)

26. STAFF PERCEPTIONS ON THE TWO-
STAGED ‘NOMINAL FOCUS GROUP’
“Yes the nominal group
technique, in the end it
brought everything together
into a sharper point again.
… It kind of made it easier
for us to identify what the
students thought was the
most important thing...”

27. STAFF PERCEPTION ON THE
‘NOMINAL FOCUS GROUP’
“dealing with basically a bullet
point, … you might get the
meaning wrong. You might
not understand, really what
they meant. Whereas [the
Focus Group the citations from
students] explained a bit more
of what they meant. ”

28. SUMMARY: FOCUS GROUP ONLY
Group energises Volume of issues?
Group helps to formulate Overall feelings?
ideas and feelings Issue bias?
„others feel the same!‟, Participant bias?
reassurance

29. SUMMARY: NOMINAL GROUP
TECHNIQUE ONLY
Immediate results If more questions?
Quantitative ranking If experiences?
indicates volume of issues Students warmed up?
Overall impression

30. SUMMARY: ADVANTAGES OF
2-STAGED NOMINAL FOCUS GROUP
Stage 1: FG Stage 2: NGT
Group energises Immediate results
Group helps to formulate Quantitative ranking
ideas and feelings indicates volume of issues
„Others feel the same!‟ Overall impression

31. SMALL-GROUP DISCUSSION
• What is the relevance of these techniques (NFG,
NGT, FG) in your context?
• What methods and techniques does your institution /
department use for evaluation of teaching?
• What works well?
• What are the challenges? Opportunities?
• Any relevance of these methods?
• Or considerations for using these methods?

32. KEY MESSAGE
We have found the two-staged
„Nominal Focus Group‟ to be
an efficient and useful method
for
evaluation of teaching &
curriculumalldevelopment.
[It is a technique to add to your repertoire of evaluation
methods. It may not suit contexts, and the full evaluation
cycle is the most important including a feedback loop and
action on results! ]

33. [OUTCOMES] YOU:
• Are now familiar with the NGT and its
stages.
• Discussed benefits and potential
challenges of NGT.
• Contrasted focus groups & NGT.
• Considered a combined approach of
FG&NGT.
• Reflected on methods in own context.

34. THANK YOU
Contact:
Tünde Varga-Atkins
tva@liverpool.ac.uk
#tundeva
#elearninglpool
http://liverpool.academia.edu/T%C3%BCndeVargaAtkins
Today‟s resources on http://slideshare.net(search for nominal
group technique)

35. REFERENCES
• Delbecq, A., Van de Ven, Andrew, & Gustafson, D.
(1975). Group techniques for program planning a :
guide to nominal group and Delphi processes.
Glenview Ill.: Scott Foresman.
• Further references in:
• Varga-Atkins, McIsaac et al (2011) Using the nominal
group technique with clickers to research student
experiences of e-learning: a project report
[http://slidesha.re/xQlBCg ]
• Varga-Atkins, McIsaac et al (2011) The Nominal Group
Technique – a practical guide for facilitators
[http://slidesha.re/AmYOgv]

36. HANDOUTS:
STAGE 1 INDIVIDUAL
RESPONSE
Participants Facilitator
Participants enter their individual Ensures everyone works on their
response on a post-it note. own and writes clearly and
legibly.
Post-its are pinned on a Facilitator helps pin up responses
flipchart. and numbers each response so
that they can be referred to
later.
Participants read out their own Facilitator , if needed, asks for a
response. brief clarification on the item.
The items are NOT discussed in
detail in this stage.

37. HANDOUTS:
STAGE 2 CONSOLIDATION
Participants Facilitator
Participants (Ps) find Facilitator prompts Ps to find
similar/same items. similar items. Facilitator asks Ps
to work together on merging
items if they are the same.
Ps discuss and agree on the Facilitator adds newly
merging of similar items (group formed/merged items as Ps
consensus). discuss these. (and making sure
items are not themed, but only
similar items are
Participants do this until all items Facilitator makes sure each item
have been grouped if relevant. is numbered.
This is the longest stage.

38. HANDOUTS:
STAGE 3 RANKING
Participants Facilitator
Participants are asked to Faciltiator hands out ranking
choose their top 3 (normally 5) sheet and explains the ranking.
most important responses to
them. The order of importance is
important.
Participants rank the items on
their ranking sheet. (or on
flipchart is also possible.) 3 points
go to the most important one, 2
points to the second most
important and 1 point to the
third most imp.
Participants hand in their ranking Facilitator calculates ranking
sheet. score.

39. RANKING: WHAT ARE THE 3 MOST
IMPORTANT ITEMS TO YOU?
Item no. Item description
3 points
2 points
1 points