Presentation about a project undertaken by James Petersen and Dr. Tod Aeby of the UHM School of Medicine to enhance engagement and participation by residents in the weekly Morbidity and Mortality Conference. The use of mobile audience response apps was added to the weekly conference with positive results

1. M&M -M&M - Final Mobile Learning ProjectFinal Mobile Learning Project
The Use of MobileThe Use of Mobile
Audience ResponseAudience Response
Apps in Post-GraduateApps in Post-Graduate
Medical EducationMedical Education
ETEC 647 - Fall12ETEC 647 - Fall12
Tod AebyTod Aeby
Jim PetersenJim Petersen
Sean MoroneySean Moroney

2. Please Join Us

3. Overview
• Review the purpose of the project
• The M&M Concerns
• What happened on November 16, 2012
• What we learned

4. Were you paying attention?
• Please enter what you you remember
about what we were trying to improve
with mobile learning

5. Were you paying attention?
• Please enter what you you remember
about what we were trying to improve
with mobile learning
arner Engagement!!

6. Mobile Audience Response: enhanced engagement for
learningThere is substantial evidence in the educational literatureThere is substantial evidence in the educational literature
indicating that increased engagement, participation, andindicating that increased engagement, participation, and
interaction in classroom and other educational settings caninteraction in classroom and other educational settings can
positively impact the learning of studentspositively impact the learning of students
The purpose of this project is to introduce the use of a free mobileThe purpose of this project is to introduce the use of a free mobile
audience response application with the goal of increasingaudience response application with the goal of increasing
engagement and participation of the residents who do not usuallyengagement and participation of the residents who do not usually
respond or participate in the conference. The assumption is thatrespond or participate in the conference. The assumption is that
increased engagement and participation will enhance the value ofincreased engagement and participation will enhance the value of
the M&M Conference as an important learning tool in the residentsthe M&M Conference as an important learning tool in the residents’’
medical education.medical education.
Recall:Recall:

7. Review of Literature
•Note: the literature concerning Audience Response Systems refers almost entirely to the more traditional “Clicker” systems
•Audience response systems (ARS) were first introduced at Stanford and Cornell Universities in the mid 1960s and became
commercially available in 1992. (Kay & LeSage, 2009)
•Since the early 1990s, the use of audience response systems in higher education, professional education, and business has grown
substantially.
•In the past two decades, there have been numerous studies concerning the use of audience response systems that have indicated
that this strategy has benefits in the teaching and learning process.
Some of these are:
•“students are more engaged in class”, students participate with peers more...to solve problems”, and “students are more
focused in class”.
•There is also evidence that the use of these devices improves formative assessment and contingent teaching. (Kay & LeSage,
2009)
•There is evidence that the use of ARS provides a more positive impact on student participation than do more traditional
audience response strategies such as surveys and note cards. (Stowell & Nelson, 2007)

8. Review of Literature
Investigators have reported that the use ARS can be efficacious in medical education; specifically in the process of
delivering medical information to postgraduate medical residents.
In one study of Family Medicine Residents, the investigators conducted a controlled crossover study and found
a significant difference in the retention of information between groups who received traditional didactic lectures,
lectures with an interactive component, and interactive lectures incorporating ARS. The investigators concluded
that “improved retention occurs with active participation in the lecture process.” (Schackow, Chavez, Loya, &
Friedman, 2004)
Another study compared the efficacy of traditional lecture style delivery of information to Obstetrics and
Gynecology Residents with interactive lecture style delivery supplemented with an audience response system. The
results of this randomized controlled trial indicated that Residents who received the ARS interactive lecture
treatment showed a pre/post test improvement approximately ten times that of the group receiving the traditional
lecture treatment. (Pradhan, Sparano, & Ananth, 2005)
A review of the literature concerning the efficacy of ARS predominantly indicates the value of interactive devices in
promoting learning, retention, and engagement. (Banks, 2006) (Schmidt, 2011) (Guse & Zobitz, 2011)
There is also evidence that the use of ARS to replace traditional didactic lecture delivery enhances motivation and positive
affective outcomes. (Cain, Black, & Rohr, 2009)
*! - important*! - important

9. Clients costcost
(50 seats)(50 seats)
ease of deploymentease of deployment
1-5 stars1-5 stars
ease of useease of use
1-5 stars1-5 stars
types of responses
available
other costs
add-ons
developer link Platforms Comments
eClicker Audience Free to student
*** ****
yes/no
agee/dis
muliple choice
No short answer
Presenter App is $15.00 http://www.bignerdranch.com/apps/eclicker-audienceiOS, html Well established and
stable company.
Merida
WebKeyPad
Get quote
*** ****
yes/no
multiple choice
NO short answer
Hard to know! http://www.meridiaars.com/audiencepollingsoftware/Android, iOS,
BlackBerry or Windows
Phone, html
Lots of polling options.
Have to email for a
price quote (sounds
expensive)
Socrative free (50 seats)
***** *****
yes/no
multiple choice
short answer
games
exit ticket
None http://www.socrative.com Android, iOS,
BlackBerry or
Windows Phone, html
Great Blog
easy to use
Solero Respond $5.00 Client
**** ****
yes/no
muliple choice
short answer
$10-30 per month
subscription
http://solaro.com/otherproducts IOS
SRN Client free to student
** **
yes/no
muliple choice
short answer
$125 (50 clients) http://studentresponsenetwork.comIOS, html personal server-based
Turning Technologies
ResponseWare
$19/yr
per student ** ****
yes/no
multiple choice
short answer
Presenter costs http://www.turningtechnologies.com/response-solutionsProprietary clickers,
Android, iOS,
BlackBerry or Windows
Phone
Can be used in mixed
environment (phones,
various clickers, etc)
Via Response free (10 seat trial)
** **
extensive (if you have to ask, you
can’t afford it - fee not
openly posted)
http://www.viaresponse.com/Whatisit.htmlIOS, html (low ratings on App
Store)
Poll Everywhere
(SMS based)
available free on smart
device *** ***
yes/no
multiple choice
$15-1400 per month
subscription
http://www.polleverywhere.complatform independent
(smartphone)
Selection of Mobile AudienceSelection of Mobile Audience
Response AppResponse App

10. Three strategies
• Test general knowledge on a subject
• What do they know and are they paying attention?
• Force commitment to test clinical
judgement and reasoning
• What would they do and are they paying attention?
• Assess the current mental model
• Retest to confirm mastery of the concepts
• Did they follow the conversation and construct the appropriate mental model

11. The Conference The project team arrived early to set
up the equipment

12. The Conference
The audience customarily seats itself in
a “culturally” determined manner, by role
group

13. The Conference The Moderator was positioned so
that he could observe the presenter,
audience, and both screens

14. The Conference
Audience questions were clearly delineated by a
special slide style

15. The Conference
Using Socrative on the iPad to present
responses from the audience

16. The Conference
Audience responses are then projected to the
smaller screen

17. The Conference
Unexpected Consequence -
Formative Assessment - immediate feedback
to moderator allowing instructional adjustment

18. The Residents present theThe Residents present the
cases...cases...

19. Other information of importance to
medical practice is also presented

26. 73% Laparoscopy 65% Laparotomy

28. 73% Laparoscopy 65% Laparotomy

32. Evidence-based
Treatment of URI
• Zinc lozenges
• High dose vitamin C
• Symptom relief

33. Vote or New List

35. Text

36. Text

37. Text

38. Text

39. Text

40. Text

41. Average 6.4 (2-10)
• 30% “Totally Got It”, 68% “Pretty Well”
• “Need bigger screen”, “Slow internet”,
“Awesome, dude”,”Kinda fun”

42. Staff ObserversStaff Observers
Three staff members served as
volunteer observers
They were provided a link to a
Google Form to record their
observations

43. Observers
• Audience size estimate at start, 39-50
• Mobile devices out/running @ 5 minutes 40 -
95%
• “Socrative taking a while”, “May need printed
directions for late comers”, “Can you mask the
entering answers so you don’t influence the
later ones”, Faculty and private docs had
mobile devices but not using Socrative”, “Most
residence [sic] seemed engaged”

44. Observers
• The observers, some known to the audience
(hospital staff) and some unknown (UH ETEC
students), unobtrusively made periodic counts
of the audience to assess involvement.
• At any time throughout the session, about two-
thirds of the audience members had a mobile
device on.
• Perhaps, not all of these were linked to
Socrative.

45. Audience Engagement
Observe
r
7:45 8:00 8:15 8:30 8:45
1 9 9 9 8 9
2 10 10 9 9 9
3 4 7 9 8 8

46. Important Logistical Points
• Late arrivals may have had difficulty logging in.
Some form of written instructions on this would
probably be handy.
• Not revealing the answers given to a question
until the question time is closed can give a
clearer picture of the thinking of the audience
members.
• Network speed can be an issue in the
briskness of the interactions.

47. JimJim’’ss
ReflectionReflection
• Process
• This was a fascinating project on a number of different levels
• The process was challenging since all of the team members are over-employed adult
professionals residing in geographically disparate locations and with vastly differing
schedules.
• Thus, online collaboration was absolutely necessary. Extensive use was made of
Google Docs, Dropbox, and BlackBoard.
• It became apparent during the course of the project that organization and
communication of information needs to be very different in an online
collaboration setting.

48. JimJim’’ss
Reflection 2Reflection 2
• Content
• One of the big realizations for me was how the app enabled the moderator to make
on-the-fly formative assessments and adjust instruction
• This project has, for me, wide extensibility in a variety of educational settings.
Several teachers in my school are now using the tool in classroom settings
• It was mentioned in the research literature and observed in the trial that there is a
strong affective component when mobile devices are used in teaching and learning.
• I learned valuable lessons about the physical layout necessary to integrate the
mobile element and I have some suggestions I will forward to the app developer to
make it a more effective tool. (e.g. ability to hide responses until all recorded)

49. Sean’s Reflection

50. Tod’s Reflection
How did all the old white-hairs all end up in the same group?
A dry run really helps
Improved participation,will it last?
Too many devices in the kitchen spoils the soup
-Have to have broad band
-What in the world am I going to do on Maui???
This has been a wonderful, eye-opening educational experience
Thanks Dr. Grace!