"Emergency Medicine Resident Training in Interprofessional Skills: Evaluating a Needs-Based Curriculum"
Sondra Zabar, M.D., Principal Investigator Associate Professor of Medicine
Linda Regan M.D., Co-Investigator New York University School of Medicine
EMPACT aims to expand on previous work by assessing and improving EM resident competency in communication and professionalism through the development, implementation, and evaluation of new curriculum and assessment measures.
Picker Institute/Gold Foundation 2012-2013 Graduate Medical Education RFP
EMPACT: Emergency Medicine Professionalism and Communication Training
1. Can Unannounced Standardized Patients
Assess Professionalism and Communication
Skills in the Emergency Department?
Sondra Zabar, MD, Tavinder Ark, MSc, Colleen Gillespie, PhD, Amy Hsieh, MPA, Adina Kalet, MD,
Elizabeth Kachur, PhD, Jeffrey Manko, MD, and Linda Regan, MD
Abstract
Objectives: The authors piloted unannounced standardized patients (USPs) in an emergency medicine
(EM) residency to test feasibility, acceptability, and performance assessment of professionalism and com-
munication skills.
Methods: Fifteen postgraduate year (PGY)-2 EM residents were scheduled to be visited by two USPs
while working in the emergency department (ED). Multidisciplinary support was utilized to ensure suc-
cessful USP introduction. Scores (% well done) were calculated for communication and professionalism
skills using a 26-item, behaviorally anchored checklist. Residents’ attitudes toward USPs and USP detec-
tion were also surveyed.
Results: Of 27 USP encounters attempted, 17 (62%) were successfully completed. The detection rate was
44%. Eighty-three percent of residents who encountered a USP felt that the encounter did not hinder
daily practice and did not make them uncomfortable (86%) or suspicious of patients (71%). Overall, resi-
dents received a mean score of 60% for communication items rated ‘‘well done’’ (SD ± 28%,
range = 23%–100%) and 53% of professionalism items ‘‘well done’’ (SD ± 20%, range = 23%-85%). Resi-
dents’ communication skills were weakest for patient education and counseling (mean = 43%,
SD ± 31%), compared with information gathering (68%, SD ± 36% and relationship development (62%,
SD ± 32%). Scores of residents who detected USPs did not differ from those who had not.
Conclusions: Implementing USPs in the ED is feasible and acceptable to staff. The unpredictability of
the ED, specifically resident schedules, accounted for most incomplete encounters. USPs may represent
a new way to assess real-time resident physician performance without the need for faculty resources or
the bias introduced by direct observation.
ACADEMIC EMERGENCY MEDICINE 2009; 16:915–918 ª 2009 by the Society for Academic Emergency
Medicine
Keywords: standardized patients, graduate medical education, assessment, OSCE, professionalism,
communication, assessment
W
hat options exist for assessing communica- Accreditation Council for Graduate Medical Education
tion and professionalism skills? As resi- (ACGME) Outcomes Project,1 robust modalities to eval-
dency programs seek to comply with the uate clinical performance and effectiveness of educa-
tion are in high demand. The ACGME’s Toolbox
From the New York University School of Medicine (SZ, TA, contains numerous tools for assessing communication
CG, AH, AK, EK, JM, LR), New York, NY; and The Johns skills,2 but many of these rely on self-assessment or
Hopkins University School of Medicine (LR), Baltimore, MD. trained observers present during patient encounters.
Received February 27, 2009; revision received May 21, 2009; Patient complaints and postvisit surveys are useful for
accepted May 22, 2009. obtaining information, but offer limited opportunity
Presented at The Gold Foundation Symposium, ‘‘How Are We for physicians to translate feedback into practice
Teaching Humanism in Medicine and What is Working?’’ change. Unannounced standardized patients (USPs)3–5
September 27–29, 2007, Chicago, IL; and the 9th Annual Inter- present a method of measuring physicians’ communi-
national Meeting on Simulation in Healthcare (IMSH), January cation and professionalism skills in a real practice set-
10–14, 2009, Lake Buena Vista, FL. ting without the artificiality inherent in observed
Supported by Picker Institute Challenge Grant 2007. structured clinical exams (OSCEs).6–8
Address for correspondence and reprints: Sondra Zabar, MD; We hypothesized that USPs can provide a real-time,
e-mail: sondra.zabar@nyumc.org. accurate alternative to direct observation and OSCEs.
ª 2009 by the Society for Academic Emergency Medicine ISSN 1069-6563
doi: 10.1111/j.1553-2712.2009.00510.x PII ISSN 1069-6563583 915
2. 916 Zabar et al. • PROFESSIONALISM AND COMMUNICATION
The purpose of this project was to 1) describe the tion, 6) review and practice with rating checklist, and
process of conducting a USP program in an emergency 7) preparatory observational visit to the ED. Actors
department (ED), 2) determine if implementing USPs in were compensated at a rate of $25 ⁄ hour for both train-
the ED is feasible, and 3) present preliminary results of ing and in-ED time.
a USP performance assessment.
USP Encounter. Unannounced standardized patients
METHODS met project staff while residents attended a required
conference. USPs were introduced to the triage nurse,
Study Design the MR administrator, and the attending. The USPs
This was a prospective, nonrandomized, cohort study were triaged per standard procedure.
to assess professionalism and communication abilities During the encounter, USPs complied with any (non-
of emergency medicine (EM) residents using USPs. invasive) exam and accepted all appointments and
Informed consent was obtained from all participants. prescriptions, which were canceled postencounter. If
Research activities in this study were approved by the the resident insisted on any course of action that made
New York University School of Medicine Institutional the USP feel unsafe, the USP was to ask for the attending,
Review Board through a resident registry wherein resi- send a short message service (SMS) text message to
dents are asked to consent to allow inclusion of their project staff, or simply leave the ED. Hospital billing
educational and performance data in a research data- canceled the visit at the end of the day. Total time in
base. Data, therefore, are reported only for those resi- the ED was 1.5 to 4 hours ⁄ visit. Immediately following
dents for whom such consent was obtained. the encounter, the USP debriefed and completed a
behaviorally anchored checklist that assessed resident
Study Setting and Population skills and the USP’s satisfaction with the visit.
The Bellevue ED is a busy Level 1 trauma center at an
academic medical center in New York City. The ED sees Post-USP Survey. At the end of the project, all EM
approximately 100,000 visits per year. residents (including those who did not see a USP;
Fifteen EM residents in their second year of post- n = 30) were surveyed about their attitudes toward
graduate training (PGY-2) participated in the EM Pro- USPs using a four-point scale (1 = strongly disagree,
fessionalism and Communication Training (EMPACT) 4 = strongly agree) and open-ended questions. To
Program. At the conclusion of the EMPACT training, determine detection rates, residents were asked if they
residents were informed that they might be visited by had encountered a USP and if so to identify the USP’s
USPs during their subsequent time working in the sex and chief complaint.
ED. However, residents were blinded as to the exact
date of the visit or patient complaint. Data Analysis
Unannounced standardized patients assessed residents’
Study Protocol professionalism and communication skills and their
Logistics. We required involvement from most ED satisfaction with the patient-centeredness9,10 of the
staff areas including nurses, attending physicians, medi- visit using a three-point scale: ‘‘not done,’’ ‘‘partially
cal records (MRs), registration, informatics, and radio- done,’’ and ‘‘well done.’’ Scores were calculated as the
logy. To ensure fidelity for each USP visit, we created a percentage of well-done items (Table 1). Professional-
preexisting MR with a unique number, patient name ism and communication skills were scored from 13
and identifying information, prior visits, and test items and patient centeredness from eight items. Over-
results. Each resident was scheduled to receive two all recommendation ratings were obtained using a
USPs in urgent care (where residents’ schedules were four-point scale. Reliability estimates (Cronbach’s
relatively predictable) during the 4 to 6 weeks after the alpha ‘t’) and descriptive statistics (means, standard
EMPACT curriculum. deviations [SDs], and ranges) are reported. Correla-
tions (Pearson’s r) between scores earned in the two
USP Scenarios. We used two USP cases previously separate cases are also reported to assess stability of
validated in OSCEs, representing common ED chal- performance.
lenges and requiring only communication-based inter-
ventions. In the first case (a misread x-ray), residents
RESULTS
needed to educate an angry patient recalled for a mis-
read x-ray (skills: delivering bad news, dealing with a Seventeen of 27 visits were successfully conducted and
challenging patient, accountability), and in the second evaluated. Resident scheduling problems explained
(a repeat visitor), care for a dissatisfied patient with most incomplete encounters. Five residents were visited
chronic pain who repeatedly uses the ED (skills: han- by USPs from both cases, and seven residents from one
dling emotion, patient education, accountability). case.
USP Training. Eight actors were recruited. On aver- USP Detection
age, each received seven hours of training consisting of Seven of 12 residents who encountered a USP pro-
1) discussion of character and situation, 2) calibration vided information on detection; four of their nine
of emotional tone, 3) role play for standardization, encounters were detected (44% detection rate). Five of
4) practice with attending and chief residents for realism, 18 residents who did not see a USP indicated that they
5) review of ‘‘ground rules’’ for safety and nondetec- did (28% false-positive rate). One of the residents who
3. ACAD EMERG MED • September 2009, Vol. 16, No. 9 • www.aemj.org 917
Table 1
Resident Performance with USPs
Domains of Assessment Items Mean, % ±SD, % Range, % Reliability
Communication* 60 28 23–100 0.91
Information gathering Used appropriate questions 68 36 0–100 0.82
Clarified information
Allowed to talk without interrupting
Relationship development Communicated concern 62 32 20–100 0.85
Nonverbal enhanced communication
Acknowledged emotions
Was accepting ⁄ nonjudgmental
Used words you understood
Education and counseling Asked questions to see what you understood 43 31 0–100 0.78
Provided clear explanations
Collaborated with you in identifying next steps
Professionalism* 53 20 23–85 0.62
Accountability Disclosed error 49 23 0–80 0.60
Personally apologized
Took responsibility for situation
Manage difficult situation Avoided assigning blame 91 16 60–100 0.85
Maintained professionalism
Giving bad news Prepared you to receive news 42 34 0–83 0.63
Gave you opportunity to emotionally respond
Provided appropriate next steps
Treatment plan and management Assessed resources 50 39 0–100 0.66
Arranged for follow-up
Discussed plan
Patient Centeredness* Fully explored my experience 43 29 0–75 0.91
Explored my expectations
Came to an agreement
Took a personal interest in me
Earned (regained) my trust
Acknowledged impact of error
Didn’t make me feel wasting time
I was given enough information
Recommendation 2.3 0.9 1.0–3.5 0.90
n = 12 residents, 17 visits; reliability assessed with Cronbach’s alpha. Four-point scale: 1 = not recommend; 2 = recommend with
reservations; 3 = recommend; 4 = highly recommend.
USPs = unannounced standardized patients.
*Mean percentage of items rated as ‘‘well done’’ (not, partly, or well done).
’’Would you recommend this physician to a family member of friend?’’
reported a false detection reported ignoring that think more (29%), or led them to be more self-aware
patient. (43%).
USP Performance DISCUSSION
The reliability of scores (Table 1) suggests adequate
internal consistency (a > 0.60). Residents performed Our results show that developing and implementing a
better in the misread x-ray case than in the repeat USP program in the ED is feasible and acceptable to
visitor case in professionalism (70% vs. 35%, t = 2.81, residents. Considering the drawbacks of OSCE assess-
p = 0.048) and patient-centeredness (66% vs. 40%, ment and direct observation, combined with increasing
t = 1.96, p = 0.05). Communication (r = 0.73, p = 0.16) demands on faculty time and decreasing funding, USPs
and recommendation scores (r = 0.81, p = 0.09) were may offer an objective, cost-effective method for evalu-
highly, albeit not significantly, correlated between the ating accurate practice skills.
two cases, but professionalism (r = 0.24, p = 0.70), and The biggest challenge faced while implementing the
patient-centeredness were not (r = 0.08, p = 0.90), sug- USP program was the unpredictability of the ED. Occa-
gesting case content matters most in these domains. sionally, USPs were mistakenly examined by another
resident. Both content (highly trained SP, realistic
Postevaluation Survey cases) and logistic factors (dedicated program coordi-
Eighty-three percent of residents who encountered a nator, electronic MRs, team collaboration) are neces-
USP felt that it did not hinder their daily practice sary for successful integration. Total cost, in terms of
and did not make them uncomfortable (86%) or sus- both time and money, is likely greater up front, with
picious of patients (71%). A minority of those resi- decreased workload, time, and expense as USPs and
dents who encountered a USP felt that the encounter staff become trained. Further study of the costs is
improved their practice behavior (14%), made them needed.
4. 918 Zabar et al. • PROFESSIONALISM AND COMMUNICATION
Even with a high detection rate, residents reported at: http://www.acgme.org/outcome/comp/compCPRL.
value in the USP program for learning and patient care. asp. Accessed Sep 20, 2008.
It is possible that informing residents that USPs would 2. Accreditation Council for Graduate Medical Educa-
be visiting them in the ED improved performance. tion, American Board of Medical Specialties.
More importantly, the majority of residents did not feel Outcome Project Toolbox of Assessment Methods.
that the possibility of encountering a USP had any neg- Available at: http://www.acgme.org/outcome/assess/
ative impact on their daily practice, suggesting that toolbox.asp. Accessed Jun 20, 2009.
USPs in the ED will not risk real patient safety. The 3. Gorter S, Scherpbier A, Brauer J, et al. Doctor-
case of the resident who reported ignoring a patient patient interaction: standardized patients’ reflec-
thought to be ‘‘unannounced’’ represents an unantici- tions from inside the rheumatological office.
pated and anomalous professionalism issue, we believe, J Rheumatol. 2002; 29(7):1496–500.
not causally related to the use of USPs; it demonstrates 4. Kravitz RL, Epstein RM, Feldman MD, et al. Influ-
how USPs can provide useful information to program ence of patients’ requests for direct-to-customer
directors. advertised antidepressants: a randomized controlled
trial. JAMA. 2005; 293(16):1995–2002.
LIMITATIONS 5. Ozuah PO, Reznik M. Using unannounced standard-
ised patients to assess residents’ professionalism.
There was a small sample size, with a relatively large Med Educ. 2008; 42(5):532–3.
proportion of failed USP visits. However, the failure 6. Fiscella K, Franks P, Srinivasan M, Kravitz RL,
rate improved as the project progressed. Even with our Epstein R. Ratings of physician communication by
small numbers, it appears that two cases and the items real and standardized patients. Ann Fam Med. 2007;
on the behaviorally anchored checklist can discriminate 5(2):151–8.
residents based on their communication skills. 7. Talente G, Haist SA, Wilson JF. The relationship
between experience with standardized patient
CONCLUSIONS examinations and subsequent standardized patient
examination performance: A potential problem with
With the ACGME placing greater importance on evalua- standardized patient exam validity. Eval Health Prof.
tion of patient outcomes, we believe that our project 2007; 30(1):64–74.
represents a new way to assess real-time resident perfor- 8. Srinivasan M, Franks P, Meredith LS, Fiscella K,
mance. Despite being time-consuming and subject to the Epstein RM, Kravitz RL. Connoisseurs of care?
unpredictability of the ED, implementing unannounced unannounced standardized patients’ ratings of phy-
standardized patients in the ED is feasible and acceptable sicians. Med Care. 2006; 44(12):1092–8.
to staff. Future comparison of unannounced standard- 9. Marshall GN, Hays RD. The Patient Satisfaction
ized patients with observed structured clinical exam Questionnaire Short Form (PSQ-18). RAND Corpo-
scores will enable educators to determine how well these ration, Paper P-7865. Available at: http://www.
methods assess performance in actual practice. rand.org/pubs/papers/P7865/. Accessed Jun 20, 2009.
10. Elwyn G, Edwards A, Wensing M, Hood K, Atwell
References C, Grol R. Shared decision making: developing the
OPTION scale for measuring patient involvement.
1. Accreditation Council for Graduate Medical Educa- Qual Saf Health Care. 2003; 12:93–9.
tion (ACGME). ACGME Outcome Project. Available