Dr. Ara Tekian discusses the process of blueprinting national examinations. A blueprint outlines the content and proportion of questions to be included in an exam. It is developed through a multi-step process involving committees of content experts. They identify important content areas and assign weightings. A draft is created and surveyed to a broader group of experts for feedback. Considering survey results and ensuring coverage of important topics, the committees finalize the blueprint to guide exam construction. The National Board of Medical Examiners and American Board of Internal Medicine were provided as examples of organizations that follow rigorous blueprinting processes.

1. Dr. Ara Tekian
University of Illinois
Plenary II: The Process of
Blueprinting National Examinations

2. The Process of Blueprinting
National Examinations
Ara Tekian, PhD, MHPE
University of Illinois at Chicago
tekian@uic.edu
TOUCHSTONE Institute – Annual Symposium
January 22, 2018, Toronto

3. Outline
•Process of developing blueprints
•Experience of the National Board of the Medical
Examiners (NBME) in Blueprinting
• Two committees of content experts
• Five step process
•American Board of Internal Medicine (ABIM)
• Blueprint for MOC exam
• Sample content distribution
• Detailed blueprint with levels of importance

4. The Blueprint
•A document that describes the CONTENT to be
included in an examination.
• Indicates the proportion of questions
• General and specific content areas
•It serves as a guide for the design of the
examination and for developing a pool of potential
test questions or items.

6. Components of a Faculty Development Program
in Assessment
(Chapter 16)
1. Basic principles
2. Methods and their alignment with
competencies
3. Blueprinting and test construction
4. Assessor training
5. Scoring and standard setting

7. System History Explanation Examination Procedures
Cardiovascular Chest pain Discharge drugs Cardiac Blood pressure
management
Respiratory Hemoptysis Respiratory Peak flow
Gastro-intestinal Abdominal pain Gastroscopy Abdominal Rectal examination
Reproductive Amenorrhea Abnormal smear Cervical smears
Nervous Headache Eyes Ophthalmology
Musculo-skeletal Backache Hip
Generic Pre-op
assessment
Consent for post-
mortem
IV cannulation
Example of an Objective Structured
Clinical Examination (OSCE) Blueprint

8. A Sample Blueprint
for National Examination

9. Experience of NBME in
Blueprinting*
Performing a job or task analysis is expensive and time
consuming
Therefore -
• If content experts can identify the tasks involved in
performing a job
• They should be able to identify what knowledge one must
have in order to perform that job.
• They should be able to identify the knowledge base that
needed to be tested in order to show competency in
performing the job.
*NBME Blueprinting, 2016

10. Acceptance of the rationale
•Asking content experts:
• What should be on the test?
• How much of it?
•Immediate appeal for the following reasons:
• addressed the blueprint question
• concerned only with concepts that could be measured by
MCQs
• relatively simple, practical, and cost-effective

11. Before beginning to develop a
content outline
•Establish the primary purpose of the examination
and the inferences to be made about examinees
from their scores
•Ensure that the content scope and breadth
represented in the blueprint support appropriate
score interpretations
• Common inference:
Examinees who pass a certification examination should have
demonstrated adequate content mastery to practice in a
specialty or subspecialty area.

12. Main questions in developing
a blueprint
•What areas of knowledge should be included in the
test?
•How much of each area should be included?
Note: While a practice analysis study can tell us what
the target examinees do on the job, what actions
they perform, and what decisions they must make, it
does not directly address the question of what should
be on the test.

13. Recruitment of two committees of
content experts
•The Advisory Committee will develop the
prototype, evaluate comments, and finalize the
blueprint.
• academic and industry leaders in the subject area
• approximately 7 members
•Subject matter experts will review the prototype
draft and suggest specific content areas to add,
remove, or expand.
• provide overall comments on the breadth, depth, and
balance of the content.
• 30 to 50 experts

14. Expertise of the two committees
•Advisory Committee
•Subject matter experts
•Experienced practitioners who are familiar with the
knowledge, skills, and abilities needed by the target
users (examinees / practitioners / educators /
influential individuals in the field).

15. 5 Step Process
•1. Information gathering
•2. Initial meeting and draft development
•3. Confirmation and review
•4. Data gathering and compilation
•5. Finalizing the blueprint

16. 5 Steps Process
1. Information gathering
•Compile background materials:
• Teaching curricula
• Job analyses
• Relevant articles
• Training manuals
• Milestones and required competencies
• Blueprints for similar existing exams
• Any professional standards
Note: this should be a collaborative process between the
client and the “national board.”

17. 2. Initial meeting and draft
development
•The advisory committee convenes at a one-day
meeting to go over the background materials and
produce a draft of the content areas and relative
weights that will make up the test blueprint
(content specifications).
•The activities of the meeting are:
• Presentation/orientation by the psychometrician,
explaining the tasks of the day.
• Review and discussion of the background materials by the
advisory committee.

18. 2. Initial meeting and draft
development (cont.)
•The activities of the meeting are (cont.):
• Discussion and definition of the major areas of content to
be included in the examination. (Test Development staff ensure
the resulting draft blueprint meets their required guidelines.)
• Definition of specific sub-areas within each major area on
which questions should be written.
• Assignment of weights to each major and sub-area in
terms of percentage and number of items to be included
in the test from that area (more important areas will have
higher percentages of items).
• Review and finalization of the draft of the blueprint.

19. 3. Confirmation and review
•Send the blueprint draft assembled by the advisory
committee to a group of subject matter experts for
suggestions, comments, critiques, alternatives, and
revisions of points and weights.

20. 4. Data gathering and compilation
•Consolidate the results of the survey into a single
document and send to the advisory committee for
review.

21. 5. Finalizing the blueprint
•The advisory committee discusses the survey
results and finalizes the blueprint (via a series of
conference calls or other means.)
•This blueprint will then be used to analyze any
existing item pools and to establish areas where
items need to be developed, as well as guide the
assembly of the examination.

22. Practice analyses
(Two parts)
•An initial data generation phase with a focused
group of experts (e.g., blueprint development
process utilizing an advisory committee)
•A confirmation phase to validate the information by
a larger group of stakeholders.
Note: This validation phase typically incorporates a
survey sent out to identified individuals.

23. Necessity of the confirmation
phase
•A small group of experts may be biased
•The initial data generation phase is based on human
judgment rather than objective reality, and is
therefore not immune to the influence of raters
who may be subject to numerous sources of
inaccuracy
Note: Most practice analyses include this subsequent
questionnaire step to have a larger group of outside experts
review the materials created by the initial advisory committee
during the data generation phase.

24. Survey
• Members of the expert survey group are selected
by the advisory committee as a broader coalition of
stakeholders with the necessary expertise to
provide input into the blueprint.
•The expert survey group also serves as a vehicle to
create buy-in from the larger professional
community.
•The survey itself provides the proposed blueprint
along with instructions to the group to provide
input. It should be stressed that the blueprint needs
to reflect the content intended to be tested.

25. American Board of Internal Medicine (ABIM)*
Maintenance of Certification (MOC) Exam Blueprint
• In 2015, ABIM invited certified general internists to provide
ratings of the relative frequency and importance of blueprint
topics in practice.
• A sample of over 300 physicians, similar to the total invited
population of internists in age, time spent in direct patient
care, and practice setting, provided the blueprint topic
ratings.
• ABIM considered the average respondent ratings of topic
frequency and importance in each of the content categories.
• A second source of information was the relative frequency of
patient conditions in the content categories, as seen by
certified Internists and documented by national health care
data.
*ABIM: MOC Exam Blueprint, 2015

26. Purpose of the
Internal Medicine MOC exam
•Evaluate whether a certified internist has
maintained competence and currency in the
knowledge and judgment required for practice.
•The exam emphasizes diagnosis and management
of prevalent conditions, particularly in areas where
practice has changed in recent years.

27. Exam format
•The exam is composed of 240 single-best-answer
MCQs, of which 40 are new questions that do not
count in the examinee’s score.
•All questions describe clinical scenarios and ask
about the work done by physicians in the course of
practice:
• Diagnosis
• Testing
• Treatment/ Care Decisions
• Risk Assessment/ Prognosis/ Epidemiology
• Pathophysiology/ Basic Science

29. How the blueprint ratings are used
to assemble the MOC exam
• Blueprint reviewers provided ratings of relative
frequency in practice for each of the detailed content
topics in the blueprint and provided ratings of the
relative importance of the topics for each of the tasks
described.
• In rating importance, reviewers are asked to consider
factors such as the following:
• High risk of a significant adverse outcome
• Cost of care and stewardship of resources
• Common errors in diagnosis or management
• Effect on population health
• Effect on quality of life

30. Frequency and Importance
• Rated on a three-point scale corresponding to:
•low, medium, or high
• parameters for selecting MOC exam questions:
• At least 75% of exam questions will address high-importance
content (indicated in green)
• No more than 25% of exam questions will address medium-
importance content (indicated in yellow)
• No exam questions will address low-importance content
(indicated in red)
• Independent of the importance and task ratings, no
more than 18% of exam questions will address low-
frequency content (indicated by “LF” following the topic
description).

33. In Conclusion
• Blueprinting of a national examination is extremely
important
• It might involve up to 5 steps and 2 committees
(Advisory and Subject matter experts)
• Practice analyses includes two parts: an initial data
generation phase, and a confirmation phase (to validate
the information (by survey)
• For the initial data generation phase, patient issues and
problems could be utilized from national databases
(e.g., Medicare, ambulatory care database) or any
billing database
• Ratings for relative frequency and relative importance
are essential considerations

34. Bibliography
• Connell, A. F., & Nord, W. R. (1996). The bloodless coup: The infiltration of organization
science by uncertainty and values. Journal of Applied Behavioral Science, 32, 407-427.
• Goldstein, I. L., Zedeck, S. & Schneider, B. (1993). An exploration of the job analysis-content
validity process. In: N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations.
San Francisco, CA: Jossey-Bass.
• Knapp, J. E., & Knapp, L. G. (1995). Practice analysis: building the foundation for validity. In:
J. C. Impara, (Ed.), Buros-Nebraska series on measurement and testing. Lincoln, NE: Buros
Institute of Mental Measurements, 93-116.
• Morgeson, F. P, & Campion, M. A. (2004). A framework of potential sources of inaccuracy in
job analysis. In: M. Wilson, R. Harvey, G. Alliger, &W. Bennett (Eds.), The handbook of work
analysis: The methods, systems, applications, and science of work measurement in
organizations. New York: Psychology Press/Taylor and Francis Group.
• Newman, L. S., Slaughter, R. C., & Taranath, S. N. (1999, April). The selection and use of
rating scales in task surveys: A review of current job analysis practice. Paper presented at
the meeting of the National Council on Measurement in Education, Montreal, Canada.
• Raymond, M. R. (2002). A practical guide to practice analysis for credentialing
examinations. Educational Measurement: Issues and Practice, 21, 25-37.

35. 2018 Perspectives Symposium
January 22, 2018
#TSINPerspectives2018