Integrated Performance Assessment: Linking Individual Development to Industry Standards

Integrated Performance Assessment (IPA)
Linking Individual Development to Industry Standards
Tammé McCowin Doctoral Candidate
University of Phoenix

1
2007 International AECT Conference, 10/27/2007
© 2007 IPA Research Foundation Transforming Professional Development

Agenda

Topic Time
(Mins.)
Literature Review 10
ISD Performance Inventory 5
Scale Development Process 10
Peer Activity: Using the ISD Performance 40
Inventory
Research Participation 5
Questions and Answers 5

2

Literature Review
Topics
• Symptoms and Issues
• Problem Statement
• Research Purpose
• Research Question
• Literature Search
• Research Findings
• IPA Research Methodology

3

Symptoms and Issues
• Instructional technology (IT) professionals must
continually update and develop their skills in an ever-
changing world market due to global competition, new
methods and tools, fluctuations in the market, business
acumen, strategy, positioning, and technological
innovations (Lui, Gibby, Quiros, & Demp, 2002; Katz, 2005)

• There is a demand for IT professionals to continually
adapt to agile work environments while acquiring new
competencies in a rapidly changing & Hempstreet, 1998; Katz, 2005)
industry
(Curtis, Hefley, & Miller, 2002; Lui et al., 2002;
Lui, Jones,

4

Symptoms and Issues
• IT professionals “have a responsibility to keep their
skills current” (Rothwell & Kazanas, 2004, p. 386)
• It is an important and essential competency to update
and improve one’s knowledge, skills, and abilities
(Richey, Fields, Foxon, Roberts, Spannaus, & Spector, 2005)

• Skill utilization, extension, and versatility are critical
components for sustaining professional success and
technical professionals must continually extend their
knowledge and skill capabilities to grow, innovate, and
lead (Katz, 2005)

5

Problem Statement

• Effective job performance can be mitigated
when no means is available to accurately and
objectively assess the performance of IT
professionals on industry defined competency
standards

Instructional Technology (IT)

6

Research Purpose
• Career Planning and Professional Development
o To provide a valid and reliable measurement methodology
to guide career planning and professional development
activities
• Leadership Development
o To develop technical leadership by focusing on professional
competencies, maturation, experience, and context
• Selection and Guidance Counseling
o To provide an effective and objective measurement
methodology to aid employers and educators with making
better decisions about job and student applicants (e.g.
classification, placement, educational programming,
recruitment, and selection)
7

Research Question

• What are the valid and reliable competencies,
methods, and tools for assessing the preparation
and performance of IT professionals?

Why ask this question?
What someone thinks he or she is capable of doing, how others
know that he or she is capable, and how that capability may be
demonstrated is not always in alignment.

Instructional Technology (IT)

8

Literature Search
Number of References
9
9
8

7
26 reference sources found
6
6
5
5
Total

4
4

3
2
2

1
0
0
Q ue s t ia P ro / P s y ER IC D is s e rt a t io ns ETR &D P IQ

Literary Sources
9

Literature Search
Number of Research Studies
11
12

10

8
Total

6
18 research studies found

4
4

1
2
1 1
0
IS D P ra c t ic e CD MD TD Ot he r

Study Type
Competency Development (CD), Model Development, (MD), Tool Development (TD)

10

Study Findings
• ISD Practice Studies
o Software tools for empowering instructional developers (Gayeski, 1991)
o What do instructional designers actually do? An initial
investigation of expert practice (Rowland, 1992)
o Designer thinking: How novice and experts think about
instructional design (Perez & Emery, 1995)
o Context sensitive instructional design models: A response to
design theory, practice, and criticism (Wedman & Tessmer, 1995)
o The present and future of ID practice (Winer & Vazquez-Abad, 1995)
o Interactive multimedia design and production processes
(Lui & Hempstreet, 1998)

o A history of instructional design and technology: Part I: A
history of instructional media 2001a)
(Reiser,

11

Study Findings
• ISD Practice Studies
o A history of instructional design and technology: Part II: A
history of instructional design (Reiser, 2001b)
o The challenge of being an instructional designer for new media
development: A view from the practitioners
(Lui, Gibby, Quiros, & Demps, 2002)

o Paradigms in the theory and practice of education and training
design (Visscher-Voerman & Gustafson, 2004)
o Instructional design practice: Career environments, job roles,
and a climate of change (Larson & Lockee, 2004)

12

Study Findings
• CD Studies
o Roles and competencies of instructional design as identified by
expert instructional designers: a qualitative study* (Atchison, 1996)
o An examination of the instructional design competencies
written by the International Board of Standards for Training,
Performance, and Instruction* (Song, 1998)
o Instructional design competencies: The standards (3rd ed.)
(Richey, Fields. Foxon, Roberts, Spannaus, & Spector, 2001)

o Building a foundation for the tomorrow: Skill standards for
information technology 2003 edition (NWCET, 2003)

Competency Development (CD) * Denotes dissertation study

13

Study Findings
• MD Studies
o ID model construction and validation: A multiple intelligences
case (Tracey & Richey, 2007)
o The construction and validation of an instructional systems
design model incorporating multiple intelligences* (Tracey, 2001)
• TD Studies
o The validity of a multiple-choice, paper and pencil instrument
in discriminating between masters and nonmasters of
instructional design* (Stepp, 1990)
• Other
o The comparative study of information competencies-using
Bloom’s taxonomy (Ven & Chuang, 2005)
Model Development, (MD), Tool Development (TD) * Denotes dissertation study

14

IPA Research Methodology
assessed
Perceived demonstrated
1 2 3 4

ISD Performance Multiple Intelligences Integrated Skills Integrated
Inventory ISD Construct Assessment Performance
Study Map Study Study Assessment Study

RQ RQ1 H1 RQ
What are the valid and What is the relationship IT professionals who are What is the relationship
reliable competencies between MI Constructs able to correctly between perceived
for assessing the and ISD competencies? respond to a large assessed and
preparation and number of competency- demonstrated
RQ2
performance of IT intelligence items will performance?
What combination of MI
professionals? exhibit skill integration.
Constructs and ISD
competencies influence
H2
an IT professional’s skill
IT professionals who are
integration?
not able to correctly
respond to a large
RQ3
number of competency-
What combination of MI
intelligence items will
Constructs and ISD
exhibit skill imbalance.
competencies influence
an IT professional’s skill
imbalance?
Research Question (RQ), Hypotheses (H), Instructional Systems Design (ISD)

15

ISD Performance Inventory
Topics
• Scale Composition
• Rating Scale
• Sample Professional Foundation Scale

16

Scale Composition

Scale C PS SI
Professional Foundations 5 26 63
Planning and Analysis 7 30 59
Design and Development 6 32 58
Implementation and Management 5 34 56
Digital Media 5 35 63
Technical Writing 5 30 52
Web Development and Administration 8 55 92
Total 41 242 443

*Scale items were written to align to industry standards.

*International Board of Standards for Training Performance and Instruction (IBSTPI)
C - Competencies, PS - Performance Statements, SI - Scale Items
*National Workforce Center for Emerging Technologies (NWCET)

17

Rating Scale

Main Scale Equivalent Scale
1. Absolutely need 1. Novice
2. Strongly need 2. Advanced Beginner
3. Definitely need 3. Competent
4. Somewhat need 4. Proficient
5. No need 5. Expert

Rating oneself as an expert on an individual item does not imply that you cannot or
do not need to ever make improvements. Rather it is an indication of where you
believe you are to date regarding where you are in your career and the current
state of the industry and based on the standards that are relevant right now.

18

Sample Professional Foundations Scale
C1: Communicate Effectively in visual and oral form
Performance Scale Item Rating Scale
Statement
Create messages that When designing instruction, write learner centric content
accommodate learner
that is factual, clear, unambiguous, and tied to relevant 1 2 3 4 5
needs and
supporting visual aids to improve learning and
characteristics,
content, and comprehension of subject matter.
objectives When designing instruction, write instructional objectives
that clearly describe a measurable performance. 1 2 3 4 5

When designing instruction, write learner centric content
that maps directly to documented instructional 1 2 3 4 5
objectives.
Write and edit text to When designing instruction, write learner centric content
produce messages
that is clear, concise, and grammatically correct. 1 2 3 4 5
that are clear,
When designing instruction, edit learner centric content
concise, and
grammatically that is clear, concise, and grammatically correct. 1 2 3 4 5
correct.
1 – Absolutely need, 2 – Strongly need, 3 – Definitely need, 4 –Somewhat need, 5 – No need

19

Scale Development Process
Topics
• Four Step Scale Development Process
• Construct Delineation
• Item Generation
• Pilot Studies
• Validation Studies
• Scale Sample Sizes

20

Four Step Scale Development Process

1 2 3 4

Construct Item
Pilot Test Validation Study
Delineation Generation

Create higher order Develop item pool Selects small study Select random sample
factors group samples
Create response format Administer instrument
Create factors Administer instrument to samples and collect
Conduct expert reviews to samples and collect data
Create variables data
Conduct factor and
Create control variables Conduct factor and reliability analysis
reliability analysis
Conduct concurrent
Refine instrument validity analysis

21

Factor-Item Structure Relationships
1 2 3

Scales
Professional Foundations
Planning and Analysis
Knowledge Higher Ordered
= = Design and Development
Domains Factors
Implementation and Management
(7) (7) Digital Media
Technical Writing
Web Development and Administration

Competency
Factors ISDC
= =
Statements
(41) (41)
(41)

Performance
Variables
= =
Statements Scale Items
(242)
(242) (443)

Performance Statements (PS), Instructional Systems Design Competencies (ISDC)

22

Item Generation
Nine-Step Review Process

Qualitative Review
Step 1: Determine statement’s behavioral characteristics
Step 2: Determine statement’s unitary value
Step 3: Determine statement’s clarity
Step 4: Determine statement’s duality focus
Step 5: Determine statement’s functional purpose
Step 6: Determine statement’s face validity

Quantitative Review
Step 7: Determine inter-rater reliability of initial item pool
Step 8: Determine statement’s measurement type
Step 9: Determine statement’s skill level

23

Pilot Studies
• Two sample groups will be Group Samples
pulled to complete pilot Phase One Instrument Development
studies • Study 1: 10-20 participants
• Study 2: 10-20 participants
• Group 1 will be used to
test the skill inventory for
reliability and factor
structure relationships
• Group 2 will be used to
refine the skill inventory
for construct validity and
to further confirm factor
structure relationships
24

Validation Studies
• There are two sample Group Samples

study groups needed to Phase Two Instrument Validation
• Study 3: 276-460 participants
complete validation • Study 4: 75 participants
studies
• Study 3 is to finalize the
All participants who register to
skill inventory and confirm participate will be randomly placed
in one or more study groups. Email
test-retest reliability communication will be sent to you at
the start of study 1 to explain the
• Study 4 is to establish details and list which group or
groups you have been randomly
concurrent and predictive placed in.
validity

25

Scale Sample Sizes
Scale C PS SI Sample
Professional Foundations 5 26 63 189-315
Planning and Analysis 7 30 59 177-295
Design and Development 6 32 58 174-290
Implementation and Management 5 34 56 168-280
Digital Media 5 35 63 189-315
Technical Writing 5 30 52 156-260
Web Development and Administration 8 55 92 276-460
Total 41 242 443 276-460

The sample size of a scale is generally five times the size of a scale
(DeVellis, 2003; Netemeyer, Bearden, & Sharma, 2003)

*International Board of Standards for Training Performance and Instruction (IBSTPI)
C - Competencies, PS - Performance Statements, SI - Scale Items
*National Workforce Center for Emerging Technologies (NWCET)

26

Peer Activity:
Using the ISD Performance Inventory
Topics
• Reviewer Types
• Instructions to Self Raters
• Instructions to Third-party Raters
• Peer Activity: How to Use the ISD
Performance Inventory

27

Reviewer Types
• Self Rater
– An individual who voluntarily uses the ISD
Performance Inventory to rate themselves to obtain
performance feedback
• Third-party Rater
– An individual who voluntarily uses the ISD
Performance Inventory to rate another person to
provide performance feedback (e.g. peer, teacher,
manager, team member, etc.)

28

Instructions to Self-Raters
• You will be presented with items for each scale
• Rate the level of professional development you
require on each item using the rating scale
• Use the ISD Skill Level Response Format
Descriptions to judge your professional
development on each item relative to the rating
scale

29

Instructions to Third-party Raters
• Use the interview rubric to interview your partner
• Rate the level of professional development your
partner requires on each item using the rating
scale
• Use the ISD Skill Level Response Format
Descriptions to judge his or her professional
development on each item relative to the rating
scale

30

Peer Activity
• Select a partner and
How to Use the ISD
determine your role (5-mins) Performance Inventory
• Third-party rater interviews C1: Communicate Effectively in visual and oral form

his or her partner (15-mins) Performance Scale Item Rating Scale
Statement
Create messages that When designing instruction, write learner centric content
accommodate learner
that is factual, clear, unambiguous, and tied to relevant 1 2 3 4 5
needs and

• Self rater and third-party
supporting visual aids to improve learning and
characteristics,
content, and comprehension of subject matter.
objectives When designing instruction, write instructional
objectives that clearly describe a measurable 1 2 3 4 5
performance.

raters complete the
When designing instruction, write learner centric content
that maps directly to documented instructional 1 2 3 4 5
objectives.
Write and edit text When designing instruction, write learner centric content
to produce messages
that is clear, concise, and grammatically correct. 1 2 3 4 5
that are clear,

inventory (10-mins)
When designing instruction, edit learner centric content
concise, and
grammatically that is clear, concise, and grammatically correct. 1 2 3 4 5
correct.

• Compare and share findings
(5-mins)
• Reporting and analytics (5-
mins)
40-mins

31

Research Participation
Topics
• Eligibility Requirements
• Participation Benefits
• How to Participate

32

Eligibility Requirements
• Anyone who falls into one of the instructional
design roles or both and has performed the
functions of any of these roles

Instructional Designer Role Instructional Developer Role

analyst, course developer, content editor, content developer, eLearning developer,
eLearning specialist, electronic publisher, instructional developer, information
evaluator, communications, designer, architect, information developer,
creative writer, document specialist, programmer, project manager, media
instructional designer, interaction designer, designer/specialist, multimedia
online publisher, project manager, technical author/specialist, multimedia developer,
communicator, technical writer, trainer, training developer, user interface developer,
training designer/specialist, usability web developer, web media developer,
designer, user interface designer, web technologist, teachers, students
designer, teacher, students

33

Participation Benefits
• When you register and fully participate in the
research study you will
– Receive your choice of one online training course to
promote professional development toward linking your
skills to industry standards
– Contribute to the research and validation of a online
for performance assessment tool professional
development
– Earn a chance to win either one 30GB video iPod or
one of two 4GB iPod Nanos

34

How to Participate
• To participate you must:
o Register at the research website, provide your name, email
address, contact number, and some general demographic
information
o Acknowledge and give your formal consent to participate
by selecting the terms and conditions check box on the
registration form
o Finalize your registration by clicking the link in the
confirmation email that will be sent to you

http://www.iparesearch.org

http://iparesearch.blogspot.com

Let’s transform professional development together!
Share the link with your colleagues and friends!
35

Questions and Comments?

36

Integrated Performance Assessment: Linking Individual Development to Industry Standards

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (6)

Semelhante a Integrated Performance Assessment: Linking Individual Development to Industry Standards

Semelhante a Integrated Performance Assessment: Linking Individual Development to Industry Standards (20)

Último

Último (20)

Integrated Performance Assessment: Linking Individual Development to Industry Standards