1. Jeff Grann
George Lucas
23nd Annual Conference, Thayer Reed
Denver, CO Jennifer Stephens
October 28, 2011 Moderator: Patricia Case
LEARNING ASSESSMENT SYSTEMS THAT WORK!
A FOLLOW-UP REPORT

2. Capella University
• Founded in 1993
• Charter AQIP participant in 2000
• Courses 100% online
• For-profit university
• Enrollment: 39,447 (average age 39)
• Doctoral: 31%
• Master’s: 47%
• Baccalaureate: 21%
• Certificate: 1%
• Enrollment status: 94% part-time
• Faculty: 1,398 (82% doctoral degree)
© 2008 Capella University - Confidential - Do not distribute 2

3. Value Creation Model
© 2008 Capella University - Confidential - Do not distribute 3

4. Academic Program Definition
© 2008 Capella University - Confidential - Do not distribute 4

5. © 2008 Capella University - Confidential - Do not distribute 5

6. © 2008 Capella University - Confidential - Do not distribute 6

7. Online Course Design and Development
© 2008 Capella University - Confidential - Do not distribute 7

8. CRITERION
© 2008 Capella University - Confidential - Do not distribute 8
LINK

11. Fully-Embedded Assessment Model
• Extends outcome assessment throughout program to
develop actionable leading indicators.
• Documents alignment of all scoring guide criteria to
course competencies and program learning outcomes.
• Projected university completion by Q4 2012.
© 2008 Capella University - Confidential - Do not distribute 11

12. FEAM Worked Example
Principles of Organizational Communication
Course Competencies
1. Analyze business communication situations.
2. Analyze the interrelationships of communication within
organizational systems.
3. Communicate effectively.
u03a1 Scoring Guide Criteria
Post-FEAM
Pre FEAM
1. Tailors coachingadvice. on analysis of a direct report’s motivational
Provides Henry based
2. needs. (3)
Identifies Henry's educational/training needs in regard to
2. Validates approach relative to the theories of motivation. (1)
communication competency.
3. Predicts stakeholder consequences communication and motivation.
Compares the relationship between of communication. (2)
4. Identifies educational/training needs based on limited work sample. (1)
Applies the theories of motivation.
5. Analyzes peer evaluations relative to communication- Confidential - Do not distribute (2)
Applies the 10 related evaluations. © 2008 Capella University
competencies. 12

13. © 2008 Capella University - Confidential - Do not distribute 13

14. Make learning matter broadly
• Implementing a fully-embedded assessment model
(FEAM) has proven effective in:
• Promoting a coherent learning experience.
• Faculty collaboration.
• Students have a shared basis to view their learning
experience:
• As a long-term effort to demonstrate professionally-relevant
learning outcomes, not just a series of assignments and
courses.
• Faculty have a shared basis to establish a community of
practice around their program’s learning outcomes:
• By sharing professional experiences, teaching
practices, instructional strategies, performance
expectations, technical innovations, and illustrative case
© 2008 Capella University - Confidential - Do not distribute 14
studies.

15. WHO WE ARE
“Educating Those Who Serve”
American Public University System has
been dedicated to educating those who
serve since our doors first opened in
1991. The American Public University
System (APUS) is a fully online system,
encompassing both American Military
University and American Public
University. We are regionally accredited
by the Higher Learning Commission (HLC)
of the North Central Association.
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

16. 2011: Celebrating 20 Years of Service and Leadership
Mission: To provide quality higher education with emphasis on educating the nation’s military
and public service communities by offering respected, relevant, accessible and
affordable, student-focused online programs, which prepare them for service and leadership in
a diverse, global society.
STRATEGY:
• Founded as American Military University (AMU) in • Focus on quality and affordability
1991 by James P. Etter, a Marine Corps Officer
• Graduate successful alumni who
make important contributions to
• In 2002, AMU expanded into the American Public society and their professions
University System (APUS), adding American Public
University (APU) • Enhance America’s
competitiveness by meaningfully
• Nationally Accredited (DETC) in 1995 increasing the population with
college degrees
• Regionally Accredited (NCA) in 2006
• Collaborate with other
universities to create solutions to
• Serving over 96,000 students worldwide U.S. higher education challenges
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

17. “Quality begins on the inside . . . and then
Data Driven Culture
works its way out.” -Bob Moawad
• Internal and
external
benchmarking
• Data tools for
stakeholders -
dashboards, prog
ram and course
level fact
books, data
warehouse, stude
nt learning
assessment
reports.
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

18. NATIONALLY BENCHMARKED INSTRUMENTS
APUS utilizes a number of nationally benchmarked tests and validated
instruments to measure student engagement, student satisfaction, and
the achievement of student learning outcomes.
National
Community
Survey of Proficiency Major Field
of Inquiry
Student Profile Tests Tests
Survey
Engagement
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO 18
SERVE

19. Direct and Indirect Measures of Assessment
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

20. Regular Dissemination of Data to Stakeholders:
Dashboards
“Experts often possess more data than
judgment.” – Colin Powell
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

21. Regular Dissemination of Data to Stakeholders:
Course and Program Level Fact Books
“In the spirit of science, there really is no such
thing as a "failed experiment." Any test that
yields valid data is a valid test.” - Adam Savage
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

22. Student Learning Assessment Report:
Alignment of Student Learning Outcomes
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

23. Student Learning Assessment Report:
Culminating Experience
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

24. Student Learning Assessment Report: Key
Course Embedded Assessments
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

25. Student Learning Assessment Report:
Assessment Data
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

26. Student Learning Assessment Report: Checklist
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

27. Student Learning Assessment Report: Changes
Based on Data
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

28. PROGRAM REVIEW PROCESS
“Quality is never an “One of the great
accident; it is always mistakes is to
the result of high judge policies and
intention, sincere programs by their
effort, intelligent Data External intentions rather
direction and skillful Collection Review than their results.”
execution; it Milton Friedman
represents the wise
choice of many”
William A. Foster Follow-Up Analysis
Program
Three Year
Review
Plan
Meeting
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

29. Data Aggregation & Analysis
Library and Curriculum External
Learning Resources Assessment Reviewer Feedback
• Course books • Student learning • Expert reviewer report
• Electronic resources outcomes • Industry Advisory
• Instructional strategies Council report
• Learning strategies
• Evaluation procedures
• Academic rigor
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
SERVE

30. Data Aggregation & Analysis
Faculty Students Learning Outcomes
Assessment
• Analysis of faculty • Student demographic • Curricular Mapping
credentials and expertise information • Assessment measures
to ensure breadth and • Enrollment History
diversity • Fact books
• Growth trends • Student learning assessment
reports
• reports
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
SERVE

31. Data Aggregation & Analysis
Program Benchmarking Program Directory Review and Findings
Summary
• Program benchmarking • Evaluation of findings • Dean’s observations
with similar programs • Program recommendations • Meeting minutes
and institutions
• Three year proposed
strategic plan
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
SERVE

32. CONTACT INFORMATION
Jennifer Stephens Helm, Ph.D.
jstephens@apus.edu
Vice-President, Institutional Research and
Assessment
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

33. Kaplan University
• Offering online programs since 2001
• 58,000+ students online; 7,000+ students at campuses
• For-profit
• Certificates, Associates, Bachelors & Masters
• Incoming students have average of four NCES-identified
risk factors
• 1,000+ courses
• 4 tracks; 14+ term starts per year; overlapping terms
• Outcomes-based institution

34. So We do All this Assessment …
Measuring activities is useful
Example: How many curriculum changes did we make this year?
Assessing what is really going on is essential
Example: How many of those changes improved retention?
Requirement:
Sustainable infrastructure to support assessment approach
Result:
Meaningful analysis of data to generate actionable information

35. Assessment-Related Infrastructure
University Outside Vendor
Course Authoring Course Delivery
• All course content
• Curriculum development Reporting and Analytics
• Structured content Database
development and Outcomes Package
management
• Centralized framework
for consistency across How do we
courses know if we are
• Content available in a • Data collection on outcomes
variety of formats
improving • Real-time assessment data
learning? reporting to faculty
• Consistency of outcome and rubric
Outcome Repository • Assignments mapped to outcomes
and rubrics
• Single source of truth for
course outcomes and
rubrics
• Course to program mapping
of outcomes
• Course to course mapping
• Versions of all outcomes
and rubrics

36. Sustainable Systems

37. Information at the Course Level

38. Information at the Institutional Level
Example: Improvement as defined by improvement in
outcomes achievement levels and student retention
Source:
Thayer E. Reed, Jason Levin & Geri H. Malandra (2011): Closing the Assessment Loop by Design, Change: The Magazine of
Higher Learning, 43:5, 44-52

39. Assessment and
Academic Quality
at the Rockies
George Lucas
Director of Academic Quality

40. University of the Rockies
Private, independent graduate school of the
social and behavioral sciences.
Two schools including the School of
Professional Psychology, and the School of
Organizational Leadership
Master's and Doctorate degree programs
both on campus and online
Clinical Specialization at the campus

41. Academic Quality and Assessment
at the Rockies are grounded in
Learning Outcomes.
Institutional
Programmatic
Course

42. Institutional
Our Mission
Programmatic
Two Schools
w/Master and
Course Doctoral Programs
Our Curriculum
and Syllabi
Using established Learning Outcomes as a Foundation,
University of the Rockies has:
• Acquired application tools and resources,
• Incorporated and leveraged them in a coordinated manner,
• Aligned our Learning Outcomes, and (this quarter)
• Developed and Launched a Comprehensive Assessment Plan

43. What are the steps we’ve
taken to reach this point?
What will we do in the coming
months and in 2012?

44. Gates Model

45. Doctor of Psychology “Gates” Model for
Assessment
Post Doc
and Post
Internship Graduation
Licensure
Dissertation
Comprehensive Process
Practicum
1 Thru 5
Courses
(early, mid, late)
Multiple Assessment Methods align
Admission with the student experience and
Process learning context including but not
limited to Holistic Assessment,
Evaluations, Grades, Annual Review,
Licensure Exams, etc.

46. Gates Model
Curriculum Mapping

47. A successful and comprehensive Curriculum
Development Process
Learning Outcomes are identified in all
Curriculum with our Development Template
Curriculum Mapping and IRMA tables in

48. Gates Model
Curriculum Mapping
“Convergence”

49. Mapping, Outcomes Alignment
Usage Statistics
Quality
Assurance,
Curriculum
Integrated Support
Rubrics, Holistic
and Analytic
Assessment
Faculty Development/Accreditation Student Lifecycle

50. Gates Model
Curriculum Mapping
“Convergence”
Continuous Improvement

51. Cycle of Continuous Improvement
Assessment that is “done right” never ends!
www.ala.org

52. Gates Model
Curriculum Mapping
“Convergence”
Continuous Improvement

53. Fitting it all together…

54. Baseline Concepts of our Assessment Strategy:
Quality and Assessment belong together.
Our ongoing goal is to assess and continually
improve Institutional Quality
and improve the student experience.
To continually improve, you must continually:
Measure
Assess
Innovate
Evolve

55. University of the Rockies Assessment Plan:
Three dimensions:
• Academic Achievement
• Student Engagement
• Faculty Engagement
Seven (w/potentially more) categories
serving as data sources:
• Surveys
• Campus Vue
• WayPoint Outcomes
• eCollege
• Library Writing Center
• Curriculum Development
• Quality Matters ™

56. Assessment
and
Quality
“Coding Table”

57. Mapping, Outcomes Alignment
Usage Statistics
Quality
Assurance,
Curriculum
Integrated Support
Rubrics, Holistic
and Analytic
Assessment
Faculty Development/Accreditation Student Lifecycle

58. Jeff Grann
George Lucas
23nd Annual Conference, Thayer Reed
Denver, CO Jennifer Stephens
October 28, 2011 Moderator: Patricia Case
LEARNING ASSESSMENT SYSTEMS THAT WORK!
A FOLLOW-UP REPORT