Presentation from the Bonner Assessment Institute by Kristen Norris

1. Designing Effective & Measureable
Student Learning Outcomes
2015 Bonner Assessment Institute
Kristin Norris, IUPUI

2. Goals for Today
• Build capacity to determine WHAT you need to assess, WHY, and how does that
translate into outcomes
• Align student learning outcomes with course/program activities and assessment

3. Getting a pulse….

4. Getting a Pulse
• On a scale of 1-10, how confident are you that….
1. Have developed measurable outcomes for your course/program?
2. Have created processes that provide good evidence of that learning outcome?
3. Are able to assess the student learning?
4. Based upon evidence, know what to do differently in your course/program in order to increase student
learning?
• Why are you/are you not confident? What is causing you to question?

5. Why Outcomes and Assessment Matter
•Commonly asked questions
•Getting beyond ‘doing good’
•Intentional decision making
•Strategic plan and civic mission of institutions

6. Importance of Good Questions
•What do you need to know and why?
•Refining your questions
•Your question should dictate everything (your plan, your methods, your
approach, your measures)

7. Rigor in Research (Assessment)
• Rigorous thinking
• Hypothesis exploration (Starts with a good question)
• Systematic approach
• Builds upon prior research
• Contributes to the field
• Triangulation across inquiry
• Creates useful, relevant knowledge
(Patton, 2012)

8. Schoen’s Reflective Practice
•Trait of a good professional
•Reflection in-action
•Attentive to new knowledge
•Curious, test ideas, adjust practices
•Revise plans and practices - why are you doing this?

9. Language

10. TERMS
• Measure/Metrics
• Assessment
• Evaluation
• Research
NOTE: There are no clear or “singular” definitions utilized across community engagement (CE),
which makes it even more important for us to have a clear understanding of how we define our work.

11. Measures/Metrics
… it’s about the outputs; how can CE be measured or quantified?
Main Example in Higher Education:
numbers or “measurements” surrounding
“performance indicators”
Examples of Measurements:
• # and % of faculty/staff/students involved in CE
• # of internships in the community
• $$ economic impact of CE

12. Assessment
…it’s about the outcomes; what are we trying to change or impact through CE?
Main Example in Higher Education:
student learning outcomes
Examples of Assessment:
• What do students learn through CE experiences?
• How does CE influence students’ development?
• How does CE bridge “us v. them” with the community?
• What skills are developed through CE experiences?

13. Evaluation
…it’s about the practice itself; is CE effective, efficient, of a certain quality, etc.?
Main Example in Higher Education:
program and/or teacher evaluation
Example Evaluation Questions:
• What is a high quality service-learning experience?
• Are CE programs cost-effective?
• How do CE programs meet their objectives (i.e. outcomes and outputs)?

14. Research
…it’s about being more generalizable, per se
• Institutional Research
• Higher Education Research
• Disciplinary Research
• Community-Engaged Research (CER)
• Community-Based Research (CBR)
Examples of research:
• Institutional: how does IUPUI do CE?
• Higher Education: how does HE do CE?
• Disciplinary: how does Geography do CE?
• CBR: How does one get a classroom engaged in the community?
• CER: How does one engage with the community in research?

15. Outputs, Outcomes, Impact
Inputs Outputs Outcomes Impacts
All the resources put
into the project to
enable the delivery
of outputs
All the activities
undertaken and
products and
services delivered
The changes,
benefits, learning or
other effects that
result from the
outputs
The effect of a
project at a higher or
broader level, in the
longer term, after a
range of outcomes
have been achieved

16. Starting with Outcomes in
Mind
What do you want to assess?

17. Where do your CE assessment interests lie?
Institution
School
Department
Course
Project 1 Project 2
Program of
Study
Course
Project
Center/Office
Program
Project 1
Project 2
1
2
3
4

19. Recommended Resource: http://ccoe.rbhs.rutgers.edu/forms/EffectiveUseofLearningObjectives.pdf
Outcome Statements Should…
• Describe what students should be able to demonstrate, represent, or produce based on their
learning histories;
• Rely on active verbs that identify what students should be able to demonstrate, represent, or
produce over time. (Maki, 2004)
• Exist at different levels:
• Lesson/Activity
• Program
• School/Dept/Unit/Center (multiple programs)
• Institutional
• ***Should be able to map your outcomes across all of these****

20. Why are student learning outcomes
important?
• At the course level
• Build a foundation for your course
• Define your expectations
• Describe how your course is relevant to the students’ personal, academic, and/or professional development
• Keep you focused on what is important
• Enable students to articulate what they are learning and have learned.
“Students will role play the problem-solving, communication, and teamwork skills necessary to effectively
resolve real-world hotel management scenarios.”

21. Why are student learning outcomes
important?
• At the unit/lesson level
• Build a structure for a unit or session
• Focus each unit or session
• Give learners a clear picture of what to expect
• Provide criteria for constructing assessments
• Guide the selection of learning activities
• Teach learners how to be successful
“Given a problem with two unknowns, students will, in a step-by-step fashion, describe how to solve the problem.”

22. Components of Effective Outcomes
• The intended outcome clarifies what you are attempting to assess
• The intended outcome is measurable
• The intended outcome is useful and meaningful
• One or more methods of assessment can be explicitly tied to the intended outcome
• The CBC Method (three parts)
• Condition: Under what conditions?
• Behavior: What should they be able to do?
• Criterion: How (well) must it be done?

23. Activity
Identify issues with the learning outcome and recommend changes
for improvement

24. What recommendations would you make for improving the learning outcome?
•Outcome 1: Students have an increased ability to identify societal issues and community
needs.
•Outcome 2: Students have an increased understanding of opportunities for community
involvement.
•Outcome 3: Students deepen their understanding of the value of civic engagement.
•Outcome 4: Students have an increased motivation to remain civically engaged after
[insert university].
•Outcome 5: Students heighten their cultural awareness and ability to work in diverse
settings

25. Self-check Questions
• Is it clear what you are assessing?
• Is the intended outcome measurable?
• Active verbs, active verbs, active verbs
• Is the intended outcome measuring something useful AND
meaningful?
• How will this outcome be measured?

27. Process for Developing
Outcomes
IUPUI’s Process

28. General Hints, Tips, and Lessons Learned
•Doesn’t happen in isolation
•Iterative process
•The process is just as important as the product – creates shared
understanding and buy-in
•Overwhelming and potentially frustrating at first

29. Steps Taken at IUPUI
•Regularly scheduled meetings (monthly) – all staff
•Bi-weekly meetings we Dir. of Assessment and Program Director
•“Imagine you had $650,000 to develop civic-minded graduates. What would
that look like”
•Post-it notes – 3-4 word statements
•Group all of our ideas into categories

30. Example from IUPUI – Civic Identity
Definition: Students embrace their role as civic agents and as social trustees of knowledge
working with others to create change in the community
Outcome:
• Possess a sense of responsibility and commitment to use knowledge and skills gained through
college experiences to contribute to the greater good of society(social trustee)
• Make informed and principled choices that positively impact society (civic agency)
• Espouse and enact values, dispositions and beliefs that lead to positive societal change
• Articulate effectively how being civically engaged has fostered and informed their civic
knowledge, skills, dispositions, and future intentions

31. Example from IUPUI – Civic
Communication Skills
Definition: Students apply communication strategies to be effective in a community context, showing the ability to
express, listen, and adapt ideas and messages based on the diverse perspectives of others
Outcomes:
• Listen critically to effectively respond to complex situations
• Value the diverse perspectives of multiple stakeholders
• Demonstrate empathy towards others
• Understand the process, outcomes, and active participation in civil dialogue by applying civic communication skills
• Value reciprocity and consensus building
• Engage in an inclusive decision-making process that maximizes strengths and leads to mutually beneficial goals

32. Assessment Tools and
Strategies for Student Learning
Part 2

33. Goals for Part 2
•Data considerations: What do you need, does it already exist, how will you
collect it
•How will you report your findings? Who needs to know?
•Example tools and strategies

34. Context Matters When it Come to
Data

35. Activity 1
Getting to know your
inner OZ

37. When you think about the context for assessing community engaged activities
at your campus [within your department, program, project, etc.], which
character from the Wizard of Oz do you most identify with and why?
•Auntie Em/Uncle Henry
•Cowardly Lion
•Dorothy Gale
•Flying Monkeys
•Glinda the Good Witch
•Munchkins
•Ozmites
•Scarecrow
•Tin Man
•Toto
•Wicked Witch of the West
•Wizard

38. My/Our Allies
[R=real; P=Potential]
Predators
[R=real; P=Potential]
Individuals, Groups,
Policies, Systems
My/Our Strengths
[WD=Well Developed;
R=Room for Growth;
U=Untapped]
Local/Global
Threats
[R=Real;
P=Potential]
ACTIVITY 2

39. Developing an Assessment
Plan
Using the Matrix

41. Completing the Matrix
• Step 1: Insert your Learning Goals and Learning Outcome/Objective
(columns 1 and 2)
• Step 2: Identify course/program activities (e.g., reflections, dialogue, readings,
events, trainings) – which learning outcome do they align with?
• Note: If an activity doesn’t align with one of your learning outcomes, then why are you
doing it?
• Note: What happens if you’ve identified a set of program outcomes yet none of your
activities can be mapped to that outcome? You have a gap in your programming

42. Completing the Matrix (cont.)
• Step 3: Identify your sources of evidence (column 4)
• Brainstorm sources of evidence for the following examples:
• Value the diverse perspectives of multiple stakeholders
• Stay abreast of contemporary issues and make informed decisions using multiple
sources of information
KEY – Be intentional and strategic

43. Data

44. So you say you have data…..
Questions to consider:
• What data do you already have?
• What types of data are you collecting on a regular basis?
• How are you currently using your data? Are you using all of it?
• internal purposes vs. external purposes
• What are you still unable to do with your data? Why?
• Who decides what you collect, how you ask the questions, and how it is used?
• Do you need more data? If so, what data do you need?
• What lens(es) are you using when you consider these questions

45. Sources of Evidence of Student Learning
• Surveys (QR Code vs. email, timing)
• Capturing the discussion during reflection sessions
• Focus Group
• Observations
• Wordle
• Second-hand reflection (in the classroom the next day)
• Pictures
SYSTEMATIC

46. see Whitley (2014) in MJCSL
Additional Considerations: Potential Factors
Influencing Student Learning
• Faculty interactions
• Reflection
• Dialogue across difference
• Direct vs indirect service
• Project vs hours
• Student self-select partner or project vs assigned
• Community partner involvement

48. Closing the Assessment Loop
• If you are not using the data you are collecting – STOP
• Who needs this information and what do they care about?
What is the best way to communicate/tell your story?
• Need to know vs Nice to know
• Starts with a good question

49. Examples & Tools

50. IUPUI’s Civic-Minded Graduate
• Civic-Minded Graduate Scale (30-items)
• Civic-Minded Graduate Scale Short-Form (6-items)
• Civic-Minded Graduate Narrative Prompt
• Civic-Minded Graduate Rubric
• Civic-Minded Graduate Rubric 2.0 (see handout)
• AAC&U National Task Force

51. Already Existing Tools or Processes
•Faculty Annual Report
•Campus Climate Survey
•Alumni Survey
•Faculty/Staff Survey
•PULSE Surveys
•Initiatives; Course Tags
•Taxonomies