1. Matching E-Portfolios to Standardized
Tests for Value-Added Assessment
Richard Robles
EDLD 812: Data Improvement
2. About the University Honors Program
UC’sinvolvement in the Voluntary System of
Accountability
Comparing the Collegiate Learning Assessment to
e-portfolio artifacts
What we have learned thus far
3. University Honors Program
Revised the program during AY 2007-08 to focus honors activities on
experiential learning in the following thematic areas:
Community engagement
Global studies
Leadership
Research/Creative arts
Comprised of1,600 students from all of the colleges within the
University (except Raymond Walters and Clermont Colleges)
Last 2 entering cohorts represent about 25% of the population
Average entering ACT is 31 (SAT 1346)
4. Voluntary System of Accountability
Voluntary initiative for 4-year public colleges and
universities
Developed through a partnership between the AASCU
and APLU to help institutions
Demonstrate accountability and stewardship to public
Measure educational outcomes to identify effective
educational practices
Assemble information that is accessible, understandable,
and comparable
5. Voluntary System of Accountability
Ohio is the only state to have all state institutions
participating in the VSA
UC’s initial participation is the implementation of the
Collegiate Learning Assessment
6. Collegiate Learning Assessment (CLA)
Developed by the Council for Aid to Education (CAE) in
2004 – www.cae.org/cla
A standardized and nationally normed test that measures
the institutional contributions (value added) to the
learning gains made by students
Direct assessment of student learning
7. Collegiate Learning Assessment (CLA)
Areas of measurement
critical thinking
analytical reasoning / problem solving
written communication
Time required:
90 minutes for a performance task (complex problem to be
solved)
75 minutes for two writing prompts (make an argument and
break an argument)
8. The CLA’s Range
First-year students and then seniors
Longitudinal study is better
Cross-sectional study is possible (but more expensive)
The institution is the primary unit of analysis
Institutions can be compared for value-added gains
9. The “Dual Pilot” study at UC
To consider the applicability of learning portfolios (and
especially e-portfolios) alongside the CLA among a
group of first-year Honors students
Best strategies towards
The transition to become student scholars
The development of General Education baccalaureate
competencies over the first year
The need for intervention mechanisms to facilitate further
transitions
10. Methodology
Administer the CLA as a course requirement to 117 first-
year, first-time baccalaureate seeking, Honors students
Average ACT is 31
Representing a range of UC colleges and programs
Half took the CLA tests to “make an argument” and to
“critique an argument,” with the other half doing the
problem-solving “performance task”
11. Additional information for gathering
Level of effort exercised in completing each instrument
Sense of engagement with this process
Sense of learned perceptions from completing the CLA
test
Reactions to and self-assessment of their performance
Role of any “ceiling effect”
12. E-Portfolio Artifacts
Two essays posted in the e-portfolio with
Focus on personal academic development
Focus on designing a experiential learning project
Use of AAC&U metarubrics
Integrative Learning
Critical Thinking
Written Communication
13. PreliminaryResults:Overall Impression
I believe the test is a waste of time and that it 6 6.6%
would not be at all useful in testing students’
abilities
I believe the test would be only mildly useful in 40 44%
testing students’ abilities
I believe this test would be useful in testing 36 39.6%
students’ abilities
I believe the test would be very useful in testing 9 9.9%
students’ abilities
I believe this test is one of the most useful tests I 0 0%
have taken and that it will provide outstanding
insight into students’ abilities
14. Preliminary Results: Testing
Experience
Taking this test was a very unpleasant experience 9 9.9%
Taking this test was a somewhat unpleasant 25 27.5%
experience
Taking this test was neither a pleasant nor 43 47.3%
unpleasant experience
Taking this test was a somewhat fun, interesting or 12 13.2%
pleasant experience
Taking this test was a fun, interesting or pleasant 2 2.2%
experience.
15. Preliminary Results: Effort Expensed
Almost no effort at all 2 2.2%
A little effort 5 5.5%
Some effort 36 36.9%
Very much effort 41 45.1%
Serious and sustained effort 7 7.7%
16. Conclusions Thus Far
Questionable reliability of results
Lack of integration into the curriculum
Disconnect between the CLA and UC’s mission
Absence of ownership by the faculty
Questionable motivation by seniors taking the test
High costs to administer the test (over $100 per student)
17. Matching E-Portfolios to Standardized
Tests for Value-Added Assessment
Richard Robles
EDLD 812: Data Improvement
Notas do Editor
SurveyMonkey survey: 1. Please rate your overall impression of the validity of the Collegiate Learning Assessment(CLA)in providing feedback about students’ abilities in such areas as critical thinking, problem solving, and effective communication:
3. Using a five-point scale in which a “5” means “serious and sustained effort” and a “1” means “almost no effort at all,” how would you rate the level of effort that you put into taking this test?