The second plenary on day 1 of our TEAM 2014 (testing, evaluation assessment masterclass) held at Meliksah University, Kayseri.

1. Principles of test design
Objectives, measurement qualities, challenges
KAYSERI, 29-31 JANUARY 2014

3. Aims
!
Definining our assessment objectives
!
Measurement qualities: ‚test usefulness
!
Testing challenges
!
Considering impact

4. Testing: a definition
!
“the systematic gathering of information for
the purpose of making decisions” (Weiss
1972)
!
“the collection of reliable and relevant
information” (Bachman 1990)
!
What information?
!
What decisions?

5. Defining our test objectives
1)
What kind of test is it going to be?
2)
What decisions do we need information for?
3)
What abilities are tested?
4)
How much detail do we need?
5)
How accurate should our results be?
6)
Is there any washback effect?
7)
What are the practical constraints?

6. Assessment in the classroom
! examinations
! formal
! oral
tests
tests
! home
! term
assignments
reports
! continuous
assessment
! projectwork,
! student
research work
self-evaluation

7. Purposes of tests
!
for placement or level testing: to decide on the level of
learners
!
for diagnostic testing: to identify individual strengths or
weaknesses
!
to evaluate progress
!
to evaluate skills for specific needs
!
to give marks for performance
!
to evaluate and update syllabus and objectives
!
also to prepare/train for exams

8. Types of tests (Madsen, 1983)
Contrasting categories of ESL tests
Knowledge
Performance (or skills)
Subjective
Objective
Productive
Receptive
Language sub-skills
Norm-referenced
Discrete-point
Proficiency
Communication skills
Criterion-referenced
Integrative
Achievement (or progress)

9. Key principles of testing
"
A correspondence between language test
performance and language use
◦
"
“In order for a particular language test to be useful for its
intended purposes, test performance must correspond in
demonstrable ways to language use in non-test situations.”
A clear and explicit definition of qualities of test
usefulness
◦
“Usefulness = Reliability + Construct validity + Authenticity +
Interactiveness + Impact + Practicality”
(Bachman & Palmer, 1996)

10. Qualities of test usefulness:
reliability
"
consistent across populations
"
consistent within the same test
irrespective of:
!
setting
!
marker
!
item set
When conditions of the test remain unchanged, it always produces
essentially the same results.

11. Qualities of test usefulness:
validity
! Types
of validity (Alderson et al, 1995)
! internal
! face
validity
validity
! content
validity
! response
! external
validity
validity
! concurrent
! predictive
! construct
!
validity
validity
validity
meaningful? appropriate? representative?

12. Construct validity
CHECKLIST FOR VALID TESTS
Tests what it claims to test?
Best format for what you want to test?
Content relevant to student’s real-life needs?
Items typical/representative of learner use?
Items don’t accidentally test other things?
Test skill, NOT knowledge, creativity or logic?
Clear and appropriate marking criteria?

13. Testing competence
from Language Testing in Practice by Bachman & Palmer (1996,
OUP)
Misconceptions
!
one best test for a given
situation
!
misunderstanding the nature
of testing and test
development
!
unreasonable expectations
!
blind faith in measurement
Resulting problems
!
inappropriate tests
!
failure to meet specific needs
!
uninformed use of popular testing
methods
!
frustration about not finding the
perfect test
!
loss of faith in self ! testing can
only be done by experts
!
defending the indefensible
(stakeholders s expectations)

14. Challenges in testing
"
"
"
"
"
"
testing = “real life”? “authentic”?
“communicative language testing”?
defining abilities in a test
selecting tasks and items ! competence?
measurement principles:
can “unidimensional scores for locally independent
test items reflect authentic language use”?
(Bachman)
outside factors ! performance?

15. Thank you!
zoltan.rezmuves@digimedia.hu
consonantvoiced.blogspot.com