1. LukSuS project
Helsinki, 30 November 2013
Helsinki, 30 November 2013
LukSuS project
Large-scale Assessment of Oral
Language Proficiency
—
A Swedish Example
Gudrun Erickson
University of Gothenburg, Sweden
Gudrun.Erickson@ped.gu.se
Overview
Context – “Facts about Sweden”
Why? – Purpose(s)
What? – Construct
How? – Methods & Models
When? – Timing and Sequencing
Who? – Actors
And…? – Uses, Consequences, ‘Issues’…
EALTA
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
1

2. LukSuS project
Helsinki, 30 November 2013
One of the protagonists about oral testing
It’s always good to be able to
pronounce words. If you’re in
England for example, you can’t
go around and write notes to
everybody you see…
Swedish 15-year-old
about the oral test of English for grade 9
The Swedish School System
• 9-year compulsory school; 98 % continue to three-year upper
secondary education (c. 75 % graduate)
• National curricula and syllabuses for subjects
• Grading from year 6 (students 11-12 years of age)
• Goal/criterion referenced grading system; six grade levels (A-E + F)
• Teachers responsible for marking national tests, and for final grading
• Grades high stakes – used for selection for higher education
• Extensive national testing programme – advisory function (however,
quantitatively undefined) – No exams
• 4 mandatory tests of EFL: ages 12, 16 // 17, 18 (approx. A2, B1, B1+, B2)
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
2

3. LukSuS project
Helsinki, 30 November 2013
[Current] Aims of national tests
• To enhance fairness and equity in assessment and grading
• To provide data for local, regional and national quality control
“The system may also contribute to”
• clarifying and operationalizing the national syllabus
• enhancing students’ learning and attainment of national objectives
A collaborative approach to test development
•
Analyses of relevant literature and research
•
Continuous work in broad groups of experts (teachers, teacher
educators, special needs teachers, researchers, L1 speakers…)
•
Small-scale piloting > Adjustments (an iterative process)
•
Large-scale pre-testing in randomly selected groups of students in
the country (n ≈ 400)
•
Analyses of results and of students’ and teachers’ views and
suggestions
•
Compilation of tests (reference groups)
•
Standard setting and benchmarking in broad groups
•
Analyses / Research / Reporting
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
3

4. LukSuS project
Helsinki, 30 November 2013
Focal areas in the national syllabuses of FL
A functional, action oriented approach (cf CEFR)
• Receptive competence
(listening and reading)
• Productive competence
(speaking and writing)
• Interactive competence
(speaking and writing)
• Intercultural communicative competence
• Strategies; Adaptation (to purpose, context & receiver)
The typical national FL test
Four subtest (LC & RC aggregated into one score)
• Oral production and interaction
• Reception: Listening comprehension
• Reception : Reading comprehension
• Written production (and interaction)
————————————————
• Strategies and Adaptation (to purpose, content and receiver) integrated
• “Culture”: an integrated aspect of test content and design
• Reflection, e.g. self assessment, focused upon in separate
sections/materials and/or in student profiles
• Results / Sub-scores presented in profiles
• Model provided for aggregation into total score
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
4

5. LukSuS project
Helsinki, 30 November 2013
Oral proficiency in the national syllabuses
(upper secondary level)
•
•
•
•
Clear distinction between production and interaction
Speaking and Writing combined
“CENTRAL CONTENT”: …Oral and written production and interaction of different
kinds, including more formal contexts, where students instruct, tell about things,
summarise, explain, comment, evaluate, justify their views, discuss and argue. –
Strategies for contributing to and actively participating in discussions related to
societal and working life. – Revision and modification of their own and others' oral
and written communication in order to vary, clarify and specify, as well as to create
structure and adapt it to purpose and situation.
PERFORMANCE STANDARDS: …In oral and written communication of different
genres, students can express themselves in relatively varied ways, relatively
clearly and relatively coherently. Students can express themselves with some
fluency and in a way that is to some degree adapted to purpose, recipient and
situation. Students revise and make improvements to their communications.
In oral and written interaction in different, including more formal, contexts, students
can express themselves clearly and with some fluency and some adaptation to
purpose, recipient and situation. In addition, students can choose and use broadly
functional strategies that will, to some extent, solve communication problems and
improve interaction.
The Development of oral tests of EFL in Sweden
• Offered as a complement to the national tests since the late 1980s
• Mandatory in grade 9 since 1998, in upper sec. since 2002
***
• One to one (teacher + student) 1991-1993
• Student interactive (pairs) + one to one 1994-1997
• Student interactive (paired orals) 1998- Production + Interaction;
Often three parts: Warm up > Production – Argue – Discuss (10-20 mins per
pair, depending on age, level etc.)
***
Interrogation… – Interview… – Interaction…
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
5

6. LukSuS project
Helsinki, 30 November 2013
Typical themes of oral tests
Year 5/6 (≈ A2)
• What Do You Think?
• The International Talk Show
Year 9 (≈ B1)
• The World Around Us
• Quality of Life
English 5 (≈ B1+)
• Doing the Right Thing
• Skills for Life
English 6 (≈ B2 )
• Integrity and Technology
• Life Balance
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
6

7. LukSuS project
Helsinki, 30 November 2013
Holistic rating supported by analytical factors
• CONTENT
Clarity – Complexity and variation (different examples and perspectives) –
Coherence and structure – Adaptation to purpose, recipient, situation
and genre
• LANGUAGE AND /SELF/ EXPRESSION
Communicative strategies (to develop and advance a conversation; to solve
linguistic problems, e.g. by rephrasing, explaining and clarifying) –
Fluency and
ease – Range, variation, complexity, clarity and accuracy (vocabulary,
phraseology and idiomatic expression; pronunciation and intonation; grammatical
structures) –
Adaptation to purpose, recipient, situation and genre
Materials to support teachers
• General guidelines
• Performance standards
• Analytical factors
• Authentic, graded benchmarks with extensive
comments related to the syllabus, standards and
factors
[Year 6: Extensive training material]
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
7

8. LukSuS project
Helsinki, 30 November 2013
Some results ( % )
Grade level Year 6
Year 9
En 5
En 6
F
5
3
4
1
>E
78
81
79
86
A
18
18
16
18
Teacher Questionnaire (oral test)
Good/Ok test
98
95
95
99
Recording
46
46
49
50
Correlations between raters
NB: Based on benchmarking data
Average
Lowest
Highest
Year 6 (11)
.75
.53
.95
Year 9 (12)
.90
.82
.97
E 5 (10)
.85
.59
.98
E 6 (8)
.80
.53
.93
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
8

9. LukSuS project
Helsinki, 30 November 2013
Co-Rating ?
The teacher as the only rater
Subtest
Year 6
Year 9
En 5
En 6
A (oral)
58
52
55
57
B (rec)
44
42
57
57
C (writ)
17
10
13
11
Opinions about oral tests & testing
• Topics good and relevant / Nice things to talk about
• Positive reflection of the syllabus / Important to know/learn
• Students really enjoy doing it / Fun to do!
****
• Practical difficulties; time consuming
• No ‘visible’ time for co-rating
• They never stop talking!
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
9

10. LukSuS project
Helsinki, 30 November 2013
ISSUES…
• Practicalities (incl. time)
• Teacher present or absent?
• Recording?
• Co-rating – how?
• Interlocutor/Partner effects
• Preparation time?
• Type of language – tasks – content…
• Balance production – Interaction
• Can teachers’ ratings be trusted – in general + of their own
students’ performances…?
EALTA* Guidelines for Good Practice
in Language Testing and Assessment
Three categories/“audiences”:
• Teacher pre- and in-service training
• Classroom assessment
• Test development in national or institutional
testing units or centres
Basic principles the same for all categories:
Respect for students/examinees, responsibility, fairness,
reliability, validity, collaboration among the parties involved
* European Association for Language Testing and Assessment
www.ealta.eu.org
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
10

11. LukSuS project
Helsinki, 30 November 2013
European student 2005:
“I think that a good language test/ assessment
should get students chance to show what they
know and devolope their skills. It should contain
of listening and reading comprehations, as well
as writing and grammar part. But the most
important thing is to speak in that language, so I
think there should be also oral exams. Speaking
is the most important, because without it, we
wouldn’t communicate. Because of that students
should also learn pronunciation.”
“ Now have I not so much more
to tell you… ”
!
#
Gudrun Erickson
University of Gothenburg, Sweden
gudrun.erickson@ped.gu.se
  #
!
THANK YOU !
TACK FÖR MIG !#
11