This document describes a study that developed and tested an email request task to assess academic writing ability. The study aimed to examine the reliability of ratings on the email task and how it assessed writing differently than other tasks. It found high inter-rater reliability but that the email task did not clearly distinguish writing ability levels. The discussion proposes making the email task more complex, such as providing more input for students to respond to, in order to better assess writing constructs.

1. IMPLEMENTING AN EMAIL REQUEST TASK
FOR ACADEMIC WRITING ASSESSMENT: A
STUDY OF RELIABILITY AND TEST
DEVELOPMENT
Randall Rebman
Randall.Rebman@nau.edu
Northern Arizona University

2. LITERATURE REVIEW
• Pragmatics is part of most models of communicative
competence (Bachman & Palmer, 2010; Canale, 1983;
Canale & Swain, 1980).
• The testing of pragmatic competence is an
underexplored area in second language assessment
(Roever, 2011).
• Tasks assessing the construct of academic writing can
include integrated (reading/writing or listening/writing),
independent, and situational-based tasks (Cumming et
al., 2000).

3. TARGET DOMAIN OF ACADEMIC WRITING
• It is important to select writing tasks from the
Target Language Use (TLU) domain that can be
developed for assessment tasks (Bachman &
Palmer, 2010).
– Select academic writing tasks that:
• occur in introductory level academic courses that
require writing (Cumming et al., 2000).
• occur in academic communication situations
between peers and faculty and in general on
campus (Youn,S.J., 2009).

4. TEST PURPOSE
• To place L2 students in different levels of writing ability.
• To decide if students matriculate into the university from the
intensive English program.
• To include a representation of writing tasks that second language
writers will be required to produce in university contexts.
• To explore how a broader coverage of the construct can be
attained using a pragmatic task for writing assessment

5. WHY AN EMAIL TASK?
• Needs
– EAP students struggle with using the proper conventions of
email for communication (Youn,S.J., 2009)
• Washback
– Potential for positive washback (Crusan, 2010) by
encouraging more coverage of pragmatic features of emails
(indirect and direct request strategies, genre markers) in
course instruction
• Adding an email task to a writing test can expand the range of the
construct of academic writing that is assessed

6. LIMITED TEST DOMAIN
Independent Task:
Prompt-based
Argumentative essay
Integrated Task:
Summary of a
Situational-based chart
Task:
Request to a
Professor

7. RESEARCH QUESTIONS FOR EMAIL TASK (RQ 1&2)
1. Can the raters produce consistent ratings of an
email writing task using a new rubric?
2. Is the email response task testing academic
writing ability in a different way than the
integrated and independent writing tasks?

8. PARTICIPANTS
• n= 103
• International students mainly representing China,
Saudi Arabia, Japan, Korea, and Kuwait
• Ages ranged from 18-25
• All were pre-university students required to take
the English placement test to determine level
placement into the intensive English program or
into Northern Arizona University

9. METHODS
• University IEP students were given the prototype email task as
part of a placement/exit test.
• A holistic scale for the prototype email task was created by the
IEP assessment team using the empirical method (Weigle, 2002)
during the piloting of the task.
• A 6-point rubric operationalized the construct of writing ability on
the email task. This resulted in a score of 0-5 given by a single
rater.
• The summed score between two raters was used to give a score
ranging from 0-10.

10. METHODS
• The Spearman rho is used to measure and inter-rater reliability
for RQ1.
• A Spearman rho is used to measure the internal consistency
between the email task, integrated task and independent task for
RQ 2.

11. METHODS
Email task rubric scoring criteria developed through the empirical
method covers the following features:
• Language use
• Grammatical and lexical features
• Register awareness, including appropriate forms of address
• Genre markers specific to emails
• Topical relevance
• Task completion

12. TEST TASK CHARACTERISTICS
Task Prompt
Ddirections: Read the question below. Plan, write, and revise an email.
Use the space below to prepare writing your email. You may begin now.
Question: You are new at XXX University. You do not know what
classes to take. Write an email to Professor Smith to do the following:
1) introduce yourself
2) explain your problem
3) ask for advice

13. RESULTS: RQ1
Correlation Coefficient for Inter-rater reliability
Ratings
Correlation Coefficient N
Email Rating 1 -- 103
Email Rating 2 .92 103
Note. CI = confidence interval; LL = lower limit; UL = upper limit
Descriptive Statistics
95% CI
Ratings M SD N LL UL
Email Rating 1 3.44 1.13 103 3.22 3.66
Email Rating 2 3.47 1.14 103 3.24 3.69
Note. df = 172; alpha .05; rho critical = .364; N = Number of pairs;

14. RESULTS: RQ2
95% CI
Writing Task M SD N LL UL
Email 6.90 2.22 103 6.46 7.33
Independent 4.91 2.02 103 4.51 5.30
Integrated 4.66 1.87 103 4.23 5.03
Note. CI = confidence interval; LL = lower limit; UL = upper limit
Correlations Between Writing Tasks
Note. df = 172; alpha .05; rho critical = .364; N = Number of pairs;

15. DISCUSSION
• Students had higher mean scores on the email task than on the
other two task types.
• What are the limitations for implementing the new task?
– The test scores on the email task did not distinguish students by
writing ability.
– The task appears to be lacking complexity or the scale made it too
easy to get a high score.

16. IMPLICATIONS FOR ONGOING TEST TASK
DEVELOPMENT
• The email task could be improved by adding complexity to the
task design.
– Give more input for learners to respond to
• Ex: a sample email from a professor to which they must respond to
– involve a higher imposition request to the professor

17. IMPLICATIONS FOR ONGOING TEST TASK
DEVELOPMENT
• The scale must be revised to better distinguish criteria expected
for different bands of rubric.
– A sample of emails to faculty members could be gathered to
identify pragmatic features lacking in current task design.
– We could take out any criteria in rubric that does not apply to
pragmatic features.

18. REVISED TASK
• Writing Task 1: Email a Professor (10 minutes total)
• Planning (2 minutes): Read the situation below. Plan and write an
email. Use the space below to prepare writing your email.
• Situation: You are a student at Northern Arizona University. You
have to turn in an essay in five days, but your writing has many
problems. Write an email to Professor Smith and do the following:
– fill in the subject line
– introduce yourself
– explain your problem
– request an appointment to get help
– ask for a reply

19. THANK YOU FOR COMING
Questions?

20. IMPLEMENTING AN EMAIL REQUEST TASK
FOR ACADEMIC WRITING ASSESSMENT: A
STUDY OF RELIABILITY AND TEST
DEVELOPMENT
Randall Rebman
Randall.Rebman@nau.edu
Northern Arizona University

21. CORRECTION FOR ATTENUATION
Correlations Corrected for Attenuation
CLICK TO EDIT MASTER TITLE STYLE