Reading 2 guideline for item writing writing test

1. Part 1: Test specifications for Writing test - VSTEP
1. General information:
 Time: 60 minutes
 General description of the test: the test which is composed of 2 parts aims at assessing
students’ written communicative competence. It targets at the test takers at level 3 to
level 5 CEFR-VN levels.
 Task demands and response format:
o Input: in written form, with the topic written in the target language at level 3 of
VSTEP. The topics are familar.
o Expected output: a letter serving different purposes of communication (task 1)
and an essay giving information, expressing and defending opinions (task 2).
 Description of each task:
o Task 1: Time limit: 20 minutes. Length: 120 words at least. Weighting: 10/30.
Test takers write a(n) letter/ email in response to a(n) given letter/email, or test
takers write a(n) letter/email serving different communication purposes. This task
tests written interaction skills.
o Task 2: Time limit: 40 minutes. Length: 250 words at least. Weighting: 20/30.
Test takers write an essay on a given topic, using his/her experience and
knowledge to support his/her arguments. This task tests written production skills.
 Number of items: 2
 Score reporting plan: A raw score out of 30 is converted to a score in 10-point scale

2. Task demands and response
format
Time
recommend
-ation
Intended operations Task types Instructions
Task 1 Input:
 Part of a(n) letter/email about
an event (80-90 words)
related to a familiar, everyday
situation. or
 A familiar, everyday situation
(80-90 words) that requires
correspondence
Expected output:
 A (reply) letter/email (at least
120 words)
Content knowledge:
familiar matters regularly
encountered in work, school,
leisure; topics which are
familiar, or of personal interest.
20 minutes  Can convey information
and ideas on abstract as
well as concrete topics
(e.g work, study or films,
music), check information
and ask about or explain
problems with reasonable
precision.
 Can write personal letters
describing experiences,
feelings and events in
some detail.
 a(n) letter/
email in
response to
a(n) given
letter/
email
 a(n) letter/
email asking
for
information,
describing
experiences,
feelings and
events.
Test takers are asked to write a(n)
letter/ email in response to a(n)
given letter/email or to give/ask for
information.

3. Task 2 Input:
 A given text/an extract from a
book/an article on a given
topic (50-60 words). The
topics may be related to
Vietnamese context.
Output: (at least 250 words)
 An essay evaluating/giving
opinions
 An essay explaining
advantages/disadvantages
 An essay explaining
causes/effects
 An essay explainin
problems/solutions
Content knowledge:
Education, health,
entertainment, transportation,
history, art, etc
40 minutes  Can write an essay
supporting arguments,
highlight main ideas and
supporting them with
relevant details.
 Can write an essay to
develop an idea, giving
reasons for arguments,
explaining the advantages
and disadvantages.
An essay Test takers are asked to write an
essay on a given topic, using his/her
experience and knowledge to
support his/her arguments.

4. 2. Sample instruction and items:
Task 1:
You should spend about 20 minutes on this task.
You have invited an American businessman, Mr. Noah Watson, to give a talk at your English
club monthly meeting. Read part of the letter from him below.
I would be glad to come and give a talk to your English club. In order to make a good
preparation, I would like some information about my audience. I could talk about either my
experience when starting up my company or some general rules when doing business. Which one
would be more interesting?
Also, I am staying in a hotel in the city centre. Could you tell me where the meeting is and how
to get there?
I look forward to meeting you soon.
Write a letter responding to Mr. Noah Watson. You should write at least 120 words. You are not
allowed to include your name. Your response will be evaluated in terms of Task fulfilment,
Organization, Vocabulary and Grammar.
Task 2:
You should spend about 40 minutes on this task.
Read the following extract from a book chapter in 1 minute: Read the following extract from a
book chapter in 1 minute:
There are fewerviolentcrimesinsome countries inrecentyears,butthe overall numberof crimesisstill
very high. People have different opinions about the best way to control the crime rate. Some suggest
preventingpeoplefroman/any illegal action before they do it while others prefer punishing criminals
more heavily.
Adapted from Academic Listening Encounters - Life in Society, Chapter 8, Page 113-114

5. Now, write an essay in 40 minutes to discuss what measures in your opinion are more effective
in controlling the crime rate.
Give reasons for your answer and include any relevant examples from your own knowledge or
experience. Your response will be evaluated in terms of Task fulfilment, Organization,
Vocabulary and Grammar.
You should write at least 250 words.
Part 2: Guidelines for the design of effective writing prompts
2.1. General principles in designing effective writing prompts
2.1.1. Clearly define the intended writing skills to be tested by the item.
Decide which of the following intended writing skill lends itself better to be assessed by an
essay question.
Example A
You should spend 40 minutes on this task. Read the following text from a book about tourism.
Tourism has become one of the fastest growing industries in the world. Millions of people today
are travelling farther and farther, not only in the summer but all year round. Some people argue
that the development of tourism has had negative effects on local communities; others think that
its influences are positive.
Express your opinion on the issue. Include reasons and relevant examples to support your
answer.
Example B
You should spend 40 minutes on this task. Read the following text from a book about tourism.
Tourism has become one of the fastest growing industries in the world. Millions of people today
are travelling farther and farther, not only in the summer but all year round. Some people argue
that the development of tourism has had negative effects on local communities; others think that
its influences are positive.
Write an essay to discuss the effects of tourism on the tourist destinations. Include reasons and
relevant examples to support your answer.

6. Example B is more useful for guiding the development of an essay question.
Specific intended writing skills are crucial to designing effective writing questions. If the
expected outcome to be assessed lacks clarity and specificity, the writing question meant to
assess students’ performance of a skill will likely assess something other than what is intended.
Example A is too general to provide clear guidance in answering the question. The directive
verb, “express” is vague and difficult to assess. It is especially unclear about what form of the
answer should be. Meanwhile, example B explicitly requires candidates “to write” and the
desired form of the answer is “an essay”.
In specifying the intended writing skills, item writers should clarify the performance that
students should be able to demonstrate as a result of what they have learned. Tested writing skills
meant to guide the development of test items will typically begin with a specific directive verb.
The statement will describe the observable behavior, action or outcome that students should
demonstrate.
2.1.2. Clearly define and situate the task within a problem situation.
a. Define the task
With some writing questions, test takers can feel like they have an infinite supply of lead to write
a response on an indefinite number of pages about whatever they feel happy to write about. This
can happen when the writing question is vague or open to numerous interpretations. Remember
that effective writing questions: provide candidates with an indication of the types of thinking
and content to use in responding to the question.
A clearly defined task will specify the performance test takers should exhibit when responding to
the writing question. A clearly defined task is composed of a directive verb and the object of that
verb. For example, consider the following tasks:

7. Limiting the subject matter for a given question depends on different factors. Among other
things, how to restrict the task can depend on the hierarchy of knowledge related to the question.
The following example illustrates the process of limiting the scope of the task for a given essay
question.
Example of an Evolving Essay Question that Becomes More Focused
1. Less focused
essay question:
Evaluate the impact of the social networking sites.
2. More focused
essay question:
Evaluate the impact of the social networking sites on the young generation.
Failure to establish adequate and effective limits for the response to the writing question allows
test takers to set their own boundaries for their response, meaning that they might provide
responses that are outside of the intended task, that are too long, or that only address a part of the
intended task. Test takers’ failure to answer within the intended limits of the essay question can
be ascribed to poor or ineffective wording of the task to some extent. Therefore, item writers are
responsible for writing questions in such a way that they provide test takers with clear
boundaries for responses. In short, item writers should avoid indeterminate questions.
One way to avoid indeterminate questions is to stay away from vocabulary that is ambiguous.
Moreover, item writers should also avoid including vocabulary that is too advanced for the test
takers. The suggested language level for both tasks are B1 (CEFR) vocabulary which may
account for 80-90% of the words used in the prompts.
b. Create a problem situation

8. Once the writing task is clearly defined and the content that test takers are to use in
accomplishing the task has been delimited, the question is ready to be situated in a problem. Both
the tasks and the problems are key elements of writing questions.
The problem in writing questions includes the unsettled matter or undesirable state of affairs that
needs to be resolved. The purpose of the problem is to provide the students with a context within
which they can demonstrate the writing performance to be assessed.
Writing problem situations requires creativity and clarity. The circumstances should be clear in
the sense that even a minimum test taker of level 3 can identify the described event and its
related issues . At the same time, the problem situation should not be completely foreign to the
test takers’ experience.
Example A: Analyze the task and the problem of the following writing question.
You should spend 20 minutes on this task
You received an email from your English pen friend, Jane, to ask information about one of your
friends. Read part of her email below.
...I’ve just got an email from your friend, Mai. She is going to do a course in London this
summer and wants to rent a room in my house. Could you tell me a bit about her (work/study,
personality, hobbies and interests) for me to see if she would fit in with my family?
…
Write an email responding to Jane. You should write at least 120 words. Do NOT include your
name.
2.1.3. Present a reasonable task.
When defining the task for a writing question, item writers need to make sure that they present a
reasonable task to test takers. One of the challenges that item writers face in composing essay
questions is that because of their extensive experience and background knowledge they may be
tempted to demand unreasonable content expertise from the test takers. Hence, item writers need
to make sure that their test takers can “be expected to have adequate material with which to
answer the question” (Stalnaker, 1952, p.520). The writing topics should be familiar and
accessible to all candidates.

9. Identifying an audience for test takers to write their responses to may also be helpful. This would
make the task more authentic. A key element in writing communication is to clearly identify and
write to a specific audience.
2.1.4. Specify the relative point value, the approximate time limit and word limit in clear
directions.
Specifying the relative point value and the approximate time limit helps test takers allocate their
time in answering several writing questions in a writing test because the directions clarify the
relative merit of each question. Without such guidelines test takers may feel at a loss as to how
much time to spend on a question. When deciding how much time should be spent on a question,
the slower candidates can have a clearer plan for their response.
Also make sure that test takers can be realistically expected to provide an adequate answer in the
given and/or the suggested time.
2.1.5. Specify the assessment criteria.
The expected response should be described clearly in terms of the criteria that will be used to
judge it. For example, a letter-writing task will be scored without consideration of grammatical
accuracy, as long as the response is comprehensible, examinees should be told as much. On the
other hand, if an essay will be scored on the basis of grammatical accuracy, vocabulary use,
content and organization, those categories should be disclosed to the test takers in the directions.
2.1.6. Improve the essay question through preview and review.
The following steps can help item writers improve an item before and after they hand it out to
test takers, making the question much more effective.
Preview (before handing out the essay question)
a. Predict responses.
Try to respond to the question from the perspective of a typical candidate. Evaluate whether
candidates have the content knowledge and the skills necessary to adequately respond to the
question. Repair any possible weaknesses of the essay question before handing out the exam.

10. b. Write a model answer.
Before using a question, write model answer(s) or at least an outline of major points that should
be included in an answer. Writing the model answer allows reflection on the clarity of the essay
question. Furthermore, the model answer(s) serve as a basis for grading responses. Once the
model answer has been written compare its alignment with the question and the tested writing
skills and make changes as needed to assure that the tested writing skills, the essay question, and
the model answer are aligned with each other.
c. Ask a knowledgeable colleague to critically review the question, the model answer, and the
tested writing skills for alignment.
Before using the essay question on a test, ask a person knowledgeable in the subject to critically
review the essay question, the model answer, and the the tested writing skills to determine how
well they are aligned with each other. Based the tested writing skills, revise the question as
needed. By having someone else look at the test the likelihood of creating effective test items is
increased. Try asking a colleague to evaluate the essay questions based on the guidelines for
constructing essay questions contained in this guideline.
Review (after receiving the responses)
After test takers complete the test, carefully review the range of answers given and the manner
in which test takers seem to have interpreted the question. Make revisions based on the findings.
Writing good questions is a process that requires time and practice. Carefully studying the
responses can help evaluate test takers' understanding of the question as well as the effectiveness
of the question.
2.2. Principles in designing writing prompts for Task 1 and Task 2
Guidelines for Task 1
a. Components of an item
There are three parts in an item of Task 1: Instruction, Letter extract and Task requirement.
 Instruction suggests how much time candidates should spend on Task 1 and includes

11. one or two sentences in italics, which often introduce the name of the sender and explain
the relationship between the sender and the responder.
 Letter extract is part of an email or a letter placed in textbox.
 Task requirement explains the expected output from the candidates, the minimum
number of words in the target response, and how candidates’ answers are assessed. The
Task Requirement is virtually the same in all items in Task 1, except for the name of the
sender.
 Prompt includes both the instruction in italics and the letter extract.
b. Principles in writing Task 1 prompt
As Task 1 in the Writing Test aims at examining candidates’ ability in writing correspondence,
the expected response is in the form of an email or a letter. When designing questions for Task 1,
item writers should select a familiar topic, one which is regularly encountered in work, school or
leisure, or of personal interest. The criterion is to make sure that even candidates around Level 3
will not be jeopardized by a less familiar topic.
Furthermore, a letter extract needs to be written in a consistent style, using either formal or
informal language throughout. This will help to set the tone of the expected response. A letter
extract in Task 1 has approximately 80-90 words. All fully-developed responses in Task 1 should
allow candidates to communicate three main ideas, which means the letter extracts should raise
three questions to be answered, or address three problems that need to be settled, or a
combination of these.
Another requirement is concerned with the complexity of language, including lexis and sentence
structures to be used in the instructions and letter extracts. Similar to the requirement about topic
familiarity, item writers should design with the level of candidates at the lowest level (around
Level 3) in mind. For all prospectus candidates to be able to comprehend the content of a
prompt, it is suggested that 80 to 90 percent of the lexis and syntax used throughout the prompt
corresponds to Linguistic competence Level 3 descriptors. Item writers, therefore, can refer to
Vocabulary English Profile (http://vocabulary.englishprofile.org/) or Cambridge Dictionaries

12. Online (http://dictionary.cambridge.org/) to check the level of a certain vocabulary item.
To make it consistent across items in the item bank, a prompt needs to provide information
about: (i) the sender; (ii) the relationship between the sender and the responder; (iii) the topic of
the letter/email extract; and (iv) the context of the letter/email (why this letter/email is being sent,
what is expected from the responder). A prompt also needs to clearly indicate what
communication purpose(s) is expected in the response (such as giving direction, asking
for/giving advice, making enquiries, thanking, accepting/refusing invitation, …).
Guidelines for Task 2
a. Components of an item
Similarly to Task 1, there are also three parts in an item of Task 2.
 Instruction suggests how much time candidates should spend on Task 2 and introduces
the genres of the prompts.
 Prompt is often an extract from a book, part of a magazine article or journal article, etc.
Prompts are placed in a textbox.
 Task requirement indicates the expected response from the candidates and explains
what candidates are required to write about. The last paragraph in Task requirement
suggests how candidates can support their argument, the minimum number of words in
their responses, and lists criteria against which candidates’ answers are assessed. The last
paragraph is the same in all items in Task 2.
b. Principles in writing Task 2 prompt
As Task 2 in the Writing Test aims to assess candidates’ ability of written production, the target
response is an essay. Topics in Task 2 include more general issues, such as education, health,
entertainment, transportation, etc.
A prompt in Task 2 should be written in academic language, with the limit of approximately 50
to 60 words. Following the prompt, Task requirement should have 40 to 50 words. Like Task 1,

13. items in Task 2 should use lexis and syntax that accommodate candidates around Level 3.
Task requirement (an essay to evaluate/ to give opinion/ to explain advantages and
disadvantages/ to discuss problem and solutions …) has to be clearly indicated.
Prompts are often written in compound sentences and follow a consistent structure. The first one
or two sentence(s) in a prompt needs to introduce the topic and provides the background for a
certain issue. The next one or two sentences can serve either of the following functions:
 to list different opinions about the topic;
 to introduce a certain problem; or
 to introduce different aspects of the topic.
REFERENCES
Andrade, H. and Du, Y. (2005), “Student perspectives on rubric-referenced assessment”,
Practical Assessment, Research & Evaluation, Vol. 10 No. 5, available at:
http://PAREonline.net/getvn.asp?v¼10&n¼3 (accessed December 14, 2006).
Arter, J. and McTighe, J. (2001), Scoring Rubrics in the Classroom, Corwin, Thousand Oaks,
CA.
Boulet, J.R., Rebbecchi, T.A., Denton, E.C., Mckinley, D. and Whelan, G.P. (2004), “Assessing
the written communication skills of medical school graduates”, Advances in Health Sciences
Education, Vol. 9, pp. 47-60.
Bresciani, M.J., Zelna, C.L. and Anderson, J.A. (2004), Assessing Student Learning and
Development: A Handbook for Practitioners, National Association of Student Personnel
Administrators (NASPA), Washington, DC.
Brown, G., Bull, J. and Pendlebury, M. (1997), Assessing Student Learning in Higher Education,
Routledge, Oxon and New York, NY.
Callison, D. (2000), “Rubrics”, School Library Media Activities Monthly, Vol. 17 No. 2, pp. 4-
36, 42.

14. Campbell, A. (2005), “Application of ICT and rubrics to the assessment process where
professional judgement is involved: the features of an e-marking tool”, Assessment &Evaluation
in Higher Education, Vol. 30 No. 5, pp. 529-37.
Carr, Nathan T. (2011). Designing and Analyzing Language Tests. Oxford: Oxford University
Press.
Cambridge ESOL (2011). Using the CEFR: Principles of good practice. Retrieved from
http://www.cambridgeenglish.org/images/126011-using-cefr-principles-of-good-practice.pdf
Council of Europe (1992). Transparency and coherence in language learning in Europe:
Objectives, evaluation, certification. Report on the R¨uschlikon Symposium. Strasbourg: Council
of Europe.
Fraser, L., Harich, K., Norby, J., Brzovic, K., Rizkallah, T. and Loewy, D. (2005), “Diagnostic
and value-added assessment of business writing”, Business Communication Quarterly, Vol. 68
No. 3, pp. 290-305, available at: http://bcq.sagepub.com/cgi/content/abstract/68/3/290(accessed
May 2, 2007).
Gopinath, C. (2004), “Exploring effects of criteria and multiple graders on case grading”, Journal
of Education for Business, Vol. 79 No. 6, pp. 317-22.
Little, D. (2011). The common European framework of reference for languages: A research
agenda. Language Teaching, 44(3), 381-393.
Reiner, C., Bothell, T., Sudweeks, R. & Wood, B. (2002). Preparing effective essay questions.
New York: New Forums Press.
Shaw, S. & Weir, C. (2008). Examining Writing. Cambridge: Cambridge University Press.