Ishii, Y., Sawaki, Y., & Tahara, T. (2017). An analysis of Japanese EFL learners’ reading-to-write task completion process: Triangulation of stimulated recall and keystroke logging data sources. The 21st Annual Conference of the Japan Language Testing Association. University of Aizu.

1. An analysis of Japanese EFL learners’ reading-
to-write task completion process:
Triangulation of stimulated recall and
keystroke logging data sources
Yutaka Ishii, Yasuyo Sawaki, & Tatsuro Tahara
Waseda University
yutakaishii@aoni.waseda.jp
The 21st Annual Conference of the Japan Language Testing Association
University of Aizu
1

2. Acknowledgement
• This work was supported by Grant-in-Aid for
Scientific Research 16K02983.
2

3. Introduction
• Reading-to-write tasks (e.g., summarization) are
important task types in conducting academic work
in English (Rosenfeld, Leung & Oltman, 2001).
• In this Kaken project we examine characteristics of
Japanese university students’ performance on
summarization tasks and develop materials for
teaching summarization in academic writing
courses.
• This presentation focuses on the process learners
engage while completing a summary task
• Stimulated recall and keystroke logging are combined for
a fine-grained analysis of learners’ writing process from
multiple perspectives.
3

4. Previous Studies on L2 writing
task completion process
• Process features (Baaijen, Galbraith, & de Glopper,
2012; Barkaoui, 2016; Zhang, & Deane, 2015)
• Relationship between English language proficiency
and keyboarding skills (Barkaoui, 2015)
• Stimulated recall (Lindgren, & Sullivan, 2003)
• Time allocation of composing time (De Larios,
Manchon, & Murphy, 2006)
• Integrated writing task process (Barkaoui, 2015;
Chan, 2017)
4

5. Research Question
• What task completion process do EFL learners in
Japan engage to respond to a summary task?
Specific foci:
• Reading the source text
• Writing the response
• Reading-writing integration
5

6. Method (1)
• Participants
• Five undergraduate students majoring in English
language and literature at a private university in Tokyo
• All participants enrolled in a required academic writing
course for second-year students
• Materials
• Two summarization tasks (each based on a single source
text in English)
• Source texts were adopted from published Eiken test
forms for Grade Pre-1 (Eiken Foundation of Japan)
6

7. Method (2): Source texts
A. The War over
Antiquities
(3rd admin. 2014)
B. The Automated
Future
(1st admin. 2014)
Structure Argumentative
Length (words) 508 504
paragraphs/
sentences
4/23 4/24
Vocabulary
Level
6000 5000
Readability
(F-K Grade Level)
12.4 12.4
Summary length Approx. 80 words
7

8. Method (3)
• Procedure
• Each learner summarized one text in English (Approx. 80
words).
• Random assignment of participants to texts
• 40-minutes per task; dictionary use allowed
• The learner’s task completion process videotaped
• Keystroke logged via Writing MaetriX
• An observation sheet comprising six categories (Read text,
Annotate, Plan, Write, Review, & Revise) completed by observers
• Upon completion of the summary task, a stimulated recall
session was conducted in Japanese.
• Prompts:
• Video of the task completion process
• Replay of Keystroke logging data
• Completed summary response
• Test booklet with the participant’s notes (annotations and
outlines)
8

9. Method (4)
Video recording
Two observers took notes on
the observation sheet.
9
Test-taker
Online dictionary (Weblio)
Recording of keystroke logging

10. Method (5)
• Scoring
• Double scoring on all rating scales by two trained raters
• Rating scales
• Integration (1-4 scale): The degree of succinct representation
of source-text content by employing appropriate Macrorules
(e.g., Brown & Day, 1983; van Dijk & Kintsch, 1983) for
integrating information across text (based on Sawaki, 2003;
Hijikata-Someya, Ono & Yamanishi, 2015)
• Language (1-4 scale): The degree of grammatical control,
syntactic variety, and appropriate word choice (adapted from
Hijikata-Someya et al., 2015)
10

11. Method (6)
Scoring results summary
Student Text
Total
Number of
words
Integration
(1-4)
Language
(1-4)
String match
(3+ words)
1 A 84 2.5 2.0 4
2 B 75 2.5 3.0 0
3 B 80 3.5 3.0 0
4 B 75 2.5 2.0 1
5 A 94 2.0 2.0 0
11

12. Method (7)
• WritingMaetriX (Kusanagi, Abe, Fukuta, &
Kawaguchi, 2014)
• A keystroke-logging program that can record, analyze,
and replay learners’ writing process
12

13. Results: Overview
• Time required for task completion: approx. 32– 40 min.
• Task completion time divided into two phases for qualitative
analyses:
• Phase 1: Reading (text reading, annotating, and planning)
• Phase 2: Writing (writing, revising, and reviewing)
Task completion process
Phase 1
Phase 2
(6-22 min.)
(18-33 min.)
13

14. Phase 1: Annotations
• Students used various kinds of annotations while
reading the source text.
• Underline: I underlined when I found a concrete example
(S5: 14)
• Mark: I have a habit of writing a reverse triangle when
looking at "however" (S4: 48)
• parenthesis: I just wrote a parenthesis to highlight the
that-clause following “argue" (S1; 9)
• Others (slash, notes on blank spaces, etc...)
See next slide for examples
14

15. Phase 1: Planning
• Students had different styles of planning
• Student 1, 2, 3: developed the structure of
summary only in their minds (without writing)
• Student 4, 5: took notes on question paper
• I thought about the structure while looking at the
(question) paper first, before I started writing. (S5:
29)
• This is an introduction, and I picked out and took
notes of the skeleton only, so that I could use them
(for summary) (S4: 80)
15

16. Phase 2: Descriptive Statistics of learners’
keystroke logging
Student1 student2 Student3 student4 Student5
Task A B B B A
First data
input (sec)
447.0 632.2 721.0 1390.6 893.9
Key input
(minute)
28.4 42.0 50.1 34.5 77.3
# Words
(minute)
3.72 2.30 2.35 1.90 2.65
Revision
(minute)
0.97 1.78 2.61 2.43 4.28
revision 0/7/22 0/24/53 0/45/89 0/16/96 0/79/151
16

17. Phase 2: Time Series Data of Learners'
Writing Process
Revision
First data input
Planning
17

18. Phase 2: Verbal protocol data
• First data input
• First of all, I wrote industrial, industrial revolution in the
introduction session. (S4: 160)
• Planning
• After I finished writing about the second paragraph, I
was about to write the third paragraph. Then, I was
thinking about how to summarize this paragraph. (S5:
76)
• Revision
• From here, I tried to reduce the word such as UNESCO.
(S5: 123)
18

19. Phase 2: Planning and revision
processes
• Planning
• Comprehensive & local planning depending on the situation
(similar to independent writing)
• Re-reading and paraphrasing the source text
• Revision
• Two types of revisions: precontextual and contextual (similar
to independent writing).
• Another precontextual revision “paraphrase” was frequently
observed compared to independent writing task.
• Planning & revision: Focus on reducing the number of
words frequently reported (meeting the length limit =
unique to source-based writing tasks)
19

20. Phase 2: Planning example
Stimulated Recall
• I may be thinking about
the paraphrase of “ship”
in this pause. I used the
word “transfer” to report
that the antiques finally
moved to UK. However,
because the meaning
was temporailiy obscure,
I used weblio at this time.
(S1: 49)
Keystroke Logging
• In recent years, illegally
imported antiques have
been returned to their
hometown. This is largly
due to the treaty by
UNESCO, but it is not
perfect. One well-known
example is the Elgin
Marbles, a heritage of the
Palthanon in Athens, Greek.
It is originally owned by
Greece, but it was
transfered to Britain for
preservation*
20

21. Phase 2: Revision processes
Stimulated Recall
• I noticed the number of
characters is not
enough and had to
summarize the content.
So, I was worried about
how to edit the content.
(S4: 187)
Keystroke Logging
• Industrial Revolution
increased the productivity
of that age while it also
made difficulties to the
labors, which thier working
was replaced by machine.
However, in this age,
information-technology
brings new reformation to
the way of people. Hanson
claims that (considers this
reformation possitively,)
21

22. Phase 2: Reading-writing integration
• Writerly reading (Hirvela, 2004): reading with
consciousness of writing summary
• I underlined a concrete example that I thought was easy
to use (for summary) (S3: 39)
• I found this the most critical in the paragraph, and this
explanation continued, so I thought I need to write it (in
my summary) and underlined that part (S5: 17)
The parts mentioned above were actually reflected in
these students’ summaries.
22

23. Phase 2: Reading-writing integration
• Mining: (Hirvela, 2004; Plakans, 2009): detecting
information for text while writing
• e.g.) Considering paraphrase, Adding/deleting examples
in summary, Comparing text with summary
• I underlined the abstract points while reading, (...) While
writing a summary, I found it bothersome to look for
(these points), I underlined them so that I could go back
to the points I used for summary (S3: 144).
• I checked if my summary made sense and if there were
no points I overlooked (S2: 91)
23

24. Discussion and Conclusion
• Annotation and planning behavior while reading differs as well as how
to use the information highlighted in their summaries
• The result suggests an important role of reading in integrated tasks, shown as
previous studies (e.g. Plakans & Gebril, 2012; Sawaki , Quinlan, & Lee, 2013).
• (The most important skill is) reading. Without deep understanding, I could not
write or organize (my summary). (S3:197)
• Some revision behaviors observed while completing independent tasks
were also observed in this study, while other unique ones were also
identified.
• Revision behavior differs between ESL and EFL learners
• Barkaoui (2016) stated that learners revised their writing most frequently in the
middle segment of their writing process, while completing an integrated writing
task.
• In contrast, learners in this study revised their writing in the final segment.
• Limitations of this study
• Small sample size
• Stimulated recall
24

25. References (1/2)
• Baaijen, V. M., Galbraith, D., & de Glopper, K. (2012). Keystroke analysis: Reflections on procedures and
measures. Written Communication, 29(3), 246-277.
• Barkaoui, K. (2016). What and When Second‐Language Learners Revise When Responding to Timed
Writing Tasks on the Computer: The Roles of Task Type, Second Language Proficiency, and Keyboarding
Skills. The Modern Language Journal, 100(1), 320-340.
• Barkaoui, K. (2015). Test Takers' Writing Activities During the TOEFL iBT® Writing Tasks: A Stimulated
Recall Study. ETS Research Report Series, 2015(1), 1-42.
• Brown, A. L., & Day, J. D. (1983). Macrorules for summarizing texts: The development of expertise.
Journal of Verbal Learning and Verbal Behavior 22, 1-14.
• Chan, S. (2017). Using keystroke logging to understand writers’ processes on a reading-into-writing test.
Language Testing in Asia, 7(10), 1-23.
• De Larios, J. R., Manchon, R. M., & Murphy, L. (2006). Generating text in native and foreign language
writing: A temporal analysis of problemsolving formulation processes. The Modern Language Journal,
90(1), 100-114.
• Hijikata-Someya, Y., Ono, M., & Yamanishi, H. (2015). Evaluation y native and non-native English
teacher-raters of Japanese students’ summaries. English Language Teaching, 8(7), 1-12.
• Hirvela, A. (2004). Connecting reading and writing in second language writing instruction. Ann Arbor,
MI: The University of Michigan Press.
• Kusanagi, K., Abe, D, Fukuta, J, & Kawaguchi, Y. (2013). Visualizing writing process using a key-logging
system: For construct feedback to enhance autonomous learning. Paper presented at the 81st Spring
Conference of the Chubu Chapter, Japan Association for Language Education and Technology (LET),
Tokai Gakuen University, Japan.
25

26. References (2/2)
• Lindgren, E., & Sullivan, K. P. (2003). Stimulated recall as a trigger for increasing noticing and language
awareness in the L2 writing classroom: A case study of two young female writers. Language Awareness,
12(3-4), 172-186.
• Plakans, L. (2015). Integrated Second Language Writing Assessment: Why? What? How?. Language and
Linguistics Compass, 9(4), 159-167.
• Plakans, L. (2009). The role of reading strategies in integrated L2 writing tasks, Journal of English for
Academic Purposes, 8(4), 252-266.
• Plakans, L., & Gebril, A. (2012). A close investigation into source use in integrated second language
writing tasks. Assessing Writing, 17(1), 18-34.
• Rosenfeld, M., Leung, S., & Oltman, P. K. (2001). The reading, writing, speaking, and listening tasks
important for academic success at the undergraduate and graduate levels. Educational Testing Service.
• Sawaki, Y., Quinlan, T., & Lee, Y. W. (2013). Understanding learner strengths and weaknesses: Assessing
performance on an integrated writing task. Language Assessment Quarterly, 10(1), 73-95.
• Sawaki, Y. (2003). A comparison of summarization and free recall as reading comprehension tasks in
web-based assessment of Japanese as a foreign language (Unpublished PhD thesis). University of
California, Los Angeles (UCLA).
• Van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic Press.
• Zhang, M., & Deane, P. (2015). Process features in writing: Internal structure and incremental value
over product features. ETS Research Report Series, 2015(2), 1-12.
• Obunsha (Ed.) (2016). 2016 nendo eiken jun-1 kyu kako 6 kai zenmondaishuu (Test papers for the last six
administrations of the Eiken Grade Pre-1 test, 2016). Tokyo: Obunsha.
26