Future of Learning

1. BLENDED READING USING
COLLABORATIVE ONLINE
ANNOTATION
hari stephen kumar
@hariteach
Instructional Designer
Amherst College
hkumar@amherst.edu
Blended Learning in the Liberal Arts Conference, Bryn Mawr, May 21, 2015

2. Collaborative Online Annotation Example:
nb.mit.edu

3. Overview
• Motivating pedagogy for blending a reading assignment
• Learning goals and teaching goals
• Collaborative annotation tools
• Tool selection and implementation
• Usage example
• Usage data, student feedback, instructor insights
• Challenges:
• Pedagogy & student orientation
• Accessibility

4. Pedagogical Overview
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE
Bloom’s Revised Taxonomy (Anderson & Krathwohl, 2001)

5. Pedagogical Overview
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE
Content
Bloom’s Revised Taxonomy (Anderson & Krathwohl, 2001)

6. Pedagogical Overview
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE
Content
Assignment
Bloom’s Revised Taxonomy (Anderson & Krathwohl, 2001)

7. Pedagogical Overview
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE
Content
Assignment
Transfer
Bloom’s Revised Taxonomy (Anderson & Krathwohl, 2001)

8. Pedagogical Overview
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE
Teaching = mentoring, coaching,
role modeling, facilitative
Learning = collaborative,
dynamic, relational, personal
Cost/volume benefits
Standardization
Grade-friendly
Bloom’s Revised Taxonomy (Anderson & Krathwohl, 2001)

9. Pedagogical Overview
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE
Blended
LearningOnline/
Out of class
Face-to-face/
In class
Bloom’s Revised Taxonomy (Anderson & Krathwohl, 2001)

10. Blended Learning Overview
• Combining class time with online learning
experiences to improve student
engagement.
• “... to link, or blend, what happens in each medium so that
face-to-face and online activities reinforce each other to
create a single, unified, course.” (Glazer, 2011)
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE

11. Blended Learning Overview
• NOT just content delivery
Often requires active learning strategies
for both the in-person and online
components.
• NOT primarily about polished video or high tech
Online content allows for a wide variety of formats,
including interactive quizzes and simpler exercises.
• (*)NOT about reducing class time with teachers
Involves making better use of in-person class time with
teachers for higher-order engagement and learning.
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE

12. Blended Reading: A Case Study
• Professor Andrew Dole
Spring 2015 Religion Course
Kernel 1:
I like PDF reading apps on my iPad, but these apps tend to
be for individual readers. What if we had a PDF reader that
was built for groups to use together?
Kernel 2:
Electronic versions of texts are not the same as print
versions: no page numbers, dynamic font sizes, etc. Hard
to recall locations of key quotes during discussion.
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE

13. Blended Reading: Teaching Goals
• Professor Andrew Dole
Spring 2015 Religion Course
Teaching goal 1:
To have visibility into page-level and paragraph-level
questions students have about a text before class
discussion.
Teaching goal 2:
Have class discussion around print text enhanced by
students’ pre-class online conversations on key pages and
paragraphs.
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE

14. Blended Reading: Learning Goals
• Professor Andrew Dole
Spring 2015 Religion Course
Learning goals:
Left at same implicit level as before: students read texts in
order to understand key concepts and arguments.
Syllabus excerpt about goal of reflection posts:
“… to express your understanding of the main project of the
reading and make it clear what parts of that project you
have difficulty understanding.”
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE

15. Blended Reading: Syllabus Excerpt
General: Students are expected to do all of the assigned readings before
class, to attend all class sessions and actively participate in classroom
discussion.
Reflection posts: Beginning in early February, for each class session
approximately 5 students will submit reflection posts on the assigned
readings (the exact number will be determined after the add/drop period).
These posts are to be submitted to the Moodle site for this course by 8:00
PM on the day before class. All students are expected to be familiar with
the contents of these reflection posts in advance of the class session. The
purpose of these reflection posts is to lay the groundwork for a profitable
class discussion. The main thing you should accomplish in your reflection
posts is to express your understanding of the main project of the reading
and make it clear what parts of that project you have difficulty
understanding. These posts are a required part of the course and will
count towards your final grade, but they will not be graded individually.
So long as you demonstrate thoughtful engagement with the material you
will receive credit. Please note that simply summarizing the assigned
reading does not count as thoughtful engagement.
For a select number of readings, reflection posts will assume a different
form. Electronic versions of the assigned texts will be posted online (via
the Moodle site or an alternative), and students will make use of a tool that
will allow them to append comments to specific parts of the assigned text.
For these readings students will be expected to both comment on one or
more passages of the assigned text and respond to at least one other
student’s comment.
In-class presentations: Each student will be assigned responsibility for
getting one class meeting’s discussion started. The best way to do this is
to speak briefly about your understanding of the contents of the assigned
text, taking note of what other students have written in their online
comments, and then pose one or more questions to the class for discussion.
Papers: There will be three formal papers for the course, due on the dates
indicated in the course schedule below.

16. Blended Reading: Tool Selection
• Professor Andrew Dole
Spring 2015 Religion Course
Criteria for selecting annotation tools:
1. Must use facsimile of print text, e.g. scanned PDF.
2. Must be accessible via course Moodle page (i.e. not
requiring third-party account signup).
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE

17. Blended Reading: Tool Selection
• Professor Andrew Dole
Spring 2015 Religion Course
Review of annotation tools:
• nb.mit.edu: Works great with scanned PDFs, but no LTI
integration with Moodle.
• eMargin: Works great with Moodle, but only works with
plain text.
• Annotation Studio: Great interface, but only works with
plain text, and does not integrate with Moodle.
• hypothes.is: Great interface, but only works with plain
text, and does not integrate with Moodle out of the box.
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE

18. Blended Reading: Tool Implementation
• Professor Andrew Dole
Spring 2015 Religion Course
Implementation choice:
• nb.mit.edu: Works great with scanned PDFs, but no LTI
integration with Moodle.
• Solution: Open source – we added LTI integration and are
contributing the code back to the NB team at MIT.
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE

19. Blended Reading: Usage Example

20. Blended Reading: Usage Example

21. Blended Reading: Usage Example

22. Blended Reading: Usage Data
• Overview: 8 students, 12 reading assignments
• 2 students per reading assignment
• Each required to post 3 comments, then post 1 reply
Student Posts Replies
A-M 4 0
B-M 13 3
C-F 10 2
D-M 7 3
E-M 13 3
F-M 7 0
G-F 3 0
H-M 6 0

23. Blended Reading: Usage Data
• Active pairs:
Student Posts Replies
A-M 4 0
B-M 13 3
C-F 10 2
D-M 7 3
E-M 13 3
F-M 7 0
G-F 3 0
H-M 6 0

24. Blended Reading: Usage Data

25. Blended Reading: Usage Data
• B & C interactions:

26. Blended Reading: Usage Data
• D & E interactions:

27. Blended Reading: Usage Data
• An interaction between C (F) & B (M):

28. Blended Reading: Usage Data
• C’s post:
In the remainder of the chapter Douglas explains how different social setups regarding
the nature of gendered dominance result in different sexual pollution and purity
concerns. In cases where male dominance is applied with "ruthless simplicity", there
are few pollution worries due to the system's overall simplicity, but in cases where
certain male and female powers/privileges seem to contradict, more purity
considerations are required to maintain the fragile and complicated social structure.
Douglas doesn't, however, explain how these differences in social structure arise, or
what they are a result of. What conditions might cause these two kinds of societies to
form? Is there any reason that one would form over the other? Is Douglas' lack of origin
explanation a weakness to her overall argument?
• B’s reply:
[C], I very much agree with this. In my opinion, the inability to explain origins is a big
fault in Douglas' argument. There are a plethora of factors that could have influenced
discrepancies in social structures and in turn, affected what a society deems to be
sexual pollution. These factors could have instead possibly shaped views on pollution
directly, without necessarily going through the process of becoming incorporated into a
"social set up.” I think Douglas, however, would say that it doesn't matter where the
origins came from. Instead, we should simply examine the structures, as they are what
shape society, and examine how these different setups lead to different results.

29. Blended Reading: Usage Data Summary
• Students seemed to treat it as a homework task to fulfill:
post 3 times, reply once.
• Content of comments were more like a mini-essay staking
a claim rather than a dialogue seeking understanding.
• Claims made in comments were often looking to critique a
flaw in a narrow area of the text’s logic, but sometimes ill-
informed about the text’s overall main argument.

30. Blended Reading: Student Feedback
• What did you find most useful about the tool?
• The tool helped streamline my reading and critical thinking process.
Instead of reading a text, annotating it, and then crafting a prose
response for it, I could more directly transfer my notes from the text
to specific points in the reading. If I had written down something on
page 50 of my book, I could easily flip to page 50 on the tool and
comment on the exact section. With the referenced text right next
to the comment, I could cut down on my summary and really
explore what was pertinent about the text.

31. Blended Reading: Student Feedback
• How did your use of the tool prepare you for discussion?
• I thought the tool was most helpful for class preparation by placing
students in conversation with each other before the class started.
Reading and replying to comments from other individuals allowed
me to gain a full new perspective and develop text-based
arguments before we even met for class. Additionally, the tool
provided me a chance to organize my thoughts about a text before
class on a diverse array of topics within the text.

32. Blended Reading: Student Feedback
• What was different about using this tool versus previous
ways of posting reflections?
• Before we started using the tool, our class had individuals write
reading responses. These reading responses gave us the same
opportunity to organize our thoughts and think critically about the
text before class. However, I found that these responses often were
too broad to be analytically effective. Additionally, they did not allow
students to converse over the text. Moreover, our reading
responses did not have the bonus of being directly appended to the
text. In this way, they had to spend more time talking about the text,
rather than what the text does, and they couldn't cover as many
topics in depth.

33. Blended Reading: Student Feedback
• How would you suggest improving the tool or improving its
use in the class?
• We only had two individuals per class use the tool. I would certainly
suggest even more students per class use the tool in order to
advance the discussion even more. As far as format - I think it
would be cool to be able to connect comments on different
passages.

34. Blended Reading: Instructor Insights
• Learning goals for “comprehension” versus
“conversation”:
Need to clarify to students the purpose of posting
questions on the text: distinguish between questions on
reading comprehension versus questions that prepare for
genuine in-class discussion.
• Shape classroom culture:
Tool provides opportunity to change classroom culture
toward more collaborative dynamic rather than practicing
individual claims.

35. Missed Alignments in Pedagogy
REMEMBER
UNDERSTAND
APPLY
ANALYZE
EVALUATE
CREATE
Blended
LearningOnline/
Out of class
Face-to-face/
In class
Most assigned readings
are at this level.
Most assigned papers
expect work at this level.
Bloom’s Revised Taxonomy (Anderson & Krathwohl, 2001)

36. Blended Reading: Challenges
• Pedagogy & student orientation:
Student focus: “assignment” layer – ‘apply’, ‘analyze’
Assigned readings: “transfer” layer – ‘evaluate’, ‘create’
Instructor focus in-class: Somewhere between “content”
and “assignment” – ensuring student understanding
through applied discussion.
• Where and how should students get explicit modeling
and teaching of the ‘analyze’ level of learning
expected in their papers?

37. Blended Reading: Challenges
• Accessibility:
NB.mit.edu is a great visual tool, but it is not accessible to
visually impaired users.
It is also not usable on mobile devices.
Overall, broad challenge with accessibility when using
online annotation tools.

38. BLENDED READING USING
COLLABORATIVE ONLINE
ANNOTATION
hari stephen kumar
@hariteach
Instructional Designer
Amherst College
hkumar@amherst.edu
THANK YOU!