1. Automatic Knowledge Structure Measures
in Online Courses
Roy B. Clariana, Kyung Kim, JooYoung Seo
Dept. of Learning and Performance Systems, College of Education
1
2. Presenter: Dr. Roy B. Clariana
Professor of Learning, Design, & Technology
2
The Story of “Knowledge Structure”
4. Ellen Taricani’s (2002) dissertation
• Undergraduates read Frank Dwyer’s heart text
• Then draw concept maps
• Treatment group receives an expert map as
‘feedback’, control group no feedback
• Posttest measures terminology, comprehension,
drawing
• My interest – do the concept maps relate to the
posttests?
But first, a few notes about concept maps…
4
5. Definition of concept map
• Novak (1972) concept maps are diagrammatic
representations of propositions arranged
hierarchically
• Mind maps (*Buzan, 1980s??))
• Semantic maps (Aly 1944)
5
(reflect specific epistemology or
belief about what knowledge is)
Theories embody or reflect an
epistemology, so I want to use
terminology that are ‘theory free’
6. Concept maps:
What can be measured?
“Regardless of the domain within which
structural knowledge has been investigated, the
concept of structural knowledge itself is thought
to consist of three components. Those three
components are (1) relevant domain concepts,
(2) the presence and/or nature of relationships
between those concepts, and (3) the strength of
those relationships. These components then
become critical to the definition and
measurement of the construct.” p.2
6
Suen, Hoi.K., & Murphy, L.C.R. (1999). Validating Measures of Structural
Knowledge through the Multitrait-Multimethod matrix. Presented at AERA.
7. Recording link and distance data of a
concept map
7 of 34
lungs
oxygenateddeoxygenated
pulmonary artery
pulmonary vein
left atrium
right ventricle
Link Array (linear, proposition specific)
a b c d e f g
a left atrium -
b lungs 0 -
c oxygenate 0 1 -
d pulmonary artery 0 1 0 -
e pulmonary vein 1 1 0 0 -
f deoxgenate 0 1 0 0 0 -
g right ventricle 0 0 0 1 0 0 -
Distance Array (relational)
a b c d e f g
a left atrium -
b lungs 120 -
c oxygenate 150 36 -
d pulmonary artery 108 84 120 -
e pulmonary vein 73 102 114 138 -
f deoxgenate 156 42 54 84 144 -
g right ventricle 66 102 138 42 114 120 -
moves through
to the
passes into
to the
Student’s concept map
raw data: (n2-n)/2 pair-wise comparisons
We noticed that participants’ spend
a lot of time making small moves
8. Distance raw data reduction by
Pathfinder KNOT
8 of 34
Pathfinder Network
a b c d e f g
a left atrium -
b lungs 0 -
c oxygenate 0 1 -
d pulmonary artery 0 1 0 -
e pulmonary vein
1
1 0 0 -
f deoxgenate 0 1 0 0 0 -
g right ventricle
0
0 0 1 0 0 -
Distance Array
a b c d e f g
a left atrium -
b lungs 120 -
c oxygenate 150 36 -
d pulmonary artery 108 84 120 -
e pulmonary vein 73 102 114 138 -
f deoxgenate 156 42 54 84 144 -
g right ventricle 66 102 138 42 114 120 -
lungs
oxygenateddeoxygenated
pulmonary artery
pulmonary vein
left atrium
right ventricle
Pathfinder network
(based on distances)
PFnet analysis: 21 distance data points reduced to 6 link data points
Pathfinder algorithm reduces the proximity raw
data using triangle inequality to define the
shortest path between all of the terms
9. In the NO feedback control, concept map link and
distance data contain different KS information
Taricani, E. M. & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps. Educational
Technology Research and Development, 53 (4), 61-78.
Poindexter, M. T., & Clariana, R.B. (2006). The influence of relational and proposition-specific processing on structural
knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184.
Taricani & Clariana (2006) Term Comp
Link data (linear props) 0.78 0.54
Distance data (relational) 0.48 0.61
9 of 34
Poindexter & Clariana (2006)Term Comp
Link data (linear props) 0.77 0.53
Distance data (relational) 0.69 0.71
KS to posttest correlations
10. What we learned here
• Open ended concept maps are difficult to score,
so constrain the activity by providing a list for
students
• Link data – verbatim propositional knowledge
• Distance data – relational knowledge, inferences
and comprehension
• Giving students an expert map after making their
own map wrecks posttest performance (note: we
saw similar damage for refutation texts in
Ntshalintshali's dissertation)
10
We sensed that we were on to something. So
what other ways of measuring KS are there?
11. A bigger picture …
11
Reading
Texting
TV
Radio
Conversations
Sign language
needs
concerns
feelings
empowerment
relationship
motivation
individual
productivity pay
plan
contingency
classical
efficiency
humanistic
measure
leadership
managementsuccess
focus
company
TQM
quality customers
goal
work
situation
employee
Knowledge structure
12. The notion of ‘above the line’ and
‘below the line’
12
The line
dialog
Church
School
Home
Pastor
Teacher
People
Meals
Auditorium
Podium
HymnalSongs
Lectern
Read
Concept mapping
Write an essay
Take IQ test
Production tasks
Implicit/tacit
Knowledge structure
Artifact structure
13. apposite KS specific cognitive tasks
“… over the course of learning, students knowledge
structures become more similar to this expert structure,
and students who have acquired better knowledge
structures tend to perform better on traditional
performance assessments (Jonassen, Beissner, & Yacci,
1993; Wilman, 1996). There is also some evidence that
knowledge structures, as reflected by conceptual
knowledge, may play a more direct causal role in enabling
good performance rather than simply reflecting
expertise.” (p. 427)
knowledge structure mental function
13
Trumpower, D.L., & Goldsmith, T.E. (2004). Structural enhancement of
learning. Contemporary Educational Psychology, 29, 426–446.
14. Aside: Why KS as noun associations?
• Deese and word net
• Burgess and HAL
• AI models (Elmann… Rumelhart…)
• Because I started this with concept maps that
link concepts (nouns, see above)
• It doesn’t make much sense to compare
different parts of speech, [red ------- water]
14
15. Application of word co-occurrences
as a visual wordnet
• Deese free recall lists WordNet: An on-line
lexical database http://kylescholz.com/projects/wordnet/
• Visual wordnet:
Many applications:
• Semantic web
• Word predictors
• Spelling apps
• Google
15
Fellbaum, Christiane (2005). WordNet and wordnets. In: Brown, Keith et al. (eds.),
Encyclopedia of Language and Linguistics, Second Edition, Oxford: Elsevier, 665-670
Deese–Roediger–McDermott (DRM) paradigm
16. Burgess: Large vector model of L1
• 70, 000 x 70, 000 terms matrix
• “The corpus that served as
input to the HAL model is
approximately 300 million
words of English text gathered
from Usenet newsgroups that
contained English text.
Properties of Usenet text that
were appealing were both its
conversational nature and its
diverse nature, making it
closer in form to everyday
speech.”
Burgess, C. (1998). From simple associations to the building blocks of language: Modeling
meaning in memory with the HAL model. Behavior Research Methods. Instruments. &
Computers, 30 (2), 188-198.
p.194
Generic L1 lexicon
16
17. Trained neural networks exhibit the
same sort of ‘categorization’
Elman, J.L. (2004). An alternative view of the mental lexicon. TRENDS in Cognitive Sciences, 8 (7), 301-306.
17
18. Dave Jonassen’s summary …
graph
building
similarity
ratings
semantic
proximity
word
associations
card
sort
ordered
recall
free
recall
additive
trees
hierarchical
clustering
ordered
trees minimum
spanning
trees
link
weighted
Pathfinder
nets
Networks
Dimensional
principal
components
MDS – multidimensional scaling
cluster
analysis
expert/
novice
qualitative
graph
comparisons
quantitative
graph
comparisons
relatedness
coefficients
scaling
solutions
C of PFNets
Trees
Knowledge
representation
Knowledge
comparison
Knowledge
elicitation
Jonassen, Beissner, & Yacci (1993), page 22
18 of 34
concept maps
written text
Social
network
analysis
19. Measuring KS in essays: started with
how reading seems to work
19 of 34
Hyona, J., & Lorch, R.F. (2004). Effects of topic headings on text processing: evidence
from adult readers’ eye fixation patterns. Learning and Instruction, 14, 131–152.
Figure 1, p.136
20. Knowledge structure (KS)
20 of 34
Imminent extinction pandas the climate
today
exclusively
in the wildlive
Imminent extinction
pandas the climate
Retrieval function
A B (propositional knowledge):
Where do pandas live? In the wild
A B,C,D (relational knowledge):
What do we know about pandas today?
Pandas are heading towards extinction in
the wild due to climate change
Retrieval structure
linear Notice for this reader: “imminent extinction”, “pandas” and “the climate”
enter the attentional sequence twice, the linear sequence begins to fold
21. Read KS Retrieval function
21 of 34
Relational
Retrieval structure Retrieval function
A B (propositional knowledge):
Where do pandas live? In the wild
A B,C,D (relational knowledge):
What do we know about pandas today?
Pandas are heading towards extinction in
the wild due to climate change
Thus the linear sequence has propositional and
relational information in the same trace (fuzzy trace theory)
a b c d
a 1 0.2 0.1 0.1
b 0.3 1 0 0.2
c 0.3 0 1 0.1
d 0 0.4 0.1 1
22. ALA-Reader: term links and distances
in students essays
22
Clariana, R.B., Wolfe, M. B., & Kim, K. (2014). The influence of narrative and expository text lesson text
structures on knowledge structures: alternate measures of knowledge structure. Educational Technology Research
and Development, 62 (4), 601-616.
Linear link data better than word distance data
23. Example: read Supertanker text
How can engineers (1) help prevent spills of oil (2) from supertankers (3)?
Supertankers (4) are huge ships (5) that carry oil (6) over the oceans (7).
A supertanker (8) can contain about a half-million tons of oil (9).
Such huge tankers (10) are each the size of five football fields (11).
A tanker's (12) cargo area could easily hold the Empire State Building (13).
Most of the world's oil (14) is transported by these supertankers (15).
Disasters (16) occur when wrecked tankers (17) spill oil (18) into the ocean (19).
As a result of these oil spills (20), the environment is damaged (21).
In 1967, a supertanker (22), named Torrey Canyon, crashed near England (23).
This crash resulted in washing ashore 200,000 dead seabirds (24).
In 1989, the tanker (25) Exxon Valdez (25) spilled oil (26) into Alaska's (27) coast.
As a result of it the 11 million gallons spilled (infer oil spill 28), 1,000 otters (29) died.
Oil spills (30) from tankers (31) also kill drifting microscopic (32) plant life.
These plants provide food for sea life (33), such as whales and shrimp (34).
They also produce 70 percent of the world's oxygen (35) supply.
Oil spills (36) result partly from limitations in supertankers' (37) engineering (38).
Supertankers (39) lack double bottom hulls (40) for extra protection.
They also lack extra power and steering equipment for safety (41).
They have only one boiler (42) to provide the ship power (43).
They have only one propeller (44) to steer (45) the huge ship (46).
Lack of such backup (47) components causes problems (48) during emergencies (49).
Emergency (49) situations include ocean storms (50) and coastal reefs (51). .
Solutions (52) to these problems (53) with oil spills (54) include three tactics. ·
Supertankers (55) must be built with added hulls (56), .boilers (57), and propellers (58).
These provide extra safety (59), control, and backup in emergencies (60).
Also, officers need top training (61) to run and maneuver their ships (62).
Supertanker (63) simulators (64) at some facilities provide top training (65).
Finally, ground control (66) stations should be installed near the shore.
Ground control (66) stations would act like airplane control (67) towers.
They would guide supertankers (68) safely on the oceans (69) along coasts.
This ensures safety (70) in shipping lanes and dangerous channels.
Manually create cmap
Manually select terms
23
24. ALA-Reader (an MS Excel file)
• ALA-Reader onetime setup (contrast with LSA)
– Add selected terms to the MS Excel file
– Correct terms for synonyms and metonyms and likely
misspelling
• Run lesson text and each student essay (e.g., copy
text from word file paste into excel file copy
excel file prx data paste into a new notepad file
save notepad prx file; repeat for each essay)
• Run Pathfinder KNOT on the notepad prx files
• See the next two slides of the pathfinder networks
(Pfnets, using the symmetric-undirected network
form) of the 18 essays that Bonnie collected in her
graduate class
Symmetric above, asymmetric below
24(you can see why this needs to be automated)
25. Ngram aside: why include Exxon Valdez and not
Torrey Canyon?
25
The answer to every question is ‘ngrams’
30. Published investigations
on this KS essay approach
30
Koul, R., Clariana, R.B., & Salehi, R. (2005). Comparing several human and computer-based
methods for scoring concept maps and essays. Journal of Educational Computing
Research, 32 (3), 261-273. link
Taricani, E. M. & Clariana, R.B. (2006). A technique for automatically scoring open-ended
concept maps. Educational Technology Research and Development, 54, 61-78.
Poindexter, M. T., & Clariana, R.B. (2006). The influence of relational and proposition-specific
processing on structural knowledge and traditional learning outcomes. International
Journal of Instructional Media, 33 (2), 177-184.
Clariana, R.B., & Wallace, P. E. (2007). A computer-based approach for deriving and measuring
individual and team knowledge structure from essay questions. Journal of Educational
Computing Research, 37 (3), 209-225. link
Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept
maps from a text passage. International Journal of Instructional Media, 35 (2), 229-
236. link
Clariana, R.B., Wallace, P.E., & Godshalk, V.M. (2009). Deriving and measuring group knowledge
structure from essays: The effects of anaphoric reference. Educational Technology
Research and Development, 57, 725-737.
Clariana, R.B. (2010). Deriving group knowledge structure from semantic maps and from essays.
In D. Ifenthaler, P. Pirnay-Dummer, & N.M. Seel (Eds.), Computer-Based Diagnostics and
Systematic Analysis of Knowledge (Chapter 7, pp. 117-130). New York, NY: Springer.
31. Published investigations
on this KS essay approach
31
Clariana, R.B., Wolfe, M. B., & Kim, K. (2014). The influence of narrative and expository text
lesson text structures on knowledge structures: alternate measures of knowledge
structure. Educational Technology Research and Development, 62 (4), in press. doi:
10.1007/s11423-014-9348-3
Kim, K., & Clariana, R.B. (2015). Knowledge structure measures of reader’s situation models
across languages: Translation engenders richer structure. Technology, Knowledge and
Learning, 20, 249-268. (L1L2)
Conference Presentations
Kim, K., Clariana, R., & Mun, Y. (2014). Using Pathfinder Network as a measure of lexical
structure of bilingual learners. Proceedings (full paper) of the 2014 IEEE International
Conference on Advanced Learning Technologies (ICALT), Athens, Greece: IEEE Computer
Society (L1L2)
Kim, K., & Clariana, R. B. (2014). Concept centrality: A useful and usable analysis method to
reveal mental representation of bilingual readers. Proceedings of Selected Research and
Development paper of the 2014 Association for Educational Communication and
Technology (AECT), Jacksonville, FL (pp. 117-124) (L1L2)
Dissertations Outside PSU: Vera Chen (2012, University of Missouri), Min Kyu Kim (2012?,
Georgia), Sabine Klois (2013, Radboud University Nijmegen), Ginger Howell (2014,
Capella)
32. Published investigations
on this KS essay approach
32
Thesis/dissertations at PSU
Fanella, D. (2015). The effects of changing the number of terms used to create proximity files on
the predictive ability of scoring essay-derived network graphs via the ALA-Reader
approach. PhD dissertation, https://etda.libraries.psu.edu/paper/26367/ (KS foundations)
Houston, V.C. (2014). Consequences of team charter quality: teamwork mental model similarity
and team viability in engineering design student teams. PhD dissertation,
https://etda.libraries.psu.edu/paper/20503/ (team collaboration)
Journal, submitted, under review
Kim, K. (under review). How the relationship between a heading and underline influences
second language reading comprehension: Knowledge structure analysis. Manuscript
submitted to the Instructional Science (L1L2)
Kim, K. (under review). The influence of first language in reading a second language expository
text: Knowledge structure analysis. Manuscript submitted to the Reading and Writing
(L1L2)
Kim, K, & Clariana, R. B. (under review). Text signals influence knowledge structure complexity
of readers: Knowledge structure analysis. Manuscript submitted to the Educational
Technology Research and Development (L1L2)
33. Presenter: Kyung Kim
Doctoral Candidate in Learning, Design, & Technology
33
GISK: Graphical Interface of Structural Knowledge
35. 35
Knowledge Structure for Blind Learners
Presenter: JooYoung Seo
Master Candidate in Learning, Design, & Technology
36. Project Aims
1. To extend the accessibility of the ALA-Reader.
2. To ensure equal access to KS feedback for
visually impaired learners.
3. To help blind learners improve their readings
and writings.
37. Traditional Ways
MCCC :: - Montgomery County Community College
banner
visited Link Graphic MCCC visited Link
navigation region
list of 7 items
Link ABOUT US
Link ACADEMICS
Link ADMISSIONS
Link STUDENT RESOURCES
Link CAMPUS LIFE
Link ALUMNI AND DONORS
Link ARTS
list end
navigation region end
navigation region
list of 1 items
Edit
search
Button
list end
navigation region end
Link Graphic login-button
banner end
123456
clickable
Link Graphic slideshow/251b00471f1c74a44efeb45a46eb4d84
Link Graphic slideshow/3f8b7c8f68406dd1707dfe98b35886d1
Link Graphic slideshow/3f8b7c8f68406dd1707dfe98b35886d1
heading level 3 Upcoming Events
Blind access vs. sighted access
Sequential/Linear navigation using screen reader
38. Traditional Ways
MCCC :: - Montgomery County Community College
banner
visited Link Graphic MCCC visited Link
navigation region
list of 7 items
Link ABOUT US
Link ACADEMICS
Link ADMISSIONS
Link STUDENT RESOURCES
Link CAMPUS LIFE
Link ALUMNI AND DONORS
Link ARTS
list end
navigation region end
navigation region
list of 1 items
Edit
search
Button
list end
navigation region end
Link Graphic login-button
banner end
123456
clickable
Link Graphic slideshow/251b00471f1c74a44efeb45a46eb4d84
Link Graphic slideshow/3f8b7c8f68406dd1707dfe98b35886d1
Link Graphic slideshow/3f8b7c8f68406dd1707dfe98b35886d1
heading level 3 Upcoming Events
Blind access vs. sighted access
Sequential/Linear navigation using screen reader
40. How to Make it Possible?
SVG
Scalable Vector Graphics
Tablet Based Haptic Output Paper Based Tactile Output
Android haptic
Feedback
Sonification
Swell Machine
or Braille
Embosser
