1. Top Concept Networks
of Health Education
Reflections
Alyssa Wise & Yi Cui
Sameen Reza
LAK ‘19, Tempe, Arizona
March 7th 2019
2. • Strategize effective
learning to meet goals
• Develop path to
lifelong learning as a
reflective practitioner
• Build a sense of
professional identity
Picture Credits: Susan Merrel 2016 dentistry.ucsf.edu/programs/post-grad/aegd
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Role of Reflection in Health Education
3. 2
One Graduated Class (“201X”)
Students 367
Practice groups
(25-30 stu per group)
14
Reflection Types
(each 2x a year)
6
Years of Dental School 4
Potential reflections
available for analysis
~17,500
deidentified
statements
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Online Dental Portfolio System
4. What they contained
Focus on “depth” of
reflection
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
From Coming-to-Know to Coming-to-Be
What they indicated
Focus on what reflections tell
us about students’ thinking
Prior Studies This Study
Manual analysis
Human coding by experts in
the domain
Automated analysis
Leverage computational
approaches
Can we identify shifts in students’ conception of the
profession and their identity as professionals ?
5. Use reflections to track changes in students’
conception of being a dental professional
from start to end of program
Identify potential linguistic markers to
detect changing student conceptions
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Objectives
6. Initial Study
• Close analysis of 2 Practice Groups
• Examined 2 Points in Time
• Start of Dental School - First reflection in D1
• End of Dental School - Last reflection in D4
• Qualitative examination to support interpretation
Recent Extensions
• Repeated study with all 14 practice groups
• Examined 8 Points in Time (Start /End all 4 years)
• Looked at students individually
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Overview of the Process
54 students
497 reflections
367 students
7510 reflections
7. Extract
Top Concepts
Pre Processing
(tokenization, stop words)
Create Network
of Co-occurrence
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Select
Reflections
Building Top Concept Networks
8. 7
GROUP A
Reflections 105
Sentences 475
Average Length 4.5
Unique Words
(Nodes) 1,139
Edges 18,618
Average Degree 16
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Start D1 End D4
GROUP B
Reflections 82
Sentences 283
Average Length 3.5
Unique Words
(Nodes) 833
Edges 14,227
Average Degree 17
GROUP A
Reflections 154
Sentences 995
Average Length 6.5
Unique Words
(Nodes) 1,604
Edges 35,147
Average Degree 22
GROUP B
Reflections 156
Sentences 1,176
Average Length 7.5
Unique Words
(Nodes) 1,746
Edges 41,375
Average Degree 24
Initial Network Overview
9. GROUP A
Word Freq %
professional (146) 2.80%
will (112) 2.15%
patient (104) 1.99%
become (85) 1.63%
dental (76) 1.46%
dentist (76) 1.46%
want (72) 1.38%
learn (60) 1.15%
work (59) 1.13%
good (53) 1.02%
year (40) 0.77%
hope (34) 0.65%
skill (27) 0.52%
8
GROUP B
Word Freq %
will (102) 2.99%
professional (83) 2.43%
patient (76) 2.23%
become (57) 1.67%
good (52) 1.52%
dental (51) 1.50%
hope (47) 1.38%
skill (47) 1.38%
year (46) 1.35%
dentist (45) 1.32%
learn (34) 0.94%
want (17) 0.50%
work (12) 0.35%
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Year 1 Top Concepts
10. GROUP A
Word Freq %
professional (146) 2.80%
will (112) 2.15%
patient (104) 1.99%
become (85) 1.63%
dental (76) 1.46%
dentist (76) 1.46%
want (72) 1.38%
learn (60) 1.15%
work (59) 1.13%
good (53) 1.02%
year (40) 0.77%
hope (34) 0.65%
skill (27) 0.52%
9
GROUP B
Word Freq %
will (102) 2.99%
professional (83) 2.43%
patient (76) 2.23%
become (57) 1.67%
good (52) 1.52%
dental (51) 1.50%
hope (47) 1.38%
skill (47) 1.38%
year (46) 1.35%
dentist (45) 1.32%
learn (34) 0.94%
want (17) 0.50%
work (12) 0.35%
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Year 1 Top Concepts
“I want to use the
hard knowledge and
skills I learn to
further my career”
“I know that professionals
need to develop necessary
inter-personal skills”
11. GROUP A
Word Freq %
patient (349) 3.12%
dental (244) 2.18%
professional (242) 2.16%
good (157) 1.40%
learn (147) 1.31%
will (146) 1.30%
become (123) 1.10%
care (114) 1.02%
school (107) 0.96%
year (103) 0.92%
feel (59) 0.53%
10
GROUP B
Word Freq %
patient (426) 3.17%
professional (285) 2.12%
dental (221) 1.65%
will (199) 1.48%
learn (168) 1.25%
feel (148) 1.10%
care (147) 1.10%
good (146) 1.09%
year (141) 1.05%
become (130) 0.92%
school (93) 0.69%
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Year 4 Top Concepts
12. GROUP A
Word Freq %
patient (349) 3.12%
dental (244) 2.18%
professional (242) 2.16%
good (157) 1.40%
learn (147) 1.31%
will (146) 1.30%
become (123) 1.10%
care (114) 1.02%
school (107) 0.96%
year (103) 0.92%
feel (59) 0.53%
11
GROUP B
Word Freq %
patient (426) 3.17%
professional (285) 2.12%
dental (221) 1.65%
will (199) 1.48%
learn (168) 1.25%
feel (148) 1.10%
care (147) 1.10%
good (146) 1.09%
year (141) 1.05%
become (130) 0.92%
school (93) 0.69%
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Year 4 Top Concepts
13. GROUP A
Word Freq %
professional (146) 2.80%
will (112) 2.15%
patient (104) 1.99%
12
GROUP B
Word Freq %
will (102) 2.99%
professional (83) 2.43%
patient (76) 2.23%
GROUP A
Word Freq %
patient (349) 3.12%
dental (244) 2.18%
professional (242) 2.16%
GROUP B
Word Freq %
patient (426) 3.17%
professional (285) 2.12%
dental (221) 1.65%
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Changes Over Time
Start D1 End D4
14. GROUP A
Word Freq %
professional (146) 2.80%
will (112) 2.15%
patient (104) 1.99%
13
GROUP B
Word Freq %
will (102) 2.99%
professional (83) 2.43%
patient (76) 2.23%
GROUP A
Word Freq %
patient (349) 3.12%
dental (244) 2.18%
professional (242) 2.16%
GROUP B
Word Freq %
patient (426) 3.17%
professional (285) 2.12%
dental (221) 1.65%
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Changes Over Time
Start D1 End D4
15. 14
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Concept Connections
Group A
Start D1 End D4
Group B
16. GROUP B
Word Freq (%)
clinician 20 (0.15%)
present 15 (0.11%)
issue 14 (0.10%)
teach 14 (0.10%)
remain 13 (0.10%)
attend 12 (0.09%)
speak 12 (0.09%)
surgery 12 (0.09%)
though 12 (0.09%)
tooth 12 (0.09%)
15
GROUP A
Word Freq (%)
graduation 22 (0.20%)
encounter 16 (0.14%)
clinician 11 (0.11%)
comprehensive 10 (0.09%)
almost 9 (0.08%)
honesty 9 (0.08%)
complex 8 (0.07%)
fill 7 (0.06%)
omfs 7 (0.06%)
sometimes 7 (0.06%)
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
“I have grown in knowledge and in skill; I have
learned to think like-a-clinician.”
Words Used Only in Year 4
17. • Overall increase D1->D4 in reflection amount,
concepts and connections
• Shift from global idea of becoming a good dental
professional in D1 to a more integrated patient-
centered conceptualization in D4
• Increasing frequency and centrality of patient may
be a useful indicator for this (desired) process.
• Initial signs in D4 of students starting to think of
themselves as members of the dental profession
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Summary of Insights from the Initial Study
18. 17
Objectives
• Identify if the shift towards patient centered thinking
occurs across all students
• Identify point(s) in time when shift occurs
• Identify students for whom the shift does not occur
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Recent Extensions
Data
All 367 students in 201X class
Across all 4 Years
7510 reflection statements
19. 18
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Start of D1
End of D4
Top Concept Networks for All Students
20. 19
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Top Concept Networks
Across 8 points in time
D1 Start
D1 End
D2 Start
D2 End
D3 Start
D3 End
D4 Start
D4 End
21. 20
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Top Concept Network: Start of D1
Start of D1
1020 Reflection Statements
22. End of D1
21
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
End of D1
513 Reflection Statements
Top Concept Network: End of D1
23. 22
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Start of D2
95 Reflection Statements
Top Concept Network: Start of D2
24. 23
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
End of D2
486 Reflection Statements
Top Concept Network: End of D2
25. 24
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Start of D3
940 Reflection Statements
Top Concept Network: Start of D3
26. 25
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
End of D3
1324 Reflection Statements
Top Concept Network: End of D3
27. 26
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Start of D4
1103 Reflection Statements
Top Concept Network: Start of D4
28. 27
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
End of D4
2109 Reflection Statements
Top Concept Network: End of D4
29. 28
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Start of D1
End of D4
Start of D3
Summary of Changes Over Time
“I started dental school with the focus of wanting to become
an excellent healthcare provider for my patients. I did not fully
understand what that meant until D3 and D4 year, when I
had my patients' oral health needs in my hands.”
30. START OF D1
Word Freq %
will 1071 2.62
professional 943 2.31
patient 866 2.12
good 682 1.67
become 681 1.67
dental 649 1.59
dentist 552 1.35
want 509 1.25
skill 474 1.16
hope 455 1.11
29
START OF D3
Word Freq %
patient 1321 3.07
professional 804 1.87
good 741 1.72
learn 618 1.44
will 603 1.4
become 531 1.24
dental 504 1.17
year 410 0.95
work 398 0.93
clinic 392 0.91
END OF D4
Word Freq %
patient 3936 3.01
professional 2454 1.87
good 2022 1.54
dental 1877 1.43
learn 1822 1.39
will 1814 1.39
become 1403 1.07
year 1348 1.03
care 1242 0.95
feel 1138 0.87
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Degree of
“patient” 1.15
Degree of “patient”
2.67
Degree of “patient”
3.53
Identifying Students Who Don’t Undergo Shift
31. Student X at Start of D1
30
INTRODUCTION MODEL INITIAL STUDY
RECENT
CONCLUSION
“Interactions and
communication with colleagues
and patients is also a key. This
helps to learn how to become
an understanding professional
who knows how to achieve
their patients needs and
communicate in an effective
manner.”
Identifying Students Who Don’t Undergo Shift
32. 31
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
I strongly believe in strong
determination, motivation and
hard work, which lead me to be
a part in dental school. After
almost 2 years in clinic I feel
that I have improved a lot in
my skills and learned various
techniques.
Student X at Start of D3
Identifying Students Who Don’t Undergo Shift
33. • Initial exploration has identified common trajectories
of students conception of becoming a dental professional
• Recent extensions have provided support to initial
results across a larger set of students and pinpoint
when the transition happens.
• In the process of refining markers that can identify
individual students who conform to or diverge from
these common patterns.
• Refining the notion of concept beyond simple word
usage (experimenting with conditional word embeddings)
32
INTRODUCTION MODEL INITIAL STUDY
RECENT
EXTENSIONS
CONCLUSION
Conclusions and Further Work
34. Thank You
Alyssa Wise | Yi Cui | Sameen Reza
@NYU_LEARN
@alywise
alyssa.wise@nyu.edu yca231@sfu.edu sr4918@nyu.edu
New York University Learning Analytics Research Network
Notas do Editor
Reflective practitioner in an ever changing field
Four boxes
Narrow focus allowed for careful interpretation of the analytics including follow up of how top concepts were used in the underlying data
System level data did not show much difference between groups (same number of reflections submitted)
Last two steps that we tried a bunch of different things
How to determine “top” and define “co-occurance”
Excited by our initial results were were eager to learn more.
The purpose of this extension was 1. VALIDATION. Do the results of the initial study extend across the complete set of students
2. Narowwing when in this 4 year time period does the change begin to appear?
3for the sake of actionable insight we wanted to identify ways to determine indivifually which students conformed to this general pattern of change and which did not
Thanks to our Dental portfolio system we had a rich data set that allowed us the affordance to carry out this analysis
In response to our validation question we looked athe top concept network created at the very start of the program and compared it to that produced at the very end. As you can see the shape of the network as well as the patient node is changes radically from D1 to D4. Thus we felt confident in assuming that the change was indeed taking place across the complete set of students
In response to our validation question we looked athe top concept network created at the very start of the program and compared it to that produced at the very end. As you can see the shape of the network as well as the patient node is changes radically from D1 to D4. Thus we felt confident in assuming that the change was indeed taking place across the complete set of students
2. Students were required to enter reflection once in each academic term. In orderIn order To Identify WHEN it appears to start happening, we collected statements posted at the very first and at the end of each dental year giving us 8 data points in time. lets see these results in a linear fashion from start to end.
There were approximately 1000 statements submitted at this data point.
The word patient was used quite frequently along with professional will and good
So here we had only about 500 statements which shows the sparseness of the graph as well. Accordingly patient and professional were both used with nearly the same frequency
The number of statements collected at this point in time were very low- only about a 100. patient is still connected to good will and dentist but its direct link with professional does not appear here
The number of statements increase to 500 again
Patient is now also associates with learn
Start of D3.. The number of statements increases to approximately 1000. tHere the graph assumes a radically circular structure and the node patient assumes the central position.
End of D4… there is not much change here
Start of D4. Frequent use of “patient” continues to rise
End of D4. Maximum use of word patient used along with good care learn treat
Becoming patient centered was an important find. It is something that is listed as a core competency in the dental school objectives. These results showed us that start of d3 isa critical time when this patient centered thinking starts to appear. Knowing that dental students start their clinics in D3 we expected this shift to happen sometime in D3 but what was surprising was that it appeared at the start of D3 rather than later. In students own words…. I…
We then looked closely at these datapoints in time to explore different ways in which we can identify patterns emerging from students thinking. One interesting feature was that only the use of the word “patient” was increasing from D1 to D3 and remained relatively stable towards D4.
Having identified that start of program, start of D3 and End of D4 were the most important times to look at , we took a deep dive into the results from the analysis at these stages.
We looked at the common derived results from all students
We compiled a list of words that was being used most frequently at each of the three points. Most of the words appeared to be the same.
We looked at what different was being said and discovered that students in D4 were mainly using more terms related to their profession and experience of being a dentist.
They were talking more about ethical issues, of upholding the prestige of their profession.
We tried to identify linguistic markers to see how reflections can help us cluster students who follow this positive trajectory and those that don’t. One thing that came up from our analysis was that very few words were going up in percentage. They were going up in numbers perhaps because of the larger number of reflections but not in percentage. “Patient” and “good” were the only words that went up in percentage from D1 to D3 and while “good decreased in percentage from D3 to D4 patient remained more stable.
The use fo the word patient was thenemployed to determine students who conformed to the pattern and who did not. Here is an example fof the top concept network of an individual student who was more patient centered in d1 thinking about achieving his patients needs.
Switch graph to left
With this one feature in hand, we tried to see for which student does this phenomenon not happen. Which students don’t talk more about patients in D3 compared to D1 and this is an example of such a case. This student talks more about patients in D1 than D3 contrary to the pattern that we had observed. The graph is now this individual students top concept graph in D1
However, s time progresses some students get more involved in their own academic progression. Their focus on patients in less obvious in their reflections. The same student in D3 was not thinking more about his improvement and skill development.
We are happy that this extension provided support to initial results and help pinpoint a key time in the academic progression where students undergo a change in thinking. We are currently working on refining markers beyond use of words and progress towards concepts. For this we are experimenting how conditional word embeddings can be made useful
Thank you for listening to our presentation. If you want to get any research update from us or NYU LEARN, please keep contact with us.
If you have any question, we’re very happy to answer it. Do you have any question?