How can a university that already has very high levels of student performance and retention use data from its Blackboard® learning management system to identify effective teaching practices and at risk students? Based on experience gained from a year-long pilot of Blackboard Analytics™ for Learn at Emory University, this presentation will discuss (1) several unique challenges associated with the use of Blackboard Analytics™ to monitor high performing students, (2) the value of Blackboard Analytics™ as a data warehouse against which to run custom queries and apply more sophisticated data mining techniques, and (3) several preliminary insights obtained through the application of those techniques at Emory University.

1. #BbWorld14
Challenges and Opportunities for
Promoting Success among the
Successful using Blackboard
Analytics™
Timothy D. Harfield / July 17, 2014
#EmoryLA

2. Outline
1. Why Learning Analytics?
2. Blackboard Analytics™ Pilot Summary
3. Preliminary Insights
4. Next Steps
2
#EmoryLA
#BbWorld14

3. 3
ABOUT ME
#EmoryLA
#BbWorld14

4. 4
#EmoryLA
#BbWorld14

5. #EmoryLA
#BbWorld14

6. 6
#EmoryLA
#BbWorld14

7. #EmoryLA
#BbWorld14

8. #EmoryLA
#BbWorld14

9. 9
#EmoryLA
#BbWorld14
It’s Cool It’s Hot

10. 10
ABOUT EMORY UNIVERSITY
#EmoryLA
#BbWorld14

11.  Founded in 1836
 11 Schools
(4 Undergraduate, 7 Graduate)
 14,513 Students
(7,836 Undergraduate, 6,677 Graduate & Professional)
 Average Freshman Retention Rate: 95.3%
 4-year Graduation Rate: 84%
 94% College Success Rate (<> DFW)
 US News National University Ranking #20
Fact & Figures as of Fall 2013 (http://www.emory.edu/home/about/factsfigures/)
#EmoryLA
#BbWorld14
A destination university internationally recognized as an inquiry-driven, ethically engaged,
and diverse community, whose members work collaboratively for positive transformation in
the world through courageous leadership in teaching, research, scholarship, health care,
and social action.

12. 12
43%
19%
11%
10%
5%
3% 3%
1% 1% 1% 1% 3%
A A- B+ B B- C+ C C- D+ D F W
#EmoryLA
#BbWorld14

13. 13
#EmoryLA
#BbWorld14
It’s Cool It’s Hot

14. “If you invest in educational
technology for the sake of fashion,
you might just find yourself
wearing nothing at all”
14
#EmoryLA
#BbWorld14

15. An Opportunity
Learning Analytics in Support of Emory’s Strategic Vision
“Ensuring Highest Student Quality and Enhancing
the Student Experience”
“Strengthening Faculty Distinction”
15
#EmoryLA
#BbWorld14

16. 16
#EmoryLA
#BbWorld14
Why
Blackboard
Analytics for
Learn™ ?
Limitations of
existing analytic
capabilities
High speed and
low cost of
implementation

17. 17
CHALLENGES
Blackboard Analytics Pilot
#EmoryLA
#BbWorld14

18. Challenges
1. Unrealistic Expectations
18
#EmoryLA
#BbWorld14

19. Challenges
2. Data Overload
19
#EmoryLA
#BbWorld14
Data
Information Knowledge Wisdom
Data Warehousing Data Mining Analytics

20. Challenges
3. Unexpected consequences
20
#EmoryLA
#BbWorld14
Yes
46%
No
12%
Maybe
42%
Would You Like to Have Access
to the Activity Compared to
Others Dashboard in Other
Courses?
6%
3%
27%
43%
21%
Dashboard
is Easy to Use
Overall
3%
23%
47%
15%
12%
Dashboard
Is Useful
Overall
Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree

21. Challenges
3. Unexpected consequences
21
#EmoryLA
#BbWorld14
“I believe it's important for professors to have this tool
available to them, so they can see early if we're not turning
up for class enough and intervention might be warranted. I
do know that I need to get in a habit of logging onto BB at
least twice a day. I need to make it part of my daily life”
“I liked being able to compare my activity to
others, but I don't think it changed my
behavior. It may be useful to others.”
“I think its useful if someone is concerned
about how they are doing in the class or if
they are uncertain about how much time
they should be committing to the class. I
personally did not find it useful but I can see
how someone may use it as a tool to
determine if they are meeting expectations”
“I think this system also puts undue pressure on students to
log in or click on specific items in blackboard more, without
taking into consideration factors such as work done offline
in the class or via email. This is especially true for distance
learners. I personally cut and paste many of my assignments
in Word so they are easier to view and manipulate. I also
like to keep files from each semester so I do a considerable
amount of work offline and I am doing very well in my
classes so far. When I logged in to check my activity, the data
I saw pretty much showed the opposite-it looked like I
wasn't very engaged at all.”
“If this is a tool that will be used by
Professors to grade participation I would
have likely used it more; however, if this tool
were indeed used by Professors it may cause
students to post unnecessarily.”

22. “Analytics doesn’t DO anything;
People do analytics”
22
#EmoryLA
#BbWorld14

23. 23
Preliminary Insights
#EmoryLA
#BbWorld14

24. Preliminary Insights
Blackboard Use
Blackboard use at Emory University is far higher than
previously thought
24
#EmoryLA
#BbWorld14
62%
38%
Course Items Added No Course Items Added
5% 5%
0%
25%
50%
75%
100%
Course Items Added No Course Items Added
DFW Rate by Blackboard Use
Course Items Added No Course Items Added
… but it has no effect on student success

25. Preliminary Insights
Course Design
Engaged instructors make successful students
25
#EmoryLA
#BbWorld14
5.3%
6.1%
4.8% 4.8%
0%
5%
10%
Files Items
DFWRate
Items vs Files
5.3% 5.4%
1.0%
4.0%
0%
5%
10%
Course Links External URLs
DFWRate
Internal Links vs External URLs
Not Used Used

26. Preliminary Insights
At Risk Students
26
#EmoryLA
#BbWorld14
6%
4%
0%
5%
10%
15%
20%
25%
Male Female
DFW Rate by Gender
9%
4%
0%
5%
10%
15%
20%
25%
Transfer Student Non-Transfer Student
First Year DFW Rate by Transfer Status
3% 4%
7%
6%
0%
5%
10%
15%
20%
25%
Freshman Sophomore Junior Senior
DFW Rate by Class

27. Preliminary Insights
Predictive Analytics …
… it’s hard to do
27
#EmoryLA
#BbWorld14
3%
6%
9% 9%
0%
2%
4%
6%
8%
10%
1st 2nd 3rd 4th
DFW Rate by Minutes Quartile
3%
6%
8%
7%
0%
2%
4%
6%
8%
10%
1st 2nd 3rd 4th
DFW Rate by Interactions Quartile
6%
3%
9%
4%
0%
2%
4%
6%
8%
10%
High Grades & Engaged High Grades & Unengaged Low Grades & Engaged Low Grades & Unengaged
DFW Rate by Blackboard Risk Assessment

28. 28
NEXT STEPS
#EmoryLA
#BbWorld14

29. Next Steps
1. Data Wrangling Approach
2. Community of Interest
3. Workshops
4. Learning Analytics Speaker Series
5. Georgia Learning Analytics Symposium
29
#EmoryLA
#BbWorld14

30. Timothy D. Harfield
Scholar in Residence (Learning Analytics)
timothy.harfield@emory.edu
@tdharfield
Please provide feedback for this session:
1. Select “Schedule” icon in BbWorld14 Mobile App
2. Find Challenges and Opportunities for Promoting
Success among the Successful using Blackboard
Analytics™
3. Click “Tap here to take a survey.”
THANKS!