1. Data Driven Decision Making
Amy Reitz
Manager, Consulting Services
Hobsons K-12

2. Agenda
Overview of Data Driven
Decision Making
Building a Data Culture
• Staff Workshops
• Student Workshops

3. Overview of Data Driven
Decision Making

4. Data Driven Decision Making is…
DDDM
D3M
The collection and analysis of data to make
decisions that improve student success.
Continual evaluation accompanied by
incremental changes.
Translation of data into knowledge and
actionable strategies.
Collaboration and communication throughout
the school, district and community.

5. Theme 1: Use data to make decisions
Data Decisions

6. What we want to happen
Helpful
Data
I need
to…

7. What happens in reality
Teacher
Evaluations
Partner
Assessments
SAT
GPA
ACT
Attendance
Activities
Standardized
Test Scores
Dem
ographics
Classroom
Activities
PSAT
AP
IBIEP
CoursePla
SchoolHistory
1600
4.0
32
1.7
365
???

8. Theme 2: Focus on outcomes

9. How do you measure success?
Staff and students
have completed all of
their assigned tasks.
Students are career
and college ready.
Productivity Outcome

10. Theme 3: Dig deeper
Illustrated via an example:
If staff members from a school
attend NSI, is the Naviance
student usage at that school
positively affected?
Following slides are from a case study for a large urban district
with staff members attending NSI 2012.

11. Impact of NSI on Student Usage
*Active Student User = Student that has logged in at least once.

12. Dig deeper
What is wrong with this analysis?
• It doesn’t consider the past.
• Schools with NSI attendees could have
already had higher usage.
How can it be improved?
• Compare growth rates.

13. Impact of NSI on Student Usage
*Active Student User = Student that has logged in at least once.

14. Dig deeper
What is wrong with this analysis?
• It doesn’t consider other support.
• Schools with NSI attendees could have had
additional support from Naviance staff.
How can it be improved?
• Include another variable: interaction with a
consultant.

15. Impact of NSI on Student Usage
Average Logins per Student
*Average logins per student include students with 0 logins.
Schools without NSI
Attendees
Schools with NSI
Attendees
Schools without
Consultant Interaction
Schools with
Consultant Interaction
1.5 2.9
4.7 5.8

16. Dig deeper
What is wrong with this analysis?
• It doesn’t consider more than 2 variables.
• Schools with NSI attendees could have had
additional training or other student variables
could influence usage.
How can it be improved?
• Analyze multiple variables to build a
predictive model.

17. Multivariable Regression
y = 10x + 2w – 5z
outcome variables
coefficients

18. Impact of NSI on Student Usage
Output of Multi-Variable Regression for Unique Logins:
Positive Coefficients

19. Impact of NSI on Student Usage
Output of Multi-Variable Regression for Total Logins:
*Total logins for students with 1+ login.
Positive Coefficients
Statistically
Insignificant
(p > 0.05)

20. Impact of NSI on Student Usage
Regression Coefficients
Unique Logins Total Logins
NSI 0.012 0.96
Consulting 0.122 2.78
Training 0.321 N/A

21. Dig deeper
What is wrong with this analysis?
• It doesn’t consider individual student
variables.
- Did an analysis with gender and class year, but
they were statistically insignificant. Still many
other variables that could be included.
How can it be improved?
• Additional variables.
• Sensitivity analysis and other statistical
models.
• Larger sample size across multiple schools,
districts, and regions.

22. Building a Data Culture

23. DDDM is a culture
To be truly effective, DDDM
needs input from everyone.
Everyone needs to see value
and be invested in collecting and
analyzing data.
Staff need to openly collaborate
and take action based on data.
In some cases, this requires a
huge attitudinal shift. Be
prepared to facilitate.

24. Building a Data Culture:
Staff Workshops

25. Staff Workshops
Involve multiple staff
members from various roles
in the development of data
processes.
Collaborate to make the
best possible decisions.
Use data for decisions and
information, not just
compliance.

26. Staff Workshop: Report Review
Purpose: Review the reports in Naviance and identify
needs.
Activities:
• Review reports in Naviance.
• Identify helpful reports.
• For each report, determine:
- Audience: Who should receive this report?
- Parameters: Which students/tasks/variables should be
included?
- Frequency: When and how often should this report be run?
Next Steps:
• Determine data needed to populate report.
- Ensure data is collected during activities throughout the
year.
• Customize and schedule reports in Naviance.

27. Staff Workshop: KPIs & Outcomes
Purpose: Define the key performance indicators
and outcomes that are important.
Activities:
• Brainstorm student outcomes. What does it
mean for students to be successful?
• For each outcome, determine associated KPIs.
- Addendum: Key Performance Indicators
Next Steps:
• Document and communicate KPIs and
outcomes.
• Map KPIs and outcomes to Naviance activities
and reports.

28. Staff Workshop: Identify Variables
Purpose: Identify variables that should be tracked to link
to outcomes and KPIs.
Activities:
• Review identified outcomes and KPIs.
• Brainstorm variables that could impact outcomes.
• Determine how variables are tracked and stored.
- SIS
- Naviance Activities
- Naviance Surveys
- Other
Next Steps:
• Incorporate into Naviance activities and data
collection.
- Addendum: Data Collection in Naviance
• Develop maintenance plan.

29. Staff Workshop: Survey Development
Purpose: Create surveys to collect data and inform
decisions.
Activities:
• Review previously identified needs.
- Direct data collection.
- Indirect collection through reflection and and
feedback.
• Brainstorm and organize questions.
Next Steps:
• Setup surveys in Naviance.
• Incorporate survey(s) into activities throughout
the year.

30. Staff Workshop: Scope & Sequence
Purpose: Define a plan for the activities that need to
occur throughout the year.
Activities:
• Review available activities in Naviance.
• Review previously identified data needs.
• Review suggested activities in Naviance
Implementation Guide and Naviance Network.
• Develop a plan for the activities to be completed by
students and staff throughout the year.
Next Steps:
• Document and communicate scope and sequence.
• Map to tasks in Success Planner and assign to
students.

31. Staff Workshop: Data Validation
Purpose: Review imported and input data to verify
accuracy and completeness.
Activities:
• Review data import history in Naviance.
• Review student profiles. Note inaccuracies or
incomplete entries.
- For imported data, correct the source.
- For input data, correct in Naviance.
• Review regular reports to verify accuracy.
Next Steps:
• Determine any patterns to identify processes to
be corrected.

32. Staff Workshop: Progress Reviews
Purpose: Regularly review progress against
KPIs and scope and sequence.
Activities:
• Review regularly scheduled reports.
• Identify successes and discuss lessons
learned.
• Identify lagging indicators. Brainstorm
causes and discuss solutions.
Next Steps:
• Make changes to address challenges.

33. Staff Workshops
What else have you
done at your school
or district?
What else have you
done at your school
or district?

34. Building a Data Culture:
Student Workshops

35. Student Workshops
Get relevant input from
students.
Help students understand
data driven decision
making.
Bolster college going
culture.
Supplement college and
career planning activities.

36. Student Workshop: Data Validation
Purpose: Verify student demographics, contact
information, and other profile information.
Activities:
• Provide background on importance of profile
information.
• Have students review their Naviance profiles
for incorrect information.
- Add email addresses.
- Note inaccuracies to be corrected in source by
staff (SIS).

37. Student Workshop: College & Career
Survey
Purpose: Introduce students to basic principles of
data collection in the context of post-secondary
planning and readiness.
Activities:
• Provide background on survey basics: question
development, tools, distribution.
• Students develop and distribute college and
career surveys.
• Provide background on basic survey analysis.
• Students analyze and present their results.

38. Student Workshop: Identify Variables
Purpose: Identify additional variables and
challenge students to consider the challenges and
steps leading to their post-secondary plan.
Activities:
• Provide students with an outcome to consider.
• Provide background information about the
concept of variables and some possible
variables affecting the identified outcome.
• Students brainstorm and present the variables
they think are important.

39. Student Workshops
What else have you
done at your school
or district?
What else have you
done at your school
or district?

40. Questions & Discussion

41. Addendums
Key Performance Indicators
Data Collection in Naviance
Additional Resources

42. Your Feedback Matters!
Thank you for attending the
Naviance Summer Institute 2013!
We greatly appreciate your feedback, please
complete a brief evaluation for this session at:
http://go.naviance.com/evaluations

43. Addendum: Key
Performance Indicators

44. Focusing your analysis
Outcomes are the ultimate goal.
Variables are the many data points for each
student. They include everything that affects a
student’s outcomes.
Key performance indicators are measurements
to determine if you are on track to attain a
particular outcome.
This addendum includes suggested KPIs using
data collected in Naviance.
Note: The source for the following slides in this addendum is the
2012 NSI presentation by Todd Bloom: KPIs for College and
Career Readiness.

45. Student Growth & Proficiency
Grade Point Average
Test score averages
• PLAN
• PSAT
• SAT
• ACT
• State assessment(s)
International
Baccalaureate scores
International
Baccalaureate scores by
course
% of students who used
PrepMe at least once
% of students who
complete the learning style
assessment
% of students who
complete Do What You
Are assessment
% of students who
complete Career Key
assessment
% of students who
complete a Course Plan
Course Plan Rigor
distribution

46. College Planning
College Power Score
distribution
Alignment of Course
Demand Forecast with
college readiness
curriculum determined by
school/district
Student interest in specific
courses that school/district
indicate align with college
readiness goals
Number of applications for
individual colleges
Number of applications for
individual colleges
% of students who submit
one or more college
applications
% of students admitted to
one or more colleges
% of students who intend
to attend college after
graduation
Meaningful and up-to-date
scholarship database
available for student use

47. Career Planning
% of students who identify careers and career
clusters of interest
% of students interested in professional careers
% of students interested in technical careers
% of students interested in careers with specific
characteristics, such as STEM, that are
determined by the school/district

48. Student Engagement
% of students who report they
understand the knowledge
and skills necessary for
success in their careers of
interest
% of students who set goals
% of students who met goal
% of students who completed
tasks that align with college
and career readiness as
determined by the
school/district(e.g. FAFSA
completion, internship/
mentorship requirement)
% of students who report
understanding their learning
styles
% of students who report
they have explored colleges
and careers based on
learning style assessment
% of students who report they
understand the links between
careers, preparation needed,
college major and projected
income

49. Alumni Performance
% of students who enrolled in
college
% of students who completed
college degrees
% of students who completed
college degrees within a
specified timeframe
% of students with positive
perceptions of college and
career readiness
% of students satisfied with
teaching or other specified
aspects of their K-12
experience
% of students who are
satisfied with their post high
school plans
% of students who enrolled in
remedial college
mathematics, English or other
courses
% of students who completed
remedial college math,
English or other courses

50. Addendum: Data Collection
in Naviance

51. Collecting Data in Naviance
Data comes into Naviance from various sources in
multiple ways.
This means that data quality can vary.
Poor data quality means less accurate analysis.
This addendum will cover some of the ways you can
improve the quality of data you are collecting.
Data is input into Naviance via various methods.
Some methods include:
• Student Information (data import)
• College Planning
• Career Planning
• Course Planning
• Success Planning
• Surveys

52. Collect Data: Student Info
Keep student data up to date.
• Automate data import
• Define a process for importing test scores
regularly
• Import all data that would be helpful for analysis.
• Review data import and data import
templates to determine what is missing and
update.
- Data import file: setup > data import > data
import history > view
- Templates located in the Help Library.

53. Collect Data: College Planning
• Engage students before senior year to add
colleges to their prospective list.
• Use Senior Survey to improve accuracy of
outcome reports.
• Turn off the option to allow students to delete
active applications.
• connections > family connection > select and update
optional features > delete active applications
• Import college application data from previous
years.

54. Collect Data: Career Planning
• Create a scope & sequence for career planning
activities that students will complete in grades 6-
12.
• Use CCR Curriculum to improve rollout and
add context for students.
• Leverage class or advisory time to work through
activities with students to ensure completion.
• Setup computers at career fairs for students to
add careers to their list.

55. Collect Data: Success Planning
• Use Success Planning to assign tasks that
improve career and college data.
• Link tasks to activities where possible.
• For example, instead of manually marking
that a student completed a workshop, create
a post-workshop survey and assign the
survey as a task.
• Utilize the built-in tasks.
• Schedule planner reports to regularly assess
progress.

56. Collect Data: Course Planner
• Configure the rigor levels for the College Power
Score.
• courses > configuration > total potential
course rigor
• Configure the rigor level for courses in the
course catalog.
• Can be included in the course catalog
import.
• Can be set manually in Naviance
- courses > course catalog > edit >
instructional level

57. Collect Data: Surveys
• Send out frequent, short surveys instead of
long, annual surveys.
• Opt for question types that make aggregate
analysis easier.
• Send surveys as a direct link via email (doesn’t
require students to login).
• connections > e-mail > send email to a group of students
and parents > select options and attach survey > preview
and send > “check this box if you want to include a direct
link to the survey”

58. • Addendum: Additional
Resources

59. Naviance Resources
• Naviance Network Community Forums:
http://community.naviance.com/t5/Community-
Forums/ct-p/succeed
• Naviance Network Help Library, Reporting
Section:
http://community.naviance.com/t5/Reporting/tkb-
p/Reporting%40tkb

60. Workshop Resources
• ATLAS – Looking at Data:
http://www.nsrfharmony.org/protocol/doc/atlas_l
ooking_data.pdf *
• Data.gov in the Classroom, Education Materials:
http://www.data.gov/education/page/datagov-
classroom
* Thanks to a participant in the NSI 2012 DDDM session for this
suggestion.

61. MS Office Resources
• Office Support: http://office.microsoft.com/en-
us/support/
• VLOOKUP (joining data in Excel):
http://office.microsoft.com/en-us/excel-
help/vlookup-HP005209335.aspx
• Excel Review, Duke University:
https://faculty.fuqua.duke.edu/~pecklund/ExcelR
eview/ExcelReview.htm

62. Misc Stats and Analysis Resources
• Data Mining: The Tool of the Information Age
Revolution, Rajan Patel, Stanford (recorded
webinar):
http://myvideos.stanford.edu/player/slplayer.aspx?
coll=2e431434-84e4-4de0-81c9-
76035c36a18f&co=12138da9-eab8-405b-a06f-
cc11f12e5871&w=true
• Introduction to Statistics and Data Analysis,
University of Michigan (open course materials):
http://open.umich.edu/education/lsa/statistics250/sp
ring2013