SlideShare uma empresa Scribd logo
1 de 15
Baixar para ler offline
Improving	
  Evalua/ons	
  and	
  U/liza/on	
  with	
  
Sta/s/cal	
  Edge:	
  Nested	
  Data	
  Designs	
  and	
  
Hierarchical	
  Linear	
  Modeling	
  (HLM)	
  
	
  
CES	
  Conference	
  -­‐	
  June	
  10	
  2013	
  
Marci	
  Pernica	
  -­‐	
  Ministry	
  of	
  Community	
  and	
  Social	
  Services	
  
Judith	
  Godin	
  –	
  J	
  Godin	
  Consul7ng	
  
•  The	
  goal	
  of	
  this	
  presenta0on	
  is	
  to	
  introduce	
  
the	
  concept	
  of	
  HLM	
  and	
  explain	
  how	
  it	
  can	
  be	
  
used	
  in	
  program	
  evalua0on	
  	
  
	
  
Introduc0on	
  
What	
  is	
  Hierarchical	
  Linear	
  Modeling	
  (HLM)	
  
•  HLM	
  is	
  a	
  sta0s0cal	
  technique	
  to	
  analyze	
  data	
  
that	
  is	
  structured	
  in	
  hierarchies	
  (or	
  “nested”)	
  
•  To	
  account	
  for	
  the	
  fact	
  that	
  people	
  that	
  are	
  “clustered”	
  or	
  
“nested”	
  within	
  the	
  same	
  group	
  have	
  more	
  in	
  common	
  than	
  
if	
  they	
  were	
  independent	
  random	
  samples	
  
	
  	
  
Classroom	
  1	
  
Classroom	
  
2	
  
Classroom	
  3	
  
Student	
  4	
  
Student	
  	
  
2	
  
Student	
  5	
  
Student	
  3	
   Student	
  8	
  Student	
  7	
  Student	
  6	
   Student	
  9	
  
Student	
  
10	
  
Student	
  
11	
  
Student	
  
12	
  
Nested	
  
data	
  
designs	
  
Student	
  
Class	
  
School	
  
District	
  
Hierarchical	
  Structure	
  –	
  mul0-­‐level	
  	
  
age	
   I.Q	
  
Measuring	
  
test	
  scores	
  
(dependent	
  variable)	
  
Independent	
  
variables	
  
•  HLM	
  enables	
  a	
  more	
  robust	
  analy0c	
  approach	
  
for	
  nested	
  data	
  (than	
  regression	
  or	
  ANOVA)	
  
•  Data	
  in	
  evalua0on	
  are	
  oZen	
  nested	
  
•  To	
  determine	
  success	
  condi*ons	
  for	
  the	
  
program	
  –	
  e.g.	
  is	
  the	
  program	
  more	
  suitable	
  
for	
  certain	
  sub-­‐popula0ons	
  or	
  more	
  successful	
  
if	
  delivered	
  in	
  a	
  certain	
  way	
  
Why	
  Use	
  HLM	
  in	
  Evalua0on	
  
Program	
  design	
  structure	
  
Data	
  structure	
  
Evalua0on	
  ques0ons	
  
-­‐	
  Which	
  par0cipant	
  or	
  site-­‐level	
  characteris0cs	
  are	
  most	
  influen0al	
  in	
  
explaining	
  the	
  varia0on	
  in	
  test	
  scores	
  among	
  the	
  program	
  
par0cipants?	
  
-­‐	
  What	
  program	
  delivery	
  characteris0cs	
  (site	
  level	
  prac0ces)	
  seem	
  to	
  be	
  
having	
  the	
  most	
  posi0ve	
  impact	
  on	
  the	
  par0cipants’	
  test	
  scores?	
  
-­‐	
  Are	
  some	
  program	
  features	
  more	
  suited	
  to	
  certain	
  sub-­‐popula0ons	
  (e.g.	
  
gender,	
  age	
  group,	
  ethnic	
  or	
  cultural	
  group)	
  
	
  
Applying	
  HLM	
  in	
  Evalua0on	
  
Par0cipant	
  
1	
  
Site	
  1	
   Site	
  2	
   Site	
  3	
  
Par0cipant	
  
4	
  
Par0cipant	
  
2	
  
Par0cipant
5	
  
Par0cipant	
  
3	
  
Par0cipant	
  
8	
  
Par0cipant	
  
7	
  
Par0cipant	
  
6	
  
Par0cipant	
  
9	
  
Par0cipant	
  
10	
  
Par0cipant	
  
11	
  
Par0cipant	
  
12	
  
Example	
  of	
  levels	
  of	
  a	
  hierarchical	
  model	
  
Par0cipants	
  (level	
  1)	
  nested	
  within	
  sites	
  (level	
  2)	
  
Assessing	
  test	
  scores	
  by	
  age	
  from	
  site	
  to	
  site	
  
Test	
  Score	
  
Age	
  
Four	
  different	
  
program	
  sites	
  
Although	
  the	
  test	
  scores	
  differ	
  from	
  site	
  to	
  site,	
  the	
  
rela0on	
  between	
  age	
  and	
  test	
  score	
  is	
  the	
  same	
  at	
  
different	
  sites	
  
Student	
  1	
   Student	
  2	
   Student	
  3	
  
Baseline	
  
Month	
  7	
  
Month	
  1	
  
Month	
  12	
  
Month	
  6	
   Month	
  9	
  Month	
  8	
  Month	
  6	
   Baseline	
   Month	
  2	
   Month	
  4	
   Month	
  6	
  
Assessing	
  change	
  over	
  0me	
  
Assessments	
  across	
  0me	
  (level	
  1)	
  are	
  nested	
  within	
  individuals	
  
(level	
  2)	
  (i.e.	
  repeated	
  measures	
  design)	
  
Assessing	
  improved	
  performance	
  over	
  0me	
  
Test	
  Score	
  
Time	
  
Four	
  different	
  
study	
  par0cipants	
  
Although	
  some	
  individuals	
  have	
  higher	
  test	
  scores	
  
to	
  start	
  with,	
  the	
  rate	
  of	
  change	
  (improved	
  
performance)	
  is	
  comparable	
  among	
  the	
  
par0cipants	
  
Tradi0onal	
  Methods	
  
Test	
  Score	
  
Age	
  
Rela0on	
  between	
  age	
  and	
  
test	
  score	
  es0mated	
  once	
  for	
  
all	
  sites	
  together	
  
Advantages	
  of	
  HLM	
  
Test	
  Score	
  
Age	
  
Four	
  different	
  
program	
  sites	
  
Here,	
  the	
  rela0on	
  between	
  age	
  and	
  test	
  score	
  varies	
  
across	
  sites.	
  
Are	
  there	
  any	
  site	
  level	
  variables	
  associated	
  with	
  the	
  
strength	
  of	
  this	
  rela0on?	
  
Design	
  Considera0ons	
  for	
  Using	
  HLM	
  
•  Sample	
  size	
  
– Par0cipant	
  level	
  
– Site	
  level	
  
– Repeated	
  measures	
  
•  Missing	
  Data	
  
– Can	
  be	
  easy	
  or	
  difficult	
  to	
  deal	
  with	
  
•  Number	
  of	
  variables	
  
– Comprehensive	
  coverage	
  
– Parsimony	
  
	
  
Final	
  Thoughts	
  	
  
•  Applying	
  HLM	
  in	
  evalua0ons	
  with	
  nested	
  data	
  
enables	
  more	
  robust	
  results	
  and	
  conclusions	
  
•  U0liza0on-­‐focused	
  	
  
– Iden0fy	
  evidence-­‐based	
  success	
  factors	
  or	
  
condi0ons	
  for	
  improving	
  the	
  program	
  delivery	
  
model,	
  to	
  ul0mately	
  achieve	
  beger	
  program	
  
effec0veness	
  	
  
– Promo0ng	
  the	
  value	
  in	
  evalua0on	
  (gathering	
  the	
  
evidence	
  to	
  determine	
  the	
  ‘success	
  factors’	
  for	
  
the	
  interven0on	
  to	
  be	
  effec0ve)	
  	
  
Ques0ons?	
  
	
  
Marci	
  Pernica	
  	
  	
  
Marci.Pernica@Ontario.ca	
  	
  
Ministry	
  of	
  Community	
  and	
  Social	
  Services	
  
	
  
Judith	
  Godin	
  	
  
sta/s/cs@jgodin.com	
  
Independent	
  Consultant	
  

Mais conteúdo relacionado

Mais procurados

Usability of Online Instruction
Usability of Online InstructionUsability of Online Instruction
Usability of Online InstructionMichael Wilder
 
Resume_Biostatistician
Resume_BiostatisticianResume_Biostatistician
Resume_BiostatisticianYang Lu
 
Factors Affecting the Sustained Use of Process Models
Factors Affecting the Sustained Use of Process ModelsFactors Affecting the Sustained Use of Process Models
Factors Affecting the Sustained Use of Process ModelsMarlon Dumas
 
Alise2014 30 40 d7
Alise2014 30 40 d7Alise2014 30 40 d7
Alise2014 30 40 d7Sarah Sutton
 
Margot Mc Neill et al 2008
Margot Mc Neill et al 2008Margot Mc Neill et al 2008
Margot Mc Neill et al 2008Diana Quinn
 
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...Online assessment and data analytics - Peter Tan - Institute of Technical Edu...
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...Blackboard APAC
 
Recommending Scientific Papers: Investigating the User Curriculum
Recommending Scientific Papers: Investigating the User CurriculumRecommending Scientific Papers: Investigating the User Curriculum
Recommending Scientific Papers: Investigating the User CurriculumJonathas Magalhães
 
Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...
Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...
Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...scotdigich
 
Redundancies in Data and their E ect on the Evaluation of Recommendation Syst...
Redundancies in Data and their Eect on the Evaluation of Recommendation Syst...Redundancies in Data and their Eect on the Evaluation of Recommendation Syst...
Redundancies in Data and their E ect on the Evaluation of Recommendation Syst...Eirini Ntoutsi
 
Helping teachers understand their learners and their needs better in WebCT
Helping teachers understand their learners and their needs better in WebCTHelping teachers understand their learners and their needs better in WebCT
Helping teachers understand their learners and their needs better in WebCTcies
 
Capstone eLearning Deck
Capstone eLearning DeckCapstone eLearning Deck
Capstone eLearning DeckZacharyCote2
 
Constructive Controversy in CMC Contexts
Constructive Controversy in CMC ContextsConstructive Controversy in CMC Contexts
Constructive Controversy in CMC ContextsAndy Saltarelli
 
Surveying the State of Library Assessment
Surveying the State of Library AssessmentSurveying the State of Library Assessment
Surveying the State of Library Assessmentchtrotter
 
NCompass Live: Conducting Surveys III: Analyzing Data and Reporting Methods
NCompass Live: Conducting Surveys III: Analyzing Data and Reporting Methods NCompass Live: Conducting Surveys III: Analyzing Data and Reporting Methods
NCompass Live: Conducting Surveys III: Analyzing Data and Reporting Methods Nebraska Library Commission
 
Usability Evaluation in Educational Technology
Usability Evaluation in Educational Technology Usability Evaluation in Educational Technology
Usability Evaluation in Educational Technology Alaa Sadik
 
Using GradeMark to improve feedback and involve students in the marking process
Using GradeMark to improve feedback and involve students in the marking processUsing GradeMark to improve feedback and involve students in the marking process
Using GradeMark to improve feedback and involve students in the marking processSara Marsham
 
Frameworking craap how we're correlating the acrl framework to content evalu...
Frameworking craap  how we're correlating the acrl framework to content evalu...Frameworking craap  how we're correlating the acrl framework to content evalu...
Frameworking craap how we're correlating the acrl framework to content evalu...Derek Malone
 
Multiple Response Questions - Allowing for chance in authentic assessments
Multiple Response Questions - Allowing for chance in authentic assessmentsMultiple Response Questions - Allowing for chance in authentic assessments
Multiple Response Questions - Allowing for chance in authentic assessmentsMhairi Mcalpine
 

Mais procurados (19)

Usability of Online Instruction
Usability of Online InstructionUsability of Online Instruction
Usability of Online Instruction
 
Resume_Biostatistician
Resume_BiostatisticianResume_Biostatistician
Resume_Biostatistician
 
Factors Affecting the Sustained Use of Process Models
Factors Affecting the Sustained Use of Process ModelsFactors Affecting the Sustained Use of Process Models
Factors Affecting the Sustained Use of Process Models
 
Alise2014 30 40 d7
Alise2014 30 40 d7Alise2014 30 40 d7
Alise2014 30 40 d7
 
Margot Mc Neill et al 2008
Margot Mc Neill et al 2008Margot Mc Neill et al 2008
Margot Mc Neill et al 2008
 
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...Online assessment and data analytics - Peter Tan - Institute of Technical Edu...
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...
 
Recommending Scientific Papers: Investigating the User Curriculum
Recommending Scientific Papers: Investigating the User CurriculumRecommending Scientific Papers: Investigating the User Curriculum
Recommending Scientific Papers: Investigating the User Curriculum
 
Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...
Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...
Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...
 
Redundancies in Data and their E ect on the Evaluation of Recommendation Syst...
Redundancies in Data and their Eect on the Evaluation of Recommendation Syst...Redundancies in Data and their Eect on the Evaluation of Recommendation Syst...
Redundancies in Data and their E ect on the Evaluation of Recommendation Syst...
 
Online survey
Online surveyOnline survey
Online survey
 
Helping teachers understand their learners and their needs better in WebCT
Helping teachers understand their learners and their needs better in WebCTHelping teachers understand their learners and their needs better in WebCT
Helping teachers understand their learners and their needs better in WebCT
 
Capstone eLearning Deck
Capstone eLearning DeckCapstone eLearning Deck
Capstone eLearning Deck
 
Constructive Controversy in CMC Contexts
Constructive Controversy in CMC ContextsConstructive Controversy in CMC Contexts
Constructive Controversy in CMC Contexts
 
Surveying the State of Library Assessment
Surveying the State of Library AssessmentSurveying the State of Library Assessment
Surveying the State of Library Assessment
 
NCompass Live: Conducting Surveys III: Analyzing Data and Reporting Methods
NCompass Live: Conducting Surveys III: Analyzing Data and Reporting Methods NCompass Live: Conducting Surveys III: Analyzing Data and Reporting Methods
NCompass Live: Conducting Surveys III: Analyzing Data and Reporting Methods
 
Usability Evaluation in Educational Technology
Usability Evaluation in Educational Technology Usability Evaluation in Educational Technology
Usability Evaluation in Educational Technology
 
Using GradeMark to improve feedback and involve students in the marking process
Using GradeMark to improve feedback and involve students in the marking processUsing GradeMark to improve feedback and involve students in the marking process
Using GradeMark to improve feedback and involve students in the marking process
 
Frameworking craap how we're correlating the acrl framework to content evalu...
Frameworking craap  how we're correlating the acrl framework to content evalu...Frameworking craap  how we're correlating the acrl framework to content evalu...
Frameworking craap how we're correlating the acrl framework to content evalu...
 
Multiple Response Questions - Allowing for chance in authentic assessments
Multiple Response Questions - Allowing for chance in authentic assessmentsMultiple Response Questions - Allowing for chance in authentic assessments
Multiple Response Questions - Allowing for chance in authentic assessments
 

Semelhante a Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling

Big Data Analytics - It is here and now!
Big Data Analytics - It is here and now!Big Data Analytics - It is here and now!
Big Data Analytics - It is here and now!Farhan Khan
 
Conformed Dimensions of Data Quality – An Organized Approach to Data Quality ...
Conformed Dimensions of Data Quality – An Organized Approach to Data Quality ...Conformed Dimensions of Data Quality – An Organized Approach to Data Quality ...
Conformed Dimensions of Data Quality – An Organized Approach to Data Quality ...DATAVERSITY
 
Using Data for Continuous Improvement Faculty Development Model - Competency-...
Using Data for Continuous Improvement Faculty Development Model - Competency-...Using Data for Continuous Improvement Faculty Development Model - Competency-...
Using Data for Continuous Improvement Faculty Development Model - Competency-...Becky Lopanec
 
Fasp pd skills & beliefs
Fasp pd skills & beliefsFasp pd skills & beliefs
Fasp pd skills & beliefsyeolhuh
 
EDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINALEDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINALEllen Wagner
 
Wsu Greg Lobdell September 2008 Data And Decision Making
Wsu Greg Lobdell September 2008 Data And Decision MakingWsu Greg Lobdell September 2008 Data And Decision Making
Wsu Greg Lobdell September 2008 Data And Decision MakingWSU Cougars
 
Thorn ODE BHK Talk 2013 09-16 handout
Thorn ODE BHK Talk 2013 09-16 handoutThorn ODE BHK Talk 2013 09-16 handout
Thorn ODE BHK Talk 2013 09-16 handoutChristopher Thorn
 
Ellen Wagner: Putting Data to Work
Ellen Wagner: Putting Data to WorkEllen Wagner: Putting Data to Work
Ellen Wagner: Putting Data to WorkAlexandra M. Pickett
 
National university assessment process
National university assessment processNational university assessment process
National university assessment processAshley Kovacs
 
From Reporting to Insight to Action
From Reporting to Insight to ActionFrom Reporting to Insight to Action
From Reporting to Insight to ActionEllen Wagner
 
Performance Management to Program Evaluation: Creating a Complementary Connec...
Performance Management to Program Evaluation: Creating a Complementary Connec...Performance Management to Program Evaluation: Creating a Complementary Connec...
Performance Management to Program Evaluation: Creating a Complementary Connec...nicholes21
 
Impact Evaluation: Balancing Rigor with Reality
Impact Evaluation: Balancing Rigor with RealityImpact Evaluation: Balancing Rigor with Reality
Impact Evaluation: Balancing Rigor with RealityDonna Smith-Moncrieffe
 
Desire2Learn Analytics Oklahoma RUF
Desire2Learn Analytics Oklahoma RUFDesire2Learn Analytics Oklahoma RUF
Desire2Learn Analytics Oklahoma RUFBarry Dahl
 
Student Activity Hub community Meeting 10-25-2017
Student Activity Hub community Meeting 10-25-2017Student Activity Hub community Meeting 10-25-2017
Student Activity Hub community Meeting 10-25-2017Brett Pollak
 
Tools And Resources For Continuous Improvement Of Technology In Schools
Tools And Resources For Continuous Improvement Of Technology In SchoolsTools And Resources For Continuous Improvement Of Technology In Schools
Tools And Resources For Continuous Improvement Of Technology In Schoolsfridayinstitute
 
OER LEARNING DESIGN GUIDELINES FOR BRAZILIAN K-12 TEACHERS SUPPORTING THE DEV...
OER LEARNING DESIGN GUIDELINES FOR BRAZILIAN K-12 TEACHERS SUPPORTING THE DEV...OER LEARNING DESIGN GUIDELINES FOR BRAZILIAN K-12 TEACHERS SUPPORTING THE DEV...
OER LEARNING DESIGN GUIDELINES FOR BRAZILIAN K-12 TEACHERS SUPPORTING THE DEV...Global OER Graduate Network
 
2009 ECNO Conference - Teaching Learning Critical Pathway
2009 ECNO Conference - Teaching Learning Critical Pathway2009 ECNO Conference - Teaching Learning Critical Pathway
2009 ECNO Conference - Teaching Learning Critical PathwayJohn Brighton
 
RDAP14 Poster: Evaluation of research data services: What things should we ev...
RDAP14 Poster: Evaluation of research data services: What things should we ev...RDAP14 Poster: Evaluation of research data services: What things should we ev...
RDAP14 Poster: Evaluation of research data services: What things should we ev...ASIS&T
 

Semelhante a Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling (20)

Big Data Analytics - It is here and now!
Big Data Analytics - It is here and now!Big Data Analytics - It is here and now!
Big Data Analytics - It is here and now!
 
Conformed Dimensions of Data Quality – An Organized Approach to Data Quality ...
Conformed Dimensions of Data Quality – An Organized Approach to Data Quality ...Conformed Dimensions of Data Quality – An Organized Approach to Data Quality ...
Conformed Dimensions of Data Quality – An Organized Approach to Data Quality ...
 
Using Data for Continuous Improvement Faculty Development Model - Competency-...
Using Data for Continuous Improvement Faculty Development Model - Competency-...Using Data for Continuous Improvement Faculty Development Model - Competency-...
Using Data for Continuous Improvement Faculty Development Model - Competency-...
 
Fasp pd skills & beliefs
Fasp pd skills & beliefsFasp pd skills & beliefs
Fasp pd skills & beliefs
 
EDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINALEDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINAL
 
Wsu Greg Lobdell September 2008 Data And Decision Making
Wsu Greg Lobdell September 2008 Data And Decision MakingWsu Greg Lobdell September 2008 Data And Decision Making
Wsu Greg Lobdell September 2008 Data And Decision Making
 
Thorn ODE BHK Talk 2013 09-16 handout
Thorn ODE BHK Talk 2013 09-16 handoutThorn ODE BHK Talk 2013 09-16 handout
Thorn ODE BHK Talk 2013 09-16 handout
 
Ellen Wagner: Putting Data to Work
Ellen Wagner: Putting Data to WorkEllen Wagner: Putting Data to Work
Ellen Wagner: Putting Data to Work
 
Documents oerc_160913_va_symp_thorn
 Documents oerc_160913_va_symp_thorn Documents oerc_160913_va_symp_thorn
Documents oerc_160913_va_symp_thorn
 
National university assessment process
National university assessment processNational university assessment process
National university assessment process
 
From Reporting to Insight to Action
From Reporting to Insight to ActionFrom Reporting to Insight to Action
From Reporting to Insight to Action
 
Performance Management to Program Evaluation: Creating a Complementary Connec...
Performance Management to Program Evaluation: Creating a Complementary Connec...Performance Management to Program Evaluation: Creating a Complementary Connec...
Performance Management to Program Evaluation: Creating a Complementary Connec...
 
Impact Evaluation: Balancing Rigor with Reality
Impact Evaluation: Balancing Rigor with RealityImpact Evaluation: Balancing Rigor with Reality
Impact Evaluation: Balancing Rigor with Reality
 
1710 track1 bagirov
1710 track1 bagirov1710 track1 bagirov
1710 track1 bagirov
 
Desire2Learn Analytics Oklahoma RUF
Desire2Learn Analytics Oklahoma RUFDesire2Learn Analytics Oklahoma RUF
Desire2Learn Analytics Oklahoma RUF
 
Student Activity Hub community Meeting 10-25-2017
Student Activity Hub community Meeting 10-25-2017Student Activity Hub community Meeting 10-25-2017
Student Activity Hub community Meeting 10-25-2017
 
Tools And Resources For Continuous Improvement Of Technology In Schools
Tools And Resources For Continuous Improvement Of Technology In SchoolsTools And Resources For Continuous Improvement Of Technology In Schools
Tools And Resources For Continuous Improvement Of Technology In Schools
 
OER LEARNING DESIGN GUIDELINES FOR BRAZILIAN K-12 TEACHERS SUPPORTING THE DEV...
OER LEARNING DESIGN GUIDELINES FOR BRAZILIAN K-12 TEACHERS SUPPORTING THE DEV...OER LEARNING DESIGN GUIDELINES FOR BRAZILIAN K-12 TEACHERS SUPPORTING THE DEV...
OER LEARNING DESIGN GUIDELINES FOR BRAZILIAN K-12 TEACHERS SUPPORTING THE DEV...
 
2009 ECNO Conference - Teaching Learning Critical Pathway
2009 ECNO Conference - Teaching Learning Critical Pathway2009 ECNO Conference - Teaching Learning Critical Pathway
2009 ECNO Conference - Teaching Learning Critical Pathway
 
RDAP14 Poster: Evaluation of research data services: What things should we ev...
RDAP14 Poster: Evaluation of research data services: What things should we ev...RDAP14 Poster: Evaluation of research data services: What things should we ev...
RDAP14 Poster: Evaluation of research data services: What things should we ev...
 

Mais de CesToronto

Why aren't Evaluators using Digital Media Analytics?
Why aren't Evaluators using Digital Media Analytics?Why aren't Evaluators using Digital Media Analytics?
Why aren't Evaluators using Digital Media Analytics?CesToronto
 
Evaluation pal program monitoring and evaluation technology
Evaluation pal program monitoring and evaluation technologyEvaluation pal program monitoring and evaluation technology
Evaluation pal program monitoring and evaluation technologyCesToronto
 
Developing Performance Measures Through a Consultative Process
Developing Performance Measures Through a Consultative ProcessDeveloping Performance Measures Through a Consultative Process
Developing Performance Measures Through a Consultative ProcessCesToronto
 
The international Partnership Initiative to promote Evaluation Capacity Devel...
The international Partnership Initiative to promote Evaluation Capacity Devel...The international Partnership Initiative to promote Evaluation Capacity Devel...
The international Partnership Initiative to promote Evaluation Capacity Devel...CesToronto
 
Ces 2013 the role of technology and social media - raising the grade
Ces 2013   the role of technology and social media - raising the gradeCes 2013   the role of technology and social media - raising the grade
Ces 2013 the role of technology and social media - raising the gradeCesToronto
 
Evaluating Libraries' Business Services
Evaluating Libraries' Business Services Evaluating Libraries' Business Services
Evaluating Libraries' Business Services CesToronto
 
Demonstrating Research Impact: Measuring Return on Investment with an Impact ...
Demonstrating Research Impact: Measuring Return on Investment with an Impact ...Demonstrating Research Impact: Measuring Return on Investment with an Impact ...
Demonstrating Research Impact: Measuring Return on Investment with an Impact ...CesToronto
 
Evaluating through a Community of Practice: How 8 Evaluators are Improving Pr...
Evaluating through a Community of Practice: How 8 Evaluators are Improving Pr...Evaluating through a Community of Practice: How 8 Evaluators are Improving Pr...
Evaluating through a Community of Practice: How 8 Evaluators are Improving Pr...CesToronto
 
Ces 2013 negotiating evaluation design
Ces 2013   negotiating evaluation designCes 2013   negotiating evaluation design
Ces 2013 negotiating evaluation designCesToronto
 
Ces conference presentation 2013 final_june 2013
Ces conference presentation 2013 final_june 2013Ces conference presentation 2013 final_june 2013
Ces conference presentation 2013 final_june 2013CesToronto
 
Monitoring and Evaluation of International Development Assistance to the Priv...
Monitoring and Evaluation of International Development Assistance to the Priv...Monitoring and Evaluation of International Development Assistance to the Priv...
Monitoring and Evaluation of International Development Assistance to the Priv...CesToronto
 
Living the Values of Engagement and the Strategic View with Program Evaluatio...
Living the Values of Engagement and the Strategic View with Program Evaluatio...Living the Values of Engagement and the Strategic View with Program Evaluatio...
Living the Values of Engagement and the Strategic View with Program Evaluatio...CesToronto
 
Avoiding Chartjunk
Avoiding Chartjunk Avoiding Chartjunk
Avoiding Chartjunk CesToronto
 
Evaluation for Development: matching evaluation to the right user, the right...
Evaluation for Development:  matching evaluation to the right user, the right...Evaluation for Development:  matching evaluation to the right user, the right...
Evaluation for Development: matching evaluation to the right user, the right...CesToronto
 
Indicators workshop ces 2013
Indicators workshop ces 2013Indicators workshop ces 2013
Indicators workshop ces 2013CesToronto
 
CES Toronto 2013 Engaging Practitioners in Evaluation using the Risk Based Co...
CES Toronto 2013 Engaging Practitioners in Evaluation using the Risk Based Co...CES Toronto 2013 Engaging Practitioners in Evaluation using the Risk Based Co...
CES Toronto 2013 Engaging Practitioners in Evaluation using the Risk Based Co...CesToronto
 
CES 2013 conference - Rethinking the Relationship between Monitoring and Eval...
CES 2013 conference - Rethinking the Relationship between Monitoring and Eval...CES 2013 conference - Rethinking the Relationship between Monitoring and Eval...
CES 2013 conference - Rethinking the Relationship between Monitoring and Eval...CesToronto
 
Closing the Knowledge Gap Between Evaluators and Stakeholders
Closing the Knowledge Gap Between Evaluators and StakeholdersClosing the Knowledge Gap Between Evaluators and Stakeholders
Closing the Knowledge Gap Between Evaluators and StakeholdersCesToronto
 
Ces 2013 - Doing Developmental Evaluation at the System Level
Ces 2013 - Doing Developmental Evaluation at the System LevelCes 2013 - Doing Developmental Evaluation at the System Level
Ces 2013 - Doing Developmental Evaluation at the System LevelCesToronto
 
Ces 2013 towards a cdn definition of evaluation
Ces 2013   towards a cdn definition of evaluationCes 2013   towards a cdn definition of evaluation
Ces 2013 towards a cdn definition of evaluationCesToronto
 

Mais de CesToronto (20)

Why aren't Evaluators using Digital Media Analytics?
Why aren't Evaluators using Digital Media Analytics?Why aren't Evaluators using Digital Media Analytics?
Why aren't Evaluators using Digital Media Analytics?
 
Evaluation pal program monitoring and evaluation technology
Evaluation pal program monitoring and evaluation technologyEvaluation pal program monitoring and evaluation technology
Evaluation pal program monitoring and evaluation technology
 
Developing Performance Measures Through a Consultative Process
Developing Performance Measures Through a Consultative ProcessDeveloping Performance Measures Through a Consultative Process
Developing Performance Measures Through a Consultative Process
 
The international Partnership Initiative to promote Evaluation Capacity Devel...
The international Partnership Initiative to promote Evaluation Capacity Devel...The international Partnership Initiative to promote Evaluation Capacity Devel...
The international Partnership Initiative to promote Evaluation Capacity Devel...
 
Ces 2013 the role of technology and social media - raising the grade
Ces 2013   the role of technology and social media - raising the gradeCes 2013   the role of technology and social media - raising the grade
Ces 2013 the role of technology and social media - raising the grade
 
Evaluating Libraries' Business Services
Evaluating Libraries' Business Services Evaluating Libraries' Business Services
Evaluating Libraries' Business Services
 
Demonstrating Research Impact: Measuring Return on Investment with an Impact ...
Demonstrating Research Impact: Measuring Return on Investment with an Impact ...Demonstrating Research Impact: Measuring Return on Investment with an Impact ...
Demonstrating Research Impact: Measuring Return on Investment with an Impact ...
 
Evaluating through a Community of Practice: How 8 Evaluators are Improving Pr...
Evaluating through a Community of Practice: How 8 Evaluators are Improving Pr...Evaluating through a Community of Practice: How 8 Evaluators are Improving Pr...
Evaluating through a Community of Practice: How 8 Evaluators are Improving Pr...
 
Ces 2013 negotiating evaluation design
Ces 2013   negotiating evaluation designCes 2013   negotiating evaluation design
Ces 2013 negotiating evaluation design
 
Ces conference presentation 2013 final_june 2013
Ces conference presentation 2013 final_june 2013Ces conference presentation 2013 final_june 2013
Ces conference presentation 2013 final_june 2013
 
Monitoring and Evaluation of International Development Assistance to the Priv...
Monitoring and Evaluation of International Development Assistance to the Priv...Monitoring and Evaluation of International Development Assistance to the Priv...
Monitoring and Evaluation of International Development Assistance to the Priv...
 
Living the Values of Engagement and the Strategic View with Program Evaluatio...
Living the Values of Engagement and the Strategic View with Program Evaluatio...Living the Values of Engagement and the Strategic View with Program Evaluatio...
Living the Values of Engagement and the Strategic View with Program Evaluatio...
 
Avoiding Chartjunk
Avoiding Chartjunk Avoiding Chartjunk
Avoiding Chartjunk
 
Evaluation for Development: matching evaluation to the right user, the right...
Evaluation for Development:  matching evaluation to the right user, the right...Evaluation for Development:  matching evaluation to the right user, the right...
Evaluation for Development: matching evaluation to the right user, the right...
 
Indicators workshop ces 2013
Indicators workshop ces 2013Indicators workshop ces 2013
Indicators workshop ces 2013
 
CES Toronto 2013 Engaging Practitioners in Evaluation using the Risk Based Co...
CES Toronto 2013 Engaging Practitioners in Evaluation using the Risk Based Co...CES Toronto 2013 Engaging Practitioners in Evaluation using the Risk Based Co...
CES Toronto 2013 Engaging Practitioners in Evaluation using the Risk Based Co...
 
CES 2013 conference - Rethinking the Relationship between Monitoring and Eval...
CES 2013 conference - Rethinking the Relationship between Monitoring and Eval...CES 2013 conference - Rethinking the Relationship between Monitoring and Eval...
CES 2013 conference - Rethinking the Relationship between Monitoring and Eval...
 
Closing the Knowledge Gap Between Evaluators and Stakeholders
Closing the Knowledge Gap Between Evaluators and StakeholdersClosing the Knowledge Gap Between Evaluators and Stakeholders
Closing the Knowledge Gap Between Evaluators and Stakeholders
 
Ces 2013 - Doing Developmental Evaluation at the System Level
Ces 2013 - Doing Developmental Evaluation at the System LevelCes 2013 - Doing Developmental Evaluation at the System Level
Ces 2013 - Doing Developmental Evaluation at the System Level
 
Ces 2013 towards a cdn definition of evaluation
Ces 2013   towards a cdn definition of evaluationCes 2013   towards a cdn definition of evaluation
Ces 2013 towards a cdn definition of evaluation
 

Último

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 

Último (20)

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 

Improving evaluations and utilization with statistical edge nested data designs and hierarchical linear modeling

  • 1. Improving  Evalua/ons  and  U/liza/on  with   Sta/s/cal  Edge:  Nested  Data  Designs  and   Hierarchical  Linear  Modeling  (HLM)     CES  Conference  -­‐  June  10  2013   Marci  Pernica  -­‐  Ministry  of  Community  and  Social  Services   Judith  Godin  –  J  Godin  Consul7ng  
  • 2. •  The  goal  of  this  presenta0on  is  to  introduce   the  concept  of  HLM  and  explain  how  it  can  be   used  in  program  evalua0on       Introduc0on  
  • 3. What  is  Hierarchical  Linear  Modeling  (HLM)   •  HLM  is  a  sta0s0cal  technique  to  analyze  data   that  is  structured  in  hierarchies  (or  “nested”)   •  To  account  for  the  fact  that  people  that  are  “clustered”  or   “nested”  within  the  same  group  have  more  in  common  than   if  they  were  independent  random  samples       Classroom  1   Classroom   2   Classroom  3   Student  4   Student     2   Student  5   Student  3   Student  8  Student  7  Student  6   Student  9   Student   10   Student   11   Student   12   Nested   data   designs  
  • 4. Student   Class   School   District   Hierarchical  Structure  –  mul0-­‐level     age   I.Q   Measuring   test  scores   (dependent  variable)   Independent   variables  
  • 5. •  HLM  enables  a  more  robust  analy0c  approach   for  nested  data  (than  regression  or  ANOVA)   •  Data  in  evalua0on  are  oZen  nested   •  To  determine  success  condi*ons  for  the   program  –  e.g.  is  the  program  more  suitable   for  certain  sub-­‐popula0ons  or  more  successful   if  delivered  in  a  certain  way   Why  Use  HLM  in  Evalua0on  
  • 6. Program  design  structure   Data  structure   Evalua0on  ques0ons   -­‐  Which  par0cipant  or  site-­‐level  characteris0cs  are  most  influen0al  in   explaining  the  varia0on  in  test  scores  among  the  program   par0cipants?   -­‐  What  program  delivery  characteris0cs  (site  level  prac0ces)  seem  to  be   having  the  most  posi0ve  impact  on  the  par0cipants’  test  scores?   -­‐  Are  some  program  features  more  suited  to  certain  sub-­‐popula0ons  (e.g.   gender,  age  group,  ethnic  or  cultural  group)     Applying  HLM  in  Evalua0on  
  • 7. Par0cipant   1   Site  1   Site  2   Site  3   Par0cipant   4   Par0cipant   2   Par0cipant 5   Par0cipant   3   Par0cipant   8   Par0cipant   7   Par0cipant   6   Par0cipant   9   Par0cipant   10   Par0cipant   11   Par0cipant   12   Example  of  levels  of  a  hierarchical  model   Par0cipants  (level  1)  nested  within  sites  (level  2)  
  • 8. Assessing  test  scores  by  age  from  site  to  site   Test  Score   Age   Four  different   program  sites   Although  the  test  scores  differ  from  site  to  site,  the   rela0on  between  age  and  test  score  is  the  same  at   different  sites  
  • 9. Student  1   Student  2   Student  3   Baseline   Month  7   Month  1   Month  12   Month  6   Month  9  Month  8  Month  6   Baseline   Month  2   Month  4   Month  6   Assessing  change  over  0me   Assessments  across  0me  (level  1)  are  nested  within  individuals   (level  2)  (i.e.  repeated  measures  design)  
  • 10. Assessing  improved  performance  over  0me   Test  Score   Time   Four  different   study  par0cipants   Although  some  individuals  have  higher  test  scores   to  start  with,  the  rate  of  change  (improved   performance)  is  comparable  among  the   par0cipants  
  • 11. Tradi0onal  Methods   Test  Score   Age   Rela0on  between  age  and   test  score  es0mated  once  for   all  sites  together  
  • 12. Advantages  of  HLM   Test  Score   Age   Four  different   program  sites   Here,  the  rela0on  between  age  and  test  score  varies   across  sites.   Are  there  any  site  level  variables  associated  with  the   strength  of  this  rela0on?  
  • 13. Design  Considera0ons  for  Using  HLM   •  Sample  size   – Par0cipant  level   – Site  level   – Repeated  measures   •  Missing  Data   – Can  be  easy  or  difficult  to  deal  with   •  Number  of  variables   – Comprehensive  coverage   – Parsimony    
  • 14. Final  Thoughts     •  Applying  HLM  in  evalua0ons  with  nested  data   enables  more  robust  results  and  conclusions   •  U0liza0on-­‐focused     – Iden0fy  evidence-­‐based  success  factors  or   condi0ons  for  improving  the  program  delivery   model,  to  ul0mately  achieve  beger  program   effec0veness     – Promo0ng  the  value  in  evalua0on  (gathering  the   evidence  to  determine  the  ‘success  factors’  for   the  interven0on  to  be  effec0ve)    
  • 15. Ques0ons?     Marci  Pernica       Marci.Pernica@Ontario.ca     Ministry  of  Community  and  Social  Services     Judith  Godin     sta/s/cs@jgodin.com   Independent  Consultant