"Unlocking the black box: what's happening in 'more effective' classrooms in India?" - by Rhiannon Moore at BAICE 2018 -

1. @yloxford
@younglivesindia
Unlocking the black box: what’s happening in
‘more effective’ classrooms in India?
Rhiannon Moore & Ana Grijalva
(Young Lives, University of Oxford)
Caine Rolleston
(UCL Institute of Education)
@rhi_moore
@agrijalvaes
@caine_rolleston

2. INTRODUCTION
• This presentation uses a unique dataset
linking large-scale school effectiveness
data with data from a smaller classroom
observation sub-study:
1. Young Lives’ 2016-17 secondary
school effectiveness survey in India
2. Young Lives’ 2017-18 classroom
observation sub-study
• Young Lives is a longitudinal study of
child poverty which has been conducted
across 4 countries since 2002
• In this presentation I will focus on Young
Lives’ education data from India

3. WHY CLASSROOM OBSERVATION?
• Young Lives’ school effectiveness survey allows us to
identify certain teachers as more or less ‘effective’.
• But we don’t know what is happening in those classrooms to
explain this.
• Classroom observation sub-study designed with the aim to
unlock the ‘black box’ of the classroom and address
policy relevant questions such as how teacher classroom
practice relates to student learning.
• Also wanted to verify findings from school effectiveness
study which identify certain teachers as more or less
effective (using value added).
• Are there patterns between value added and what classroom
observation captures for these teachers in the classroom?

4. Measuring teacher effectiveness & classroom practice

5. • School effectiveness survey design collected
test score data at 2 points in time.
• From this, we can calculate ‘value-added’:
a measure of student progress over a
defined period of time
• Value-added measures if students in a
particular school make more or less progress
than others with a similar starting score
• It is designed to compare ‘like for like’
• This provides us with a summary measure of
school or teacher quality, centred on zero:
• Teachers with VA above 0 have students who
make more progress than expected
• Teachers with VA below 0 have students who
make less progress than expected
WHAT DO WE MEAN BY ‘EFFECTIVENESS’?

6. CLASSROOM OBSERVATION USING CLASS-S
• Classroom Observation was
undertaken using the Classroom
Assessment Scoring System-Secondary
(CLASS-S) method
• Designed by Robert Pianta at University of
Virginia
• CLASS aims to measure teacher-
student interactions identified as
being important to how students
learn
• Has also been used in many countries in
North and South America, Europe, Africa,
and East Asia - but never in India before
this study

7. HOW IS CLASS SCORED?
• CLASS is based on 3 domains of teacher-student
interactions: Emotional Support; Classroom Organisation;
and Instructional Support.
• These are further split into 11 dimensions, e.g. Positive Climate,
Instructional Dialogue, Behaviour Management etc
• Teachers are observed for 2 lessons, and scored for each
dimension (out of 7). Their score for each dimension is
averaged across these lessons.
• We have standardised scores for each dimension so that
each has a mean of zero and SD of 1
• These standardised scores have then been added to create
a total CLASS score across the 11 dimensions

8. CLASSROOM OBSERVATION STUDY DESIGN
• 45 Grade 9 maths/English teachers in
23 schools
• In 4 districts (2 in AP, 2 in Telangana)
• Teachers purposively sampled using
data from 2016-17 school effectiveness
survey, based on their VA score
• High VA teachers
• Average VA teachers
• Low VA teachers
• Sample also included teachers from
different school management types and
urban/rural areas

9. Findings: Is teacher value-added related to CLASS score?

10. THERE APPEARS TO BE A POSITIVE TREND
-20-10
0
1020 VA significantly below average Average VA VA significantly above average
Maths teacher English teacher
Graphs by uncon_sig

11. -20-10
0
1020
-50 0 50
Value-added (unconditional)
Low VA Average VA
High VA Another reason
PARTICULARLY IF WE CONSIDER SAMPLING
Most of those with below average
CLASS scores were sampled
because they have low VA or for
another reason (e.g. replacement
sample)
While many of those with
higher CLASS scores were
sampled because they have
high or average value-added

12. APPEARS TO VALIDATE VALUE-ADDED
• This type of purposive sample aimed at verifying the
estimates of value-added obtained from the school
effectiveness survey
• Could have attempted something similar with a large-scale
classroom observation study, but more expensive / less efficient
• Initial descriptive analysis appears to verify the
relationship between what teachers do in the classroom
and their value-added estimates
• Positive, especially as this is the first time CLASS has
been used in India

13. Findings: Who are the teachers achieving higher scores?

14. SCHOOL TYPE IS IMPORTANT
-20-10
0
1020
-50 0 50
Value-added (unconditional)
Private Aided Private Unaided
State Govt Tribal Welfare

15. -20-10
0
1020
-50 0 50
Value-added (unconditional)
Private Aided Private Unaided
State Govt Tribal Welfare
Fitted values - PU Fitted values - SG
AND IMPACTS CLASS-VA ASSOCIATION
Stronger positive
relationship for
State Govt schools
Negative association
for Private Unaided
schools

16. SCHOOL DISTRICT IS ALSO RELEVANT
-20-10
0
1020
-50 0 50
Value-added (unconditional)
Srikakulam Anantapur
Karimnagar Mahbubnagar
Most teachers in this half of the graph are in Srikakulam or
Karimnagar…
…while most in this half are in Anantapur and Mahbubnagar

17. BUT URBAN/RURAL APPEARS LESS SO
-20-10
0
1020
-50 0 50
Value-added (unconditional)
Rural Urban

18. WHAT DOES THIS TELL US?
• There appears to be a stronger, more positive
relationship between CLASS and teacher value-added in
State Govt schools than in Private Unaided schools
• Important to consider this in further analyses
• The district of the school appears to be relevant,
although need to remember sampling limitations
• Urban/rural appears less relevant
• These descriptives reveal importance of situating
analysis of teacher classroom practice and teacher
effectiveness – need to understand what teachers do in
the context of where they are teaching

19. Discussion and implications

20. DISCUSSION AND IMPLICATIONS
• Findings suggest that CLASS is predictive of teacher
effectiveness in the Indian context
• Verifies the teacher effectiveness estimates from the school
survey
• Suggests could be a role for continued use of CLASS in India
• This relationship is particularly strong and positive for
govt teachers, revealing that school management type
interacts with CLASS score to different effect
• Suggests that there is no one way to improve student learning
outcomes – Private Unaided schools doing something different?
• Emphasises importance of teachers within Govt school system.
What they do in the classroom is strongly associated with student
learning outcomes – fewer other inputs to support learning than in
other school types?

21. CLASSROOM OBSERVATION VIDEOS
In addition to live observations, we also filmed selected teachers in
the classroom, and have coded these videos to produce a set of clips
for each CLASS dimension in the Indian context. These can be found
here:
http://www.younglives.org.uk/content/classroom-video-clips

22. Thank you!
Any questions or
comments?