1. The Shared Data Experience How We Learn to Love Our Data! Akron Ready Steps Assessment Presentation AKRON, OHIP 30 November 2009 Karen Burstein, PhD, MPH Project Evaluator

2. SWI 2 ALL SUCCESSFUL PROJECTS BEGIN AND END WITH THOUGHTFUL, ACCURATE ASSESSMENT.

3. Data Drives Our Lives SWI 3

4. Teachers as Scientists SWI 4

5. Rate Yourself- What data (or not) prompts you to choose a curriculum? SWI 5

6. Evaluation as a Key Component SWI 6

7. Let Your Application be Your Guide SWI 7

8. Caveat 1 SWI 8

9. Caveat 2 SWI 9

10. SWI 10

11. So, Why Assess Children? SWI 11

12. SWI 12

13. So, Why Assess Teachers? SWI 13

14. SWI 14

15. Assessment drives the planning process; which drives our teaching & instructional activities; which are assessed for efficacy based on children’s performance. AND THE CYCLE CONTINUES SWI 15

16. Implications for Assessment SWI 16

17. SWI 17

18. Reliability SWI 18

19. Validity SWI 19

20. Bias SWI 20

21. Planning Annual ERF Assessments- SWI 21

22. Set Up a System SWI 22

23. Setup the Assessment Setting SWI 23 A table for each assessment/ examiner A place for children in between assessments Limit visual and auditory distraction Make sure teachers have comfortable chairs Make sure that children’s feet are flat on the floor

25. Prepare a Dataset SWI 25

26. Moving Children Through Assessments Use the dataset to create name badges A method by which to track the assessments given A daily check sheet of assessments given SWI 26

27. ARS Assessments SWI 27

28. Kindergarten Screener SWI 28

29. Kindergarten Screening Tool A screening tool is a short assessment of the presence of behaviors that is a reliable and valid indicator of a skill. A screening tool should be used to decide whether or not to take specific actions such as seeking referrals, further testing, and instructional activities. A test is a more rigorous and complex assessment of knowledge and skill with more extensive reliability and validity properties. SWI 29

30. Why Screen? Research shows that learning to read is a process that begins long before children enter kindergarten. During the pre-kindergarten years, children develop the early literacy skills that help them to learn how to read during the first few years of elementary school. Early screening and intervention are the keys to overcoming reading difficulties and avoiding the problems that go along with them. Early literacy screening: Helps parents and early childhood professionals understand each child's progress towards developing literacy concepts and acquiring skills. Helps to identify developmentally appropriate experiences and teaching that can be used to support early literacy learning. Alerts early childhood professionals and parents to seek additional advice if a child does not appear to be making appropriate progress. SWI 30

31. GRTR Screener Use this 20-question research-based screening tool with your 4-year-olds. The score will show if a child's pre-reading skills are weak, strong, or somewhere in between. And activities and resources to improve those skills will be provided. This tool is designed to screen a child twice during the year before kindergarten. Use the tool first in the fall one year before the child enters kindergarten, and again the next fall before kindergarten begins, to measure the child's progress. Don't use the tool more than three times in a year. It's not designed to measure small changes, and children develop new skills gradually. SWI 31

33. SWI 33

34. Peabody Picture Vocabulary Test-IV Measure of Receptive Language SWI 34

35. Peabody Picture Vocabulary Test, Fourth EditionBy Lloyd M. Dunn, PhD and Douglas M. Dunn, PhD

36. SWI 36

38. SWI 38

39. TRAINING ITEMS Form B ages 4+

41. Assessment of Language and Literacy Measure of Expressive Language, Phonological Awareness, Alphabet Knowledge, Concepts of Print SWI 41

42. SWI 42

43. The Assessment of Literacy and Language (ALL™) aids in early detection of language disorders that could lead to reading difficulties. ALL assesses spoken language and written language skills, including: Listening comprehension Language comprehension Semantics Syntax Phonological awareness Alphabetic principles/phonics Concepts about print With ALL, you can identify language disorders, language and emergent literacy deficits, emergent literacy deficits, and weak language and emergent literacy. View case studies to see how ALL can impact a child you know. SWI 43

44. Details the nature and severity of the problem Aids in making recommendations for intervention Provides information about where a child should be in his or her reading and language skills based on grade level. Strategies for improving a child’s language skills in each of the areas assessed are included in the Examiner’s manual.The Parent Questionnaire provides additional information about a child’s language and literacy history and current skills. Norm-referenced scores Criterion-referenced scores SWI 44

46. GET It, Got It, Go! Measure of Progress Monitoring SWI 46

47. GGG (IGDI) Three subtests Rapid Object Naming (1 minute) Rhyming (2 minutes) Alliteration (2 minutes) SWI 47

48. Standardized Administration DOES Matter (These assessments are formally called Individual Growth and Development Indicators, or IGDIs.) Standardization of administration procedures, which includes: timing, appropriate prompting, administering sample items—is critical so that you get accurate results. By standardizing, we mean that IGDIs are administered in exactly the same way, for each child, across different assessment periods. This assures that IGDIs are administered accurately and reliably. Standardized administrations will minimize any systematic or unnecessary error. By standardizing, we are increasing the accuracy or reliability of our scores from one administrator to the next, from one child to the next, and for an individual child's scores from time a to b to c to d. SWI 48

49. Each IGDI comes with a set of administration instructions. On the administration cards, bold print indicates exactly what the examiner should say. SWI 49

50. Always Begin with Sample Items If the child meets the criteria indicated in the sample items, move to administration. Discontinue criteria are provided for the picture naming, rhyming, and alliteration measures. These criteria provide an indication of whether a child a). understands the task at hand and b). has sufficient skills for continuing with the administration. For example, the discontinue criteria for picture naming is as follows: The child is administered four sample cards (baby, bear, apple, cat). The administrator may stop administration if the child does not name all four of these pictures correctly. SWI 50

51. Timing Each IGDI should be administered using a stopwatch according to the exact time specified. SWI 51

52. Prompting Be Careful Not To Provide Corrected Feedback During Administration During IGDI "Administration" (not referring to sample card administrations) it is important not to prompt or provide feedback for correct or incorrect responses. Reinforcing statements such as "great," "good job," etc. are encouraged. However, feedback should not correspond to correct or incorrect responses made during administration (e.g., "that was right, " "that was wrong, etc)." SWI 52

53. What Happens When Administration Is Not Standardized "Melissa's" teacher administered the rhyming IGDI to Melissa on Oct. 14th and got a score of 7. She then administered rhyming again on Nov. 14th and got a score of 14. It appears that Melissa grew considerably in rhyming skill during this month. However, Melissa's teacher indicated that she didn't use a stopwatch. Instead, she used a clock with no second-hand during both administrations. She simply estimated 2-minutes time. Here, we have a situation where we can not be sure whether Melissa actually grew in rhyming skill, since it may be that there was simply more time allotted to her during her second administration. Furthermore, lets say Melissa's teacher administered rhyming to all her students without timing exactly 2 minutes. In this case, we can not make accurate comparisons across students since maybe some student may have done better than others, simply because they were allotted more time. SWI 53

54. GGG (IGDI) Picture Naming (Ages 3-5) The format of this indicator presents a child with images of objects commonly found in preschoolers' natural environments (i.e., home, classroom, community), one at a time, asking a child to name the pictures as fast as possible. Categories of objects used in this format include animals, food, people, household objects, games and sports materials, vehicles, tools, and clothing. Each photograph and line drawing is printed on an 8" x 5" index card. After providing a set of sample items, the examiner asks the child to look at each card and name it as quickly as possible. After exactly one minute, the examiner stops the activity and counts the total number of pictures named correctly. SWI 54

55. Alliteration (Ages 3-5) We identified a set of words commonly known by preschoolers and obtained images of these words. Stimulus cards include one image at the top of each card (e.g., rain) and a set of three images in a row at the bottom of each card (e.g., house, rake, pig), one of which starts with the same sound as the target picture. After providing a set of sample items, the examiner asks the child to look at each card and point to one of the three pictures at the bottom of the card with the same initial sound as the fourth, target picture. The task continues for a total of two minutes. The score generated from this format is the number of pictures the child correctly identifies within two minutes. SWI 55

56. Rhyming (Ages 3-5) Stimulus cards include a target photo or line drawing at the top of each card (e.g., bees) and a set of three photos/drawings in a row at the bottom of each card (e.g., pants, gate, cheese), one of which rhymes with the target picture. After providing a set of sample items, the examiner asks the child to look at each card and point to one of the three pictures at the bottom of the card that sounds the same as (or rhymes with) the fourth, target picture. The task continues for a total of two minutes. The score generated from this format is the number of pictures the child correctly identifies within two minutes. SWI 56

57. Useful Data SWI 57

58. Bar Graph of Performance by Class SWI 58 Year 1 Baseline PPVT Standard Scores by Teacher by Kinder Status KINDER BOUND 100 GOING TO KINDERGARTEN RETURNING TO PRESCHOOL 80 60 96.8 95 91.9 89.3 87 85 84 82.4 40 80.5 79.6 78.7 77.2 76 20 0 6 5 7 2 8 3 1 4 Class

59. Line Graph of Class Performance SWI 59

60. Sample Weekly Data SWI 60

61. GRTR Pretest 20.00 GRTR 2nd Test 15.00 10.00 5.00 0.00 Class 1 Class 2 Class 3 Site . Get It Got It Go (N = 46, participating in both pretest and 2nd test) SWI 61

62. Teacher Report SWI 62

63. Plan, Plan, Plan SWI 63

64. Review, Review, Review SWI 64

65. Celebrate Your Success (based on your data) For more information please contact: Karen Burstein, PhD, Evaluator 480-222-8800 x 1 k.burstein@swifamilies.org SWI 65