The document discusses the process of developing tests and items to measure specific cognitive abilities based on the Cattell-Horn-Carroll (CHC) theory of cognitive abilities. It focuses on developing tests and items for the visualization (Vz) ability. Key steps include defining the ability based on CHC theory, developing prototype items, trying out items on small samples, revising items based on empirical data, administering a calibration version to a large sample to norm the scale, analyzing data using item response theory, and developing a final normed test form. The goal is to develop valid, reliable measures of cognitive abilities grounded in psychological theory.

1. The Art and Science of Test Development—Part B Test and item development The basic structure and content of this presentation is grounded extensively on the test development procedures developed by Dr. Richard Woodcock Kevin S. McGrew, PhD. Educational Psychologist Research Director Woodcock-Muñoz Foundation

2. Theoretical Domain = Cattell-Horn-Carroll (CHC) theory of cognitive abilities – Gv domain & 3 selected narrow Gv abilities Gv Substantive Stage of Test Development: Develop Test Design and Specification Blueprint – Item Scale Development Measurement or empirical domain What types of tasks, items, format, materials, examiner responses, etc.? How do we minimize construct irrelevant variance and maximize construct relevant variance ? There is a universe of potential Vz item tasks . Which type/format should be selected? ?

21. Vz test design decision: Unspeeded 3-D Block Rotation type task

23. Theoretical Domain = Cattell-Horn-Carroll (CHC) theory of cognitive abilities – Gv domain & 3 selected narrow Gv abilities Gv Substantive Stage of Test Development: Develop Test Design and Specification Blueprint – Item Scale Development Measurement or empirical domain Develop pool of items , from easy to difficult, that will then be ordered from low to high on the underlying Block Rotation measurement scale (analogy = developing a yardstick or ruler) Low ability/easy items High ability/difficult items

24. This is the “try out” phase of the new test idea Pay extra attention to the development of very “easy” and very “difficult” items Be prepared for the possibility that your initial item type/format may not be the one you end up with. The empirical may data send you in a different direction, or down a slightly modified path. Typically these initial prototype try out items are not of publication quality Often the hardest part of item development is the development of the examiners verbal instructions – need to be succinct, easy to understand, not complicated, etc.

25. This is one of the reasons for try out phase. To catch possible problems with the items.

31. Theoretical Domain - CHC Cylinders = broad CHC abilities Circles = narrow CHC abilities The process just described occurs for all the tests you are developing g Gf Gv Glr Gs Gc Gsm Ga Measurement or empirical domain

32. End of Part B. Additional steps in test development process will be presented in subsequent modules as they are developed