HMCS Max Bernays Pre-Deployment Brief (May 2024).pptx
Applied Psych Test Design Part F: Psychometric/technical statistical analysis: Internal
1. The Art and Science of Test Development—Part F Psychometric/technical statistical analysis: Internal The basic structure and content of this presentation is grounded extensively on the test development procedures developed by Dr. Richard Woodcock Kevin S. McGrew, PhD. Educational Psychologist Research Director Woodcock-Muñoz Foundation
2. “ In god we trust….all others must show data” (unknown source) Test authors and publishers have standards-based responsibility to provide supporting psychometric technical information re: tests and battery Typically in the form of a series of technical chapters in manual or a separate technical manual
3.
4. Calculate summary statistics (n, means, SDs, SEM) and reliabilities for all tests and clusters by technical age groups etc… etc…
6. Special reliability analyses for all tests More complex repeated measures reliability analysis (McArdle and Woodcock, 1989—see WJ-R Technical Manual)
11. Structural/internal validit y Confirmatory factor analysis by major age groups ( exploratory factor analysis if not theory-driven test blueprint) .67 .53 .40 .42 .43
12.
13.
14. The WJ III factor structure model provided the best fit to the data when compared to six alternative models The conclusion was the same across 5 age-differentiated samples Structural/internal validity: Confirmatory factor analysis model comparisons by major age groups
15. Internal validity evidence example: g-loadings for differentially weighted General Intellectual Ability cluster
19. = White Non-White Structural/internal validity Evaluating structural invariance with Multiple Group CFA
20. = Male Female Structural/internal validity Evaluating structural invariance with Multiple Group CFA
21. = Hispanic Non-Hispanic Structural/internal validity Evaluating structural invariance with Multiple Group CFA
22.
23.
24.
25.
26.
27. End of Part F Additional steps in test development process will be presented in subsequent modules as they are developed
Editor's Notes
d
NOTE: Reliability statistics calculated for all WJ III tests across their range of intended use. Included all norming subjects tested at each technical age level. (25 levels - by year 2-19, by 10 yr.intervals 20-79, 80+) Split-half coefficients corrected for length using Spearman-Brown correction Rasch analysis procedures used for calculating reliability for 8 multi-point tests (4 in COG, 4 in ACH) and 8 speeded (5 in COG, 3 in ACH) Test-retest study also conducted for 8 speeded tests Interval set at 1 day to minimize changes in test scores due to changes in the subjects’ states or traits. These reliabilities are lower than those for same tests obtained using Rasch analysis procedures. Test-retest is lower bound for speeded tests - other reliabilities are upper bound. TESTS: 38 of 42 are .80 or higher (15 are .90 or higher) (4 are in .70s) CLUSTERS: 36 of 42 are .90 or higher (6 are in .80s) Consult: Chapter 3 of Technical Manual for more information. Summary information in ASB 2.