O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

TCI in general pracice - reliability (2006)

Use of the TCI questionnaire in General Practice: response rates and reliability

  • Entre para ver os comentários

  • Seja a primeira pessoa a gostar disto

TCI in general pracice - reliability (2006)

  1. 1. Use of the TCI questionnaire in General Practice: response rates and reliability Evan Kontopantelis Stephen Campbell David Reeves NPCRDC
  2. 2. Team Climate Inventory • 65 item measure with six subscales that attempts to quantify the working climate within a practice (Anderson & West, 1994) • All items on a scale of 1 to 5 • The six subscales (factors) are: Participation Task style Support for innovation Reviewing processes Objectives Working
  3. 3. Data collection • The questionnaire was distributed to all clinical, nursing and administration staff working in a sample of 60 practices in 1998 and 42 of the same practices in 2003 • Response rates varied greatly by practice 1998 2003 number of practices 60 42 average respondents per pract 9.5 12.2 average resp rate per pract 63.1% 65.1%
  4. 4. The question • What level of response is required to obtain a reliable/accurate TCI subscore for a practice?
  5. 5. Data structure • Three levels: items, respondents & practices • But which is the exact form? – When each respondent in a practice evaluates a different set of items (e.g. “I generally prefer to work as part of a team”)  I:R:P – When each respondent in a practice evaluates the same set of items (i.e. “My team has a lot of team spirit”)  (I×R):P • Unfortunately the questionnaire is a mixture of both
  6. 6. Aggregate-level variables and reliability • Methods are based on concepts from the generalizability theory • The universe score is the score that an object of measurement – e.g. a practice – would receive on a characteristic – e.g. participation – if its score was based on the mean of all relevant predefined conditions of measurement – e.g. all possible respondents and questions (O’Brien 1990)
  7. 7. Defining reliability • Reliability is defined as the ratio of the universe score’s variance to the expected observed score variance: 2 2 True Measure p    • It is an indicator of how different the observed score would have been, if another random set of respondents and/or questions had been selected
  8. 8. Variance components graph variability symbol solid circle expected observed score outer ring True score (practices) grey ring Error (respondents) centre circle Error (random error + items) 2 p 2 ,r pr 2 , , , ,i pi ri pri e Shavelson & Webb 1991
  9. 9. Estimating Reliability • For practice j, with nj respondents and k items and according to the I:R:P design: (Marsden et al. 2006) • Variance components need to be calculated 2 2 2 , , , , ,2         p j r pr i pi ri pri e p j jn n k 2 2 2 , , , , ,2 ˆ ˆ ˆ ˆ ˆ         p j r pr i pi ri pri e p j jn n k
  10. 10. Accuracy, defined • It is the likely amount of error on the observed score compared to the true score • Using the central limit theorem we estimate a 95% CI for the TCI score and the subscores: • We defined a score as accurate if the 95% CI for it was: • That is, 0.5 points on the scale of 1 to 5 2 , , , ,2 , 95% ˆ ˆ ˆ 1.96       i pi ri pri e r pr j KCI n ˆ ˆ[ -0.5, +0.5]μ μ
  11. 11. Model & estimation parameters • Only the 3-level random-effect model was described but more were estimated: – Mixed-effect models in which items was treated as a fixed factor – 2-level models that use the aggregate score of the items (R:P) • Variances are estimated within STATA, using ANOVA and MLGLM (StataCorp, 2005) • The GLLAMM command is used to estimate the parameters of the ml linear models we employed (Rabe-Hesketh et al. 2005)
  12. 12. Model & estimation parameters
  13. 13. Results – Review of processes 03 model Variances 2l_anova_reflex 0.045 0.336 0.622 17.3 5.2 2l_anovau_reflex 0.045 0.338 0.619 17.5 5.2 2l_gllamm_reflex 0.046 0.335 0.626 17.0 5.1 3l_f_anova_reflex 0.045 0.259 0.594 0.620 17.4 5.1 3l_f_anovau_reflex 0.044 0.268 0.546 0.617 17.6 5.2 3l_f_gllamm_reflex 0.046 0.253 0.653 0.626 17.0 5.1 3l_r_anova_reflex 0.045 0.247 0.691 0.622 17.3 5.1 3l_r_anovau_reflex 0.045 0.251 0.682 0.618 17.6 5.2 3l_r_gllamm_reflex 0.046 0.248 0.691 0.625 17.0 5.1 ,, 2 , , ˆi pi ri ri ep ˆavp  jn 0.5 95%  n2 ˆp 2 , , ˆr pr e 2 , ˆr pr
  14. 14. Results – ANOVA unpooled model 1998 2003 2l_anovau_TCI 0.679 10.7 3.3 0.764 8.8 3.1 3l_r_anova_obj 0.464 26.2 7.3 0.565 21.9 4.9 3l_r_anova_part 0.644 12.5 4.4 0.814 6.5 4.5 3l_r_anova_reflex 0.576 16.7 6.3 0.618 17.6 5.2 3l_r_anova_supinv 0.567 17.4 4.9 0.791 7.5 4.0 3l_r_anova_task 0.463 26.3 8.6 0.626 17.0 6.7 3l_r_anova_work 0.616 14.2 2.7 0.650 15.3 3.4 ˆavp  jn 0.5 95%  n ˆavp  jn 0.5 95%  n
  15. 15. Why are accuracy and reliability scores so different? • Reliability coefficients are affected by the low variation in practice scores • Practice mean scores did not vary by more than 2 points on the scale of 1 to 5 • TCI may not be particularly good at detecting practice differences in climate 1998 2003 μ min max μ min max part 3.6 2.7 4.5 3.7 2.7 4.6 supinv 3.4 2.5 4.3 3.5 2.6 4.3 reflex 3.4 2.2 4.1 3.4 2.9 4.0 work 3.6 2.8 4.4 3.7 3.1 4.2 obj 3.7 2.7 4.4 3.7 3.0 4.4 task 3.4 2.6 4.4 3.5 2.7 4.1 TCI 3.5 2.8 4.3 3.6 2.9 4.2 Descriptives, practice mean scores
  16. 16. Summary • TCI is a measure that attempts to quantify the working climate within a practice • Assuming that the I:R:P structure best describes our data we calculate: – the variances in the design – reliability & accuracy measures • Small between practice variances affect the reliability score but don’t affect the accuracy
  17. 17. Future work • Use of a finite population correction for practices • Examine the (I×R):P structure and compare results to the I:R:P one
  18. 18. References • Anderson, N. & West, M. A. 1994, Team climate inventory : manual and user's guide Windsor : ASE. • O'Brien, R. M. 1990, "Estimating the reliability of aggregate-level variables based on individual-level characteristics", Sociological Methods and Research; 18 (May 90) p.473-504 • Shavelson, R. J. & Webb, N. M. 1991, Generalizability theory : a primer Newbury Park ; London : Sage Publications. • Marsden, P. V., Landon, B. E., Wilson, I. B., McInnes, K., Hirschhorn, L. R., Ding, L., & Cleary, P. D. 2006, "The reliability of survey assessments of characteristics of medical clinics", Health Serv.Res., vol. 41, no. 1, p.265-283 • StataCorp 2005, Stata Statistical Software: release 9.2 College Station, TX. • Rabe-Hesketh S., Skrondal A., & Pickles A. 2004, GLLAMM Manual U.C. Berkeley.

×