Different Measures of Validity and Reliability
| Type . | Definition . | Statistical Measures and Their General Interpretation of Minimum Levels of Acceptability . | |
|---|---|---|---|
| Reliability | |||
| Internal consistency | Items that are measuring the same construct should have correlated results | Omega | >0.7 is acceptable, >0.8 is excellent, >0.9 suggests item redundancya |
| Cronbach α | |||
| Split-half reliability | |||
| Test-tetest | Survey items have temporal stability | Correlation coefficients dependent upon type of datab: | |
| Pearson r for continuous variables | |0.3|-|0.49| is weak correlation | ||
| |0.5|- |0.69| is moderate | |||
| |0.7| - |0.89| is strong | |||
| Spearman ρ for ordinal variables | |0.9| - |1| is very strong | ||
| Tetrachoric correlation coefficient for dichotomous variables | Note: Positive values indicate positive correlation, whereas negative values indicate negative correlation | ||
| Validity | |||
| Construct | |||
| Exploratory factor analysis | Statistical technique to reduce data into theoretical “factors” of the construct of interest | Number of factors to be determined byc,d: | |
| Kaiser-Guttman rule | Number of factors is equal to the number of factors with eigenvalues >1.0 | ||
| Scree plot | “Elbow” of the graph where eigenvalues level off is the point of significant factors | ||
| Parallel analysis | Retain the number of factors where eigenvalues of the sample data are higher than those from simulated data | ||
| Confirmatory factor analysis | Statistical technique to confirm if data fit hypothesized factor structure | Model fit usinge: | |
| Root mean squared error of approximation | <0.06 excellent | ||
| Comparative fit index | >0.90 acceptable, >0.95 excellent | ||
| Tucker-Lewis index | >0.90 acceptable, >0.95 excellent | ||
| Goodness of fit indices | >0.95 acceptable | ||
| Standardized root mean squared residual | <0.8 acceptable fit | ||
| Convergent | Ability to strongly positively or negatively correlate with other instruments that measure similar constructs | Correlation coefficients dependent upon type of datab: | |
| Pearson r for continuous variables | |0.3|-|0.49| is weak correlation | ||
| |0.5|- |0.69| is moderate | |||
| Divergent | Ability to not correlate with other instruments that measure not-similar constructs | Spearman ρ for ordinal variables | |0.7| - |0.89| is strong |
| |0.9| - |1| is very strong | |||
| Tetrachoric correlation coefficient for dichotomous variables | Note: Positive values indicate positive correlation, whereas negative values indicate negative correlation | ||
| Criterion | |||
| Concurrent | Ability of an instrument to predict current outcomes | Correlation coefficients dependent upon type of datab: | |
| Pearson r for continuous variables | |0.3|-|0.49| is weak correlation | ||
| |0.5|- |0.69| is moderate | |||
| Predictive | Ability of an instrument to predict future outcomes | Spearman ρ for ordinal variables | |0.7| - |0.89| is strong |
| |0.9| - |1| is very strong | |||
| Tetrachoric correlation coefficient for dichotomous variables | Note: Positive values indicate positive correlation, whereas negative values indicate negative correlation | ||
| Known group and divergent | Comparing known groups on survey outcomes to detect hypothesized differences | t tests, analyses of variance, regression models, etc. | |
| Type . | Definition . | Statistical Measures and Their General Interpretation of Minimum Levels of Acceptability . | |
|---|---|---|---|
| Reliability | |||
| Internal consistency | Items that are measuring the same construct should have correlated results | Omega | >0.7 is acceptable, >0.8 is excellent, >0.9 suggests item redundancya |
| Cronbach α | |||
| Split-half reliability | |||
| Test-tetest | Survey items have temporal stability | Correlation coefficients dependent upon type of datab: | |
| Pearson r for continuous variables | |0.3|-|0.49| is weak correlation | ||
| |0.5|- |0.69| is moderate | |||
| |0.7| - |0.89| is strong | |||
| Spearman ρ for ordinal variables | |0.9| - |1| is very strong | ||
| Tetrachoric correlation coefficient for dichotomous variables | Note: Positive values indicate positive correlation, whereas negative values indicate negative correlation | ||
| Validity | |||
| Construct | |||
| Exploratory factor analysis | Statistical technique to reduce data into theoretical “factors” of the construct of interest | Number of factors to be determined byc,d: | |
| Kaiser-Guttman rule | Number of factors is equal to the number of factors with eigenvalues >1.0 | ||
| Scree plot | “Elbow” of the graph where eigenvalues level off is the point of significant factors | ||
| Parallel analysis | Retain the number of factors where eigenvalues of the sample data are higher than those from simulated data | ||
| Confirmatory factor analysis | Statistical technique to confirm if data fit hypothesized factor structure | Model fit usinge: | |
| Root mean squared error of approximation | <0.06 excellent | ||
| Comparative fit index | >0.90 acceptable, >0.95 excellent | ||
| Tucker-Lewis index | >0.90 acceptable, >0.95 excellent | ||
| Goodness of fit indices | >0.95 acceptable | ||
| Standardized root mean squared residual | <0.8 acceptable fit | ||
| Convergent | Ability to strongly positively or negatively correlate with other instruments that measure similar constructs | Correlation coefficients dependent upon type of datab: | |
| Pearson r for continuous variables | |0.3|-|0.49| is weak correlation | ||
| |0.5|- |0.69| is moderate | |||
| Divergent | Ability to not correlate with other instruments that measure not-similar constructs | Spearman ρ for ordinal variables | |0.7| - |0.89| is strong |
| |0.9| - |1| is very strong | |||
| Tetrachoric correlation coefficient for dichotomous variables | Note: Positive values indicate positive correlation, whereas negative values indicate negative correlation | ||
| Criterion | |||
| Concurrent | Ability of an instrument to predict current outcomes | Correlation coefficients dependent upon type of datab: | |
| Pearson r for continuous variables | |0.3|-|0.49| is weak correlation | ||
| |0.5|- |0.69| is moderate | |||
| Predictive | Ability of an instrument to predict future outcomes | Spearman ρ for ordinal variables | |0.7| - |0.89| is strong |
| |0.9| - |1| is very strong | |||
| Tetrachoric correlation coefficient for dichotomous variables | Note: Positive values indicate positive correlation, whereas negative values indicate negative correlation | ||
| Known group and divergent | Comparing known groups on survey outcomes to detect hypothesized differences | t tests, analyses of variance, regression models, etc. | |
Tavakol M, Dennick R. Making sense of Cronbach’s α. Int J Med Educ. 2011;2:53–55.
Schober P, Boer C, Schwarte LA. Correlation coefficients: appropriate use and interpretation. Anesth Analg. 2018;126(5):1763–1768.
Kline TJB. Psychological Testing: A Practical Approach to Design and Evaluation. Sage Publications; 2005.
Watkins MW. Exploratory factor analysis: a guide to best practice. J Black Psychol. 2018;44(3):219–246.
Brown TA. Confirmatory factor analysis for applied research. The Guilford Press; 2006.