predictive validity and adverse impact of an assessment center, a multirater appraisal system and top-down appraisal for selecting retail store managers

This study compares the levels of criterion-related validity and the extent of statistical adverse impact against minorities based on supervisors' top-down (TD) performance appraisals, a multirater appraisal (MRA) system and a traditional assessment center (AC). Data from all three sources were collected as a part of a predictive validity study using criterion data completely independent of the predictor data and where managers who provided criterion data had no knowledge of the predictor scores. The performance of the selected managers was used as the dependent variable. This is the first study in which direct comparisons were made for predictive validity and adverse impact among these three methods of staffing decisions for the same subjects. As hypothesized, the assessment center (uncorrected r = .32) and multirater methods (uncorrected r = .31) had the highest levels of predictive validity. The predictive validity of the top-down managerial assessment was significantly lower than the other two methods, and the correlation with subsequent managerial performance was not statistically significant (uncorrected r = .13). It was also hypothesized that multirater appraisal and assessment centers would result in less adverse impact than top-down appraisal. After examination by several commonly used methods, only partial support was found for these hypotheses. Analyses using the 80% rule and z-scores found that data from the AC and MRA methods did not indicate adverse impact against minorities while TD data resulted in adverse impact. Possible reasons for the discrepant findings of the various methods of analysis and implications for expert witness testimony are discussed.