Abstract
This article describes an application of generalizability theory (GT) to data from three different groups of supervisors/managers evaluated for developmental purposes using a brief, internally developed 360-degree feedback measure. Major goals of this research were to examine the generalizability of 360-degree feedback ratings, and to compare the 360-feedback measure to operational selection tools previously used in this organization (structured behavioral interview and writing assessment). GT results are presented separately for three groups of assessees (first-line supervisors, second-line supervisors, and middle managers). In all cases, the 360-degree measure demonstrated disappointingly low levels of reliability (range of 0.27 to 0.51 depending on assumptions made), while the interview and writing assessment provided high levels of reliability (0.87 to 0.91 under the same assumptions). The results presented here, in combination with other generalizability analyses of multi-rater feedback, suggest that practitioners should carefully consider the appropriateness of applications of 360-degree feedback measures, and demonstrate that reliability of such measures is insufficient for applications such as selection or compensation.