Assessing the quality of students' ratings of faculty members at Mu'tah University
Main Article Content
Students are not used as raters to evaluate or judge faculty members’ academic performance in some universities because of the lack of trust. In this study I examined the extent to which students can give consistent and reliable ratings. Nineteen graduate students were asked to give ratings to the academic performance of a faculty member on 2 occasions with 2 weeks in between ratings. Results showed evidence of interrater agreement and rater consistency. With appropriate training, students can be a reliable source of information about faculty members’ academic performance.
In a number of domains, the use of raters to assess performance on specific tasks is common. Judgments given on air crew resource management skills determine whether or not an individual is expected to be a successful pilot. Judgments given by a panel of raters determine the chance of getting a particular job. Ratings of gymnastics judges determine which participant will win the gold medal. These examples, and many others, show how important the decision could be and how precise, reliable, and unbiased ratings and judgments should be.
Evaluation of faculty involves the gathering of information for understanding and improving performance as well as for judging its quality. The purposes of faculty evaluation can be reduced to two: on the one hand, faculty evaluation has a formative function – that is, the results are used to support faculty development, growth, and self-improvement; on the other hand, it has a summative function – that is, the results are used to make personnel decisions on tenure, promotion, reappointment, and salary. However, the majority of community college faculty members and administrators surveyed identified faculty development as the primary purpose, with the provision of information on promotion, retention, dismissal, and normal salary increments as a secondary purpose (Rifkin, 2002). Quantitative ratings given by students provide one of the most common methods for evaluation of faculty and teaching effectiveness. Seldin (1984) found that administrators utilized student rating data in two-thirds of 616 institutions surveyed. However, it is also the method that raises the most concerns (Rifkin, 2002). Besides the grades given in the course, the rigor of the course, teaching style, and teacher’s personal characteristics are possible factors affecting the accuracy of ratings (Jean, Ashok, & Dawn 2002). Obtaining accurate and reliable performance ratings is a challenge faced in most educational and employment settings. Reliability coefficients of ratings given in practical settings are typically quite low, ranging from 0.30 to 0.75. As mentioned, ratings of performance are frequently used as the sole basis for making extremely important personnel decisions. It may be argued that the reliability of performance ratings is generally inadequate to make decisions as important as those mentioned (Raymond & Houston, 1990). Therefore, the need to acknowledge the nature of student evaluations is paramount.
This study was aimed at investigating interrater agreement and rater consistency. The specific substantive issues considered were:
Whether or not students, as raters, demonstrate consistency in their ratings of a faculty member’s performance in an academic setting over time.
Whether or not students demonstrate an acceptable level of interrater agreement.
Method
Participants
Participants were fifty-six graduate students (40 males and 16 females) registered at Mu’tah University, Jordan during the second semester 2003-2004. They were asked to rate the teaching performance of a faculty member (of their choice) who was not teaching them during that particular semester, but who had taught them in the previous semester. This condition was set so students would not feel threatened. It happened that the largest number of students (19) rated the same teacher. Ninety per cent of the participants were teachers at schools. The reason for selecting graduate students only was the proposition that ratings at this level were expected to be mature, rational, and unbiased.
Instrument
A survey of 23 items called the Teaching Evaluation Form (TEF) was developed for the purpose of the study. To check the appropriateness of the survey, 12 faculty members at the College of Educational Sciences were asked to examine each item and determine whether or not it was appropriate on a 2-point scale. Only 10 judges gave responses. An item was considered appropriate if a minimum of 8 judges indicated that it was so. Three items were dropped due to irrelevance. With two weeks in between, the test-retest reliability coefficient was found to be significant (r = 0.681, p < .000). Furthermore, principal components analysis with varimax rotation was run to check the instrument’s dimensionality. This revealed three main factors with eigenvalues of 12.04, 2.41, and 1.63, which explained 33.67%, 23.55%, and 23.24% of the variance, respectively. The factors were labeled as follows: a) in-class teaching effectiveness, b) interpersonal student-teacher relationship, and c) student assessment fairness.
Procedure
The survey was administered for the first round (FR) four weeks before the final examination. Instructions for filling out the survey were given to ensure understanding of the way of responding. It was made clear that the information was being collected for research purposes and that it would be treated as confidential. Each student had been asked to write a secret symbol or word or number to use as a code. Instructions ensured that students’ grades would not be affected in any way by their responses to the survey. Students were encouraged to respond freely and honestly. Two weeks later, a second round (SR) of the TEF was administered to the same group. Students were asked to use the same code they used for the first round.
Results
The purpose in this research was to investigate the notion of interrater agreement and rating consistency. Therefore, we checked agreement between different raters, and consistency of the same rater in rating a group of items on two occasions.
Interrater Agreement
For each student, the total score ratings given for all items were computed for both FR and SR. To check the differences between ratings given by all students at both occasions, a paired-sample t test was used (t test value = -0.91, p = 0.37). This means that total ratings given by different raters of both samples were generally consistent on both occasions. This was true for the first sample data set as well.
Law and Sherman (2000) used an unbiased index of interrater agreement developed by James, Demaree, and Wolf (1984; 1993). It is the within-group interrater reliability coefficient (rwg), which assesses interrater agreement. The rwg was defined as the proportional reduction to error variance of a distribution of obtained ratings compared to a distribution representing random responding (i.e., a distribution at which each point of the rating scale has equal probability of being utilized.)
Table 1. rwg between FR and SR Performed for Each Item
Where S2x equals the variance of the observed ratings, and σe2 equals the population variance of a discrete rectangular distribution of ratings which represents a random response pattern (i.e., an equal number of ratings in each possible response category). Values of rwg can range from 0.0 to 1.0. A value of
0.0 reflects the fact that there is no agreement among raters (i.e., when the variance of the obtained scores is random (S2x = σe2). When all raters give the same rating (i.e., S2x = 0.0), then rwg = 1, reflecting that there is perfect agreement among all raters.
To assess the interrater agreement the rwg coefficient was used. Coefficients were computed for all items and are presented in Table 1.
Coefficients ranged between 0.18 to 0.83 and between 0.17 to 0.84 for the FR and SR simultaneously. In the FR items 12 and 18 had low coefficients (i.e., less than 0.2). In the SR, it was only item 14 which showed low coefficients. This low percentage of items with low coefficients is evidence of strong agreement among raters for 95% of the items. Note that the items were related to the academic performance of the same teacher. The correlation coefficient between reliability coefficients of the FR and the SR was 0.607, α = 0.006.
Table 2. Correlations Between Ratings Given by the Same Rater at the FR & SR
Note: **Correlation is significant at p < .01 level (two-tailed).
Rater Consistency
To check the consistency of ratings given by the same rater on both occasions correlation coefficient was employed. Of the 19 raters utilized, 10 raters showed significant correlations between their rating at the FR and ratings at the SR. One other rater (Rater 6) has no r value because he/she gave one rating category for all items on both occasions. Ratings of the remaining eight raters showed nonsignificant correlations. This means that 52% of the raters gave consistent and reliable ratings.
Conclusion
This study was aimed at investigating the notion of interrater agreement and raters’ consistency when the raters are students. Despite the lack of clarity as to the goals of evaluation and their application, and how they should be involved in the evaluation process, practitioners agree that evaluation is a necessary part of teaching and learning. They also agree in principle that evaluation should help instructors grow professionally, but are unclear as to how to achieve that goal.
Students, as raters, showed evidence of agreement and reliability regarding what they have rated. However, it is possible that rater training helps to enhance the level of agreement and consistency among students as raters. In conclusion, students should be part of the process of evaluating faculty members. Such results should encourage universities and other academic institutions to reactivate and improve the process of evaluating faculty members. Students could be an important part of this process. It is recommended that more research be conducted to find ways of making students’ role in evaluating faculty members’ academic performance more useful.
James, L. R., Demaree, R. G., & Wolfe, G. (1984). Estimating within-group interrater agreement. Journal of Applied Psychology, 69, 85-98.
James, L. R., Demaree, R. G., & Wolfe, G. (1993). rwg: An assessment of within-group interrated agreement. Journal of Applied Psychology, 78, 306-309.
Law, J. R., & Sherman, P. J. (2003). Do raters agree? Assessing inter-rater agreement in the evaluation of air crew resource management skills. Available at: http:\\www.psy.utexas.edu/psy/helmreich/osurlpsd.htm
Raymond, M. R., & Houston, W. M. (1990). Detecting and correcting for rater effects in performance assessment. Educational Resources Information Center document, 336429.
Rifkin., T. (2002). The status and scope of faculty evaluation. Available at: http:\\www.ed.gov/databases/ERIC_Digests/ed385315.html.
Seldin, P. (1984). Faculty evaluation: Surveying policy and practices. Change, 16, 28-33.
Sojka, J., Ashok, K. G., & Deeter-Schmelz, D. (2002). Students and faculty perceptions of student
Table 1. rwg between FR and SR Performed for Each Item
Table 2. Correlations Between Ratings Given by the Same Rater at the FR & SR
Note: **Correlation is significant at p < .01 level (two-tailed).
Appreciation is due to reviewers including
Mousa Alnabhan
Psychology Department
Faculty of Education
Mu&rsquo
tah University
PO Box 79
tah
Jordan
Abdalla Alsmadi, Department of Psychology, Mu’tah University, PO Box 79, Mu’tah, Karak 61710, Jordan. Email: [email protected]