Online Evaluations in Higher Education - Utilising Information Systems in Assessing the Validity of Student Evaluations

Abstract

Practically all university students have responded to at least some student evaluations in the course of their studies. For their part, most lecturers also collect them in one way or another. The use of student evaluations is extensive, ranging from developing a single course to even salary and tenure decisions of the whole faculty. Because of the significance of the evaluation results, also the validity and reliability of student evaluations have become issues of heated debate. Many research findings have been published dealing with such topics, yet the debate still goes on unresolved.

During the last decade, many universities have replaced traditional paper-based evaluations with online evaluation systems, but in most cases this has simply meant converting the existing paper-based questionnaires into electronic format. Also research into online ratings has focused mainly on investigating the differences between online and paper-and-pencil ratings systems and on describing experiences of the developed online evaluation systems. What has not hitherto been recognised is the potential of information technology to implement and analyse the evaluations.

This study concentrates on assessing the additional value to be gained by connecting online student evaluations to universities’ other databases, especially to analyse the validity of student ratings. The study approach follows the principles of decision-oriented research approach. After initial determination of the research problem, the first phase involved identifying the gaps and contradictions in the current knowledge in the field. Next, an online survey was developed and implemented, the results of which were linked up (with respondents’ permission) to other university databases for further analysis. These were then compared with previous research findings.

The results of this study suggest that an interlinked system can aid in assessing the validity of student evaluations by enabling observations at an individual respondent level instead of using averages and estimations. Such a system also reduces the need for complex statistical analyses which demand expertise to be interpreted correctly. This study also gives examples of issues which either have been actively researched but unsatisfactorily resolved or which have earlier been impossible to analyse. For example, this study demonstrates that it is possible to analyse individual final course grades connected to the individual students’ survey responses even at a single questionnaire item level. The effect of the order in which individual students undertake their courses connected to the same students’ evaluation results and their individual final grades from the same courses, was an example of an issue not previously investigated at the individual level.