Measuring change in training programs: An empirical illustration

Abstract

The implementation of training programs often requires a complex design if effectiveness is to be accurately evaluated. Part of the difficulty lies in the fact that trainees must be presented with a series of ever-changing tasks in order to avoid biases due to learning or carryover effects. The aim of the present study is to experiment and illustrate a simple procedure, based on a special case of the linear logistic test model (LLTM), used to evaluate the effectiveness of a training program. The procedure is empirically applied to a dataset derived from a moped riding skills training program. The sample is composed of 207 high school students who took part in three training sessions using a riding simulator. A different task presentation order was assigned to each subject and the whole design was completely balanced. The procedure applied allowed us to obtain estimates of the overall change in ability that occurred over the course of the training process. Furthermore, we were able to obtain estimates of item and subject parameters unbiased by the influence of change in ability due to training. Implications of the results are discussed and suggestions for future research are presented.