A critical review of validation methods for man-in-the loop simulators

This review examines the methodological concepts, paradigms and pitfalls related to validation- and fidelity studies of man-in-the-loop simulators. A distinction is made between validation methods for training simulators and for research simulators. Validation methods for training simulator are applied in experiments which assess effects of simulator variables (e.g. resolution of the display, cue augmentation, moving base characteristics) on the effectiveness of a simulator as a training device. Validation methods for research simulators are applied in experiments which assess the effects of simulator variables on the effectiveness of a simulator as a research tool. The review is particularly focussed on the various artefacts that may affect the outcome of such validationhe artefacts are separately described for each single validation method. It will be demonstrated that validation of simulators is a very complicated matter and prone to various methodological flaws and confounding factors. After the discussion of the common methods including their advantages and disadvantages the following recommendations for future research are given:
1 Terminology in the field of simulator research is ambiguous. It is advised to standardise terms, which will lead to more comprehensible communication among researchers. 2 Validity is not a single, independent attribute. The term validity in simulator research only makes sense if related to functional aspects of simulators, such as the purpose of the simulator (training, research) and the tasks and training methods involved. This will reduce the amount of overgeneraliza-tions that are now encountered too frequently. 3 Always take face validity into consideration. If people do not believe in the simulator they are not v they are not very likely to use it properly.
4 Apply more than one method in a simulator validity study. Combination of e.g. objective with subjective methods reduces the risk of erroneous conclusions and combines the benefits of both kinds of methods.
5 Aim more research at creating task-specific formulas, which relate physical simulator variables to psycho-physical and human performance variables. This will reduce the need to measure human task performance in simulator validation studies. 6 Always allocate substantial effort to find a practical method that still compares simulator performance with on-the-job performance by subjects. A relatively simple and practical method is proposed to assess the effectiveness of a driving simulator training.