Experimental results about the assessments of conditional rank correlations by experts: Example with air pollution estimates

Science-based models often involve substantial uncertainty that must be quantified in a defendable way. Shortage of empirical data inevitably requires input from expert judgment. How this uncertainty is best elicited can be critical to a decision process, as differences in efficacy and robustness of the elicitation methods can be substantial. When performed rigorously, expert elicitation and pooling of experts' opinions can be powerful means for obtaining rational estimates of uncertainty. Causes of uncertainty may be interrelated and may introduce dependencies. Ignoring these dependencies may lead to large errors. Dependence modelling is an active research topic, and methods for dependence elicitation are still very much under development. Dependence measures such as rank correlations are commonly used in different types of models. Eliciting rank correlations and conditional rank correlations from experts have been proposed and used in the past. Conditional rank correlations are not elicited directly from experts, rather the experts are asked to estimate some other related quantities. In this paper two methods for eliciting conditional rank correlations via related quantities are compared in order to obtain insight about which of the two renders more accurate estimates of conditional rank correlations. Our data shows that good performance in uncertainty assessments does not automatically translates into good performance in dependence estimates. We show that, analogously to uncertainty estimates, combining experts' estimates of dependence according to their performance results in better estimates of the dependence structure. © 2014 Taylor & Francis Group, London..