On the appropriateness of appropriateness judgments

Abstract

We compare experts' judgments of the appropriateness of a treatment (interferon treatment for melanoma) on the basis of important attributes of this disease (thickness, ulceration, lymph node involvement and type of metastases) to a decision analytic model in which the probabilities of deterioration are derived from the medical literature and from epidemiological studies. The comparison is based on what we call extit{the linearity test}, which examines whether appropriateness judgments are a linear function of the epidemiological value of $p_2$, the probability of deterioration of the patient condition if he would have received the treatment. This comparison allows for the assessment of the validity of the experts' judgments under the assumption that the decision analytic model is valid, or alternatively, the assessment of the validity of the decision analytic model under the assumption that the experts' judgments are valid. Under the former assumption the results indicate that appropriateness judgments are by and large accurate. Under the latter assumption the results support the idea of a extit{constant treatment effect}, the idea that efficacy of a treatment is constant over various levels of severity of the disease. Our results also support the idea that experts' aggregate judgments far exceed individuals' judgments.