We evaluated Samaniego and Reneau's 1994 novel weight method for eliciting subjective probability estimates. Experiment 1 replicated their experiment (subjects weighed their prior estimate against 10 new observations), with an additional weight judgment against 50 observations. In Experiment 2, subjects gave prior estimates to questions in a domain more familiar to them. In Experiment 3, subjects weighed priors against samples of 10 or 50 on a modified scale. We find substantial individual differences in sensitivity to sample size in this task, and also evidence for implicit learning: Subjects who first judged N = 50 subsequently place lower weights on N = 10. People cannot evaluate the quality of their prior knowledge accurately, nor can they consistently produce calibrated priors and marginal posteriors. These results do not support the use of the weight-judgment method for parameter estimation , but do encourage its continued development and improvement.
Li, Yuelin and Krantz, David H.
"Experimental Tests of Subjective Bayesian Methods,"
The Psychological Record:
2, Article 5.
Available at: http://opensiuc.lib.siu.edu/tpr/vol55/iss2/5