Multiplicity and Scientific Inference

From: lillyblog2, MD, Family Medicine, 12:34PM Nov 2, 2009.

Here is a thought experiment. Suppose you have 15 dice with all six sides having a single letter of the alphabet. In the first part of the experiment, you roll all 15 dice and your job is to find at least one 7-letter word from the 15 letters that are showing face up. Assuming the 15 die have a reasonable representation of letters, there is a very high probability that you will be able to find a 7-letter word.

Now, in part 2 of the experiment, you roll all 15 dice and see if you can make the word 'stephen'. The probability of being able to do this is quite small.

The point is this: When you collect a lot of data from almost any experiment, and you do not have a specific idea/hypothesis/analysis that you specify in advance, you are quite likely to find some pattern or relationship in the data - including ones that may appear biologically plausible. In statistics, this is known as the multiplicity problem: If one tests enough hypotheses (prospectively or retrospectively, as in post-hoc analysis), it is quite likely that some statistically significant result will be found.

Good science (up-front planning, a priori definition of hypothesis), coupled with appropriate statistical methodologies to control the probability of 'false positive' findings, is essential to demonstrating credible scientific findings. Here is what we were wondering: How many physicians appreciate the issue of multiplicity? When you read medical journal articles, how do you evaluate/know/decide whether the results are part of an a priori analysis or a post hoc analysis? How does this impact the way you view published data and incorporate it in your clinical decision-making?

Some researchers don't have a good understanding of statistics and this can lead to problems, for example see Why-Published-Research-Findings-Are-Often-False.

The inability to repeat experimental results is also problematic, see The Truth Wears Off.