Distinguishing true from false positives in genomic studies: p values

Linda Broer, Christina M. Lill, Maaike Schuur, Najaf Amin, Johannes T. Roehr, Lars Bertram, John P. A. Ioannidis, Cornelia M. van Duijn

Distinguishing true from false positive findings is a major challenge in human genetic epidemiology. Several strategies have been devised to facilitate this, including the positive predictive value (PPV) and a set of epidemiological criteria, known as the "Venice" criteria. The PPV measures the probability of a true association, given a statistically significant finding, while the Venice criteria grade the credibility based on the amount of evidence, consistency of replication and protection from bias. A vast majority of journals use significance thresholds to identify the true positive findings. We studied the effect of p value thresholds on the PPV and used the PPV and Venice criteria to define usable thresholds of statistical significance. Theoretical and empirical analyses of data published on AlzGene show that at a nominal p value threshold of 0.05 most "positive" findings will turn out to be false if the prior probability of association is below 0.10 even if the statistical power of the study is higher than 0.80. However, in underpowered studies (0.25) with a low prior probability of 1 × 10(-3), a p value of 1 × 10(-5) yields a high PPV (>96 %). Here we have shown that the p value threshold of 1 × 10(-5) gives a very strong evidence of association in almost all studies. However, in the case of a very high prior probability of association (0.50) a p value threshold of 0.05 may be sufficient, while for studies with very low prior probability of association (1 × 10(-4); genome-wide association studies for instance) 1 × 10(-7) may serve as a useful threshold to declare significance.