Clinical and practical importance versus statistical significance: limitations of conventional statistical inference.

Decisions about support for therapies in light of data are made using statistical inference. The dominant approach is null-hypothesis-significance-testing. Applied correctly it provides a procedure for making dichotomous decisions about zero-effect null hypotheses with known and controlled error rates. Type I and type II error rates must be specified in advance and the latter controlled by a priori sample size calculation. This approach does not provide the probability of hypotheses or the strength of support for hypotheses in light of data. Outcomes allow conclusions only about the existence of non-zero effects, and provide no information about the likely size of true effects or their practical / clinical value. Magnitude-based inference, allows scientists to estimate the ‘true’ / large sample magnitude of effects with a specified likelihood, and how likely they are to exceed an effect magnitude of practical / clinical importance. Magnitude-based inference integrates elements of subjective judgement central to clinical practice into formal analysis of data. This allows enlightened interpretation of data and avoids rejection of possibly highly-beneficial therapies that might be ‘not significant’. This approach is gaining acceptance, but progress will be hastened if the shortcomings of null-hypothesis-significance testing are understood.