Should low p-values no longer be regarded as the “Holy Grail”?

It is nowadays unimaginable and almost impossible to publish scientific results without having performed a robust statistical analysis supporting the presented results. This statistical analysis is usually evaluated by a p-value to test the significance of an observation. The p-value has thus become the gold standard in science. Although it is necessary to use vigorous statistical analyses, it is even more important to provide an accurate interpretation of the results to avoid misunderstandings. This is at least the opinion shared by Amrhein and over 800 of his fellow peers in a recently published article in NATURE.

The authors warned about the misinterpretation of p-values and confidence intervals. For instance, discarding results as not being statistically significant because of a high p-value should be avoided. Similar warnings also go for confidence intervals that should not be considered as irrelevant just because a zero value is included in the interval. Likewise, obtaining a low p-value should not become the "Holy Grail" as results still have to be interpreted within the context of the study. Categorising studies as "statistically significant" and "statistically non-significant" bears the danger of a potential loss of valuable information and could lead to false conclusions.

Instead of blindly applying a threshold, an arbitrary number to assign a statistical significance to a study, the authors advocate to move from this categorising concept and proposed different solutions to facilitate this shift. These include:

• Renaming confidence intervals as "compatibility intervals" and interpreting what the observed values mean within the context of the study;
• Discussing the point estimate in light of the different uncertainties to avoid making false claims;
• Understanding that the p-value and the default 95% confidence interval are both arbitrary values that should not be overinterpreted outside of this context;
• Assessing statistical assumptions with the accurate statistical analyses.

Overall, it is clear that a change towards moving away from this categorisation concept to embracing an interpretation-based approach is happening. This shift could influence the establishment of new regulations by authorities such as the IQWiG and the G-BA in the assessment of an added benefit of drugs.

For all questions concerning the preparation of a value dossier for the German AMNOG process, SKC consulting is as a competent and dedicated partner at your side.

BY Prof. Matthias P. Schönermark, M.D., Ph. D. and Dr. rer. nat. Esther Nkuipou Kenfack

Reference:
Amrhein V, Greenland S, and McShane B. Retire statistical significance. Nature 2019, 567, 305-307.