Course: The Fragility of Statistically Significant Results in Randomized Clinical Trials for COVID-19
CME Credits: 1.00
Released: 2022-03-18
Key Points
Question In randomized clinical trials (RCTs) of COVID-19 that report statistically significant results, what is the fragility index, ie, the minimum number of participants who would need to have had a different outcome for the RCT to lose statistical significance?Findings In this cross-sectional study of 47 RCTs with a total of 138,235 participants that had statistically significant results, the median fragility index was 4. That is, a median of 4 events was required to change the analysis findings from statistically significant to not significant.
Meaning In this study, many RCTs for COVID-19 had a low fragility index, challenging confidence in the robustness of the results.
Abstract
Importance Interpreting results from randomized clinical trials (RCTs) for COVID-19, which have been published rapidly and in vast numbers, is challenging during a pandemic.Objective To evaluate the robustness of statistically significant findings from RCTs for COVID-19 using the fragility index.
Design, Setting, and Participants This cross-sectional study included COVID-19 trial articles that randomly assigned patients 1:1 into 2 parallel groups and reported at least 1 binary outcome as significant in the abstract. A systematic search was conducted using PubMed to identify RCTs on COVID-19 published until August 7, 2021.
Exposures Trial characteristics, such as type of intervention (treatment drug, vaccine, or others), number of outcome events, and sample size.
Main Outcomes and Measures Fragility index.
Results Of the 47 RCTs for COVID-19 included, 36 (77%) were studies of the effects of treatment drugs, 5 (11%) were studies of vaccines, and 6 (13%) were of other interventions. A total of 138,235 participants were included in these trials. The median (IQR) fragility index of the included trials was 4 (1-11). The medians (IQRs) of the fragility indexes of RCTs of treatment drugs, vaccines, and other interventions were 2.5 (1-6), 119 (61-139), and 4.5 (1-18), respectively. The fragility index among more than half of the studies was less than 1% of each sample size, although the fragility index as a proportion of events needing to change would be much higher.
Conclusions and Relevance This cross-sectional study found a relatively small number of events (a median of 4) would be required to change the results of COVID-19 RCTs from statistically significant to not significant. These findings suggest that health care professionals and policy makers should not rely heavily on individual results of RCTs for COVID-19.
Educational Objective
To identify the key insights or developments described in this article
View Full Course