Evaluation of reported claims of sex-based differences in treatment effects across meta-analyses: a meta-research study

Importance Differences in treatment effects between men and women may have important implications across diverse interventions and diseases. Objectives We aimed to evaluate claims of sex-based differences in treatment effects across published meta-analyses. Eligibility criteria Published meta-analyses of randomised controlled trials (RCTs) that had any mention of sex (male/female) subgroup or related analysis in their abstract Information sources PubMed (searched up to 17 January 2024). Synthesis We determined how many meta-analyses had made claims of sex-based differences in treatment effects. These meta-analyses were examined in depth to determine whether the claims reflected sex-treatment interactions with statistical support or fallacious claims, and we categorised the frequency of different fallacies or genuine interactions. We also investigated how many of the genuine and fallacious claims were considered and discussed in Up-To-Date. Whenever possible, we reanalysed the p value for sex-treatment interaction. Main outcomes and measures Number of claims with statistical support and fallacious claims; clinical implications of subgroup differences as well as the credibility of subgroup analyses assessed by the Instrument to assess the Credibility of Effect Modification Analyses criteria. Results 216 meta-analysis articles fulfilled the eligibility criteria. Of them, 99 stated in the abstract that there was no sex-based difference, and 20 mentioned a sex-based subgroup analysis but without reporting results in the abstract. The other 97 meta-analyses made 115 claims of sex-based differences. 27 of the 115 positive claims for subgroup differences made across 21 articles had statistical support at p<0.05, of which 4 were mentioned in Up-To-Date, with none leading to different recommendations for men and women. 39 of the 115 positive claims made across 35 articles were fallacious, where the sex-treatment interaction was not statistically significant. The most common form of fallacy (29/115) was made in instances where there was a significant effect in one sex, but not in the other, with no true difference between the two groups. In 7/115 other claims, there were larger effects in one sex, again, with no true difference between the two groups, and 3/115 other claims had various forms of fallacies. Another 44 articles made 49 claims based on potentially fallacious methods (44 based on meta-regression, and 5 provided the results of only one group), but proper data were unavailable to assess statistical significance. Conclusions and relevance Few meta-analyses of RCTs make claims of sex-based differences in treatment effects, and most of these claims lack formal statistical support. In the present sample, statistically significant and clinically actionable sex-treatment interactions were rare.