A new study in The BMJ, partly funded by HAI, found that half of cancer drugs approved by the European Medicines Agency (EMA) between 2014 and 2016 were based on potentially biased clinical trials.

The study, which was published in the BMJ last week, looked at the clinical studies which supported new cancer drugs approved by the EMA between 2014 and 2016. The EMA approved 32 new cancer drugs on the basis of 54 pivotal studies. 76% of these studies were randomised controlled trials, typically considered best practice.

However, the team of researchers, led by Dr Huseyin Naci, Assistant Professor of Health Policy at London School of Economics, found issues with a significant proportion of these pivotal trials. These issues ranged from how a trial was designed, how it was conducted (whether there was a risk of bias), and how its findings were reported.  This blog aims to unpack some of the findings of the report.

A short video explanation of navigating the Naci et al. paper.

Trial design

The study found that only 10 of 39 of the randomised controlled trials investigated evaluated overall survival as either a primary or co-primary endpoint. This means the majority of new cancer drugs had other factors—‘surrogates’, such as progression free survival and disease responses—as their endpoint.

Typically, surrogate endpoints are chosen because they mean a clinical trial can end sooner and a drug can be authorised more quickly: timely access to new medicines can be a good thing, but only where efficacy, as well as safety, are absolutely guaranteed. Decisions on the design of clinical trials should be based on these factors, and not on how quickly a pharmaceutical company can turn a profit.

The other concern with surrogate endpoints is that the drug may have no impact on overall survival. Recent research shows that these other factors do not consistently translate to survival gains or quality of life benefits1, 2, 3,4,5, and could even prove harmful6. Indeed, the BMJ study shows that ‘evidence of overall survival benefit might never emerge for cancer drugs approved on the basis of surrogate measures alone’. What’s more, no overall survival data emerged post-marketing in 90% of cases where there was no such benefit indicated at the point of marketing authorisation. Use of surrogate endpoints isn’t an adequate alternative for evidence of improvements in overall survival—and companies should be expected to produce this evidence even after the drug goes onto the market. Testing and improving of drugs shouldn’t stop when the drug starts to be used by patients: if anything, this is a crucial period for determining the real life benefits of a treatment. 

All these issues are important given the debate around the added therapeutic value of drugs, and the inevitably high price tags they come with—and when we consider the expectations of patients and families that a new drug will be better than what’s already available.

cancer drug

Bias and reporting

Naci et al’s research investigated the risk of bias in clinical trials, not the existence of bias itself. Nonetheless they found several pivotal trials at risk of bias across multiple domains. The risk of bias was caused by several factors: how participants were randomised; changes to the intended intervention the trial was testing (which is related to the surrogate endpoints we’ve already discussed); and how the findings of the trial were reported. The differences between how trials were reported in different sources (for example, on a database and in a medical journal), were so significant that they changed the analysis of the study. Moreover, this study indicates a higher risk of bias in studies with surrogate endpoints than with overall survival as their endpoint, indicating another reason to be cautious about this type of trial.

This research also found a substantial amount of missing data including missing outcomes and, as we’ve seen, it’s highly unlikely for this evidence to appear post-marketing authorisation. These drugs, therefore, are on the market without completed, transparently-reported clinical trials to support them. Transparency in clinical trials is vital to democratic, rational access to medicines, so these findings are concerning—doubly so if the EU Clinical Trial Regulation, which makes reporting of trials compulsory, is not being followed.

Conclusions and next steps

Although significant, this study shouldn’t be of undue concern to patients and families, and we certainly aren’t implying that the EMA is at fault for authorising these drugs. But it’s clear that increased scrutiny is required to ensure drugs entering the market are robustly evidenced as increasing therapeutic value, and have met EU regulations around transparency. As its authors conclude, these findings should lead to improvements in the design, conduct and reporting of cancer drugs trials, and regulatory action to ensure pharmaceutical companies are evaluating products meaningfully, so that patients, clinicians and healthcare systems have the information they need.

In fact, the report itself has exemplified how information can be better shared: Aero Data Lab have turned the findings of this research into a living website which allows medical professionals, policy makers, and equally importantly, patients and their families, to search through its findings and understand how each trial included performed. This is a great step towards the kind of transparency in medicines that HAI hopes to see everywhere.

1. Prasad V, Kim C, Burotto M, Vandross A. The strength of association between surrogate end points and survival in oncology: a systematic review of trial-level meta-analyses. JAMA Intern Med2015;175:1389-98. doi:10.1001/jamainternmed.2015.2829 pmid:26098871

2. Haslam A, Hey SP, Gill J, Prasad V. A systematic review of trial-level meta-analyses measuring the strength of association between surrogate end-points and overall survival in oncology. Eur J Cancer2019;106:196-211. doi:10.1016/j.ejca.2018.11.012 pmid:30528804

3. Kovic B, Jin X, Kennedy SA, et al. Evaluating progression-free survival as a surrogate outcome for health-related quality of life in oncology: a systematic review and quantitative analysis. JAMA Intern Med2018;178:1586-96. doi:10.1001/jamainternmed.2018.4710 pmid:30285081

4. Amir E, Seruga B, Kwong R, Tannock IF, Ocaña A. Poor correlation between progression-free and overall survival in modern clinical trials: are composite endpoints the answer?Eur J Cancer2012;48:385-8. doi:10.1016/j.ejca.2011.10.028 pmid:22115991

5. Hwang TJ, Gyawali B. Association between progression-free survival and patients’ quality of life in cancer clinical trials. Int J Cancer2019;144:1746-51. doi:10.1002/ijc.31957 pmid:30374970

6. Kumar S, Rajkumar SV. Surrogate endpoints in randomised controlled trials: a reality check. Lancet2019;394:281-3. doi:10.1016/S0140-6736(19)31711-8. pmid:31354129