This presentation is based on the article “How to Read a Clinical Trial Paper: A Lesson in Basic Trial Statistics” published on Gastroenterology and Hepatology in 2012 by Dr Higgins and Dr Govani. Its aim is to help clinicians to incorporate the results of clinical trials into their daily clinical practice.
Nowadays clinical trials, which travel at the rate of 18,000 new studies in the United States alone, make it impossible for gastroenterologists to keep up with the results. Despite the increasing number of clinical trials performed yearly, their results are not always directly applicable to clinical practice.
Basic statistics can help clinicians introduce the results of clinical trial papers in their clinical practice.
A list of ten points can help clinicians to define an easy road map to follow when reading the findings of a new study. Here are presented the first 4.
Are These Subjects Like My Patients?
A well-studied trend demonstrates that patients enrolled in clinical trials usually have conditions worse than the patients a clinician sees in his or her practice. Thus, it is crucial to check if enrolled patients are representative while reading a clinical paper.
The first table of a clinical trial publication usually summarizes not only the population’s age, race, and gender but also the typical disease severity, disease duration, and medication history. The results of a study are applicable only if all these factors are similar to those of the patients seen in clinical practice.
Another aspect to consider is the recruitment method as it can influence the make-up of a study and bring contrasting response rates.
What Happened to the Subjects? Did They Drop Out? Why?
The statistical significance of a study emerges from a dropout rate due to the lack of benefit in the treatment arm lower than in the placebo arm. This information is placed in the Consolidated Standards of Reporting Trials (CONSORT) diagram, usually the first figure in the paper. The CONSORT diagram shows the flow of subjects from recruitment through the end of the study (or early exit from the study). If the reasons for leaving the study are not specified, the exit rate must be compared for each arm. A lower exit rate should be a direct consequence of the more effective treatment arm in a well-tolerated treatment.
Is the Study Design Biased?
Comparator group, allocation of subjects to treatment arms, and blinding are three crucial aspects of study design and need to be checked with special attention.
The comparator is the standard-of-care therapy in disease states with established therapies such as inflammatory bowel disease (IBD).
The placebo effect can affect the results, for this reason, a placebo group and/or a proven effective comparator have to be included.
The allocation of subjects to each treatment arm should be well defined and concealed while investigators cannot anticipate which arm a new subject will enter.
The CONSORT diagram should be easy to follow and similarly detailed for each arm.
Randomization may be at the population level or clustered at the site level; in the latter case, all the subjects at 1 site are in a single arm.
The blinding process can be single- or double-blind. Single-blind means the patient does not know its allocation, double-blind means even investigators do not know the patient’s allocation.
An overestimation of the treatment effect is evident in studies conducted without blinding.
The blinding process can be challenging, but it is crucial, especially in trials involving a procedure. The success of the blinding process was only 2% of 1,599 studies a 2007 analysis says.
Does the Study Include an Intention-to-Treat Analysis?
An analysis that includes all randomized subjects, even if they do not receive the intervention or may drop out of the study, is called an intention-to-treat (ITT) analysis. On the other side, an analysis, in which only subjects who received their assigned intervention are counted, is named a per-protocol analysis (PPA).
As a high rate of early dropout can damage clinical trials, researchers prefer to present the data of a PPA.
Unfortunately, a PPA is reliable only as a secondary endpoint, that is after the presentation of the ITT results.
While reading a clinical paper, clinicians should openly find out if the presented data are part of an ITT analysis or not.
Let’s proceed with the list of ten points that can help clinicians while reading the findings of a new study. Here are presented the other 4.
Is This a Test of Superiority? Equivalence? Non-inferiority?
The intent to prove that one therapy is superior to another is the goal of many clinical trials. An equivalence or non-inferiority to standard treatment study is intended to demonstrate that a new therapy is more convenient or cheaper.
As the definitions of non-inferiority and equivalence can be ambiguous, those studies can lead to difficulties in the results’ interpretation.
The sample size needed to determine non-inferiority or equivalence is always much higher than the size needed to determine superiority.
The reader should be able to replicate the sample size calculation if necessary, and the methods section of a clinical study should reproduce every single step of the calculation.
In an equivalence study, the 95% confidence interval of the difference between the treatments needs to be less than the previously defined acceptable difference. As a direct consequence, the required sample size of an equivalence study is about 4 times larger than the sample size needed for a superiority study.
If initially a study was designed to test for superiority, but the goal was missed, the decision to switch to an equivalence study results inappropriate, because the acceptable difference was not defined in advance, and the sample size is too small.
Does the Measurement Matter? Is It Reproducible? Accurate?
The study’s measurement of success can make the results of a clinical trial difficult to apply to the everyday practice of a clinician.
Considering the three aspects below can facilitate the transferring of those results to clinical practice:
- Reproducibility and accuracy of a measurement.
Many measures are not very reproducible and some instruments, such as the endoscopic grading or the histologic grading of dysplasia, present a high level of disagreement in the middle of the scale. This disagreement produces noise in the data.
- Amount of change seen in the clinical index or score and significant clinical meaning to real-world patients.
The results of a published paper can be statistically significant, but it does not mean that they are also clinically meaningful. Only clinically meaningful results deserve to be translated into clinical practice.
- Validation of the measurements.
The validity fulfils all the below aspects:
- The index is measuring accurately all the important aspects of a disease.
- The measurement is reproducible in subjects whose condition has not changed.
- The instrument is responsive to small but clinically important changes.
How Are Missing Data Addressed?
The 3 common methods for handling missing data are:
- The non-responder imputation considers any subject who has missing data as one who would have failed to meet the endpoint. The success rates (and placebo rates) are smaller, and the results look not so impressive.
- The last-observation-carried-forward approach uses the last observation at which the subject was measured in place of the missing data. It is used in maintenance studies with repeated measurements. The success rates of both the primary intervention and the control arm may inflate.
- The imputation uses other data to estimate what the missing data would have been. The impossibility to check the accuracy of the estimates makes this approach questionable.
Do the Design and Methods Conform to the Prestudy Guidelines?
Clinical trials, registered on the ClinicalTrials.gov website, give to an interested reader the chance to check if the endpoints changed during the study and if the trial was truly prospective.
If it comes out that some of the outcomes mentioned on the initial registration page are not reported in the published paper, it is reasonable to presume that the outcome results were disappointing.
How Good Are the Results?
Using only the P-value to determine if a clinically important effect is present leads to misinterpretations.
The underlying assumption of the P-value is that the null hypothesis is true. If the treatment arms are equally effective (Null hypothesis true), the observed differences between the 2 treatment arms will be due to chance alone and the P-value gives this probability.
In other words, the P-value serves to exclude that the observed differences are due to chance alone.
While reading the results of a study, two measurements combined with P-value play an important role in results interpretation: effect size and precision.
The effect size is the difference between two groups, and the precision is the variation of that difference represented by the confidence interval.
Even a small difference between the two groups (effect size) can be statistically significant in a very large study (>500 patients per arm), on the other side a big effect size could scarcely be significant in a small study.
For example, a difference of 2,5 mmHg for blood pressure between treatment A and treatment B in a 500 patients study is statistically significant (P-value 0.0498), on the other side, a difference of 12 mmHg for blood pressure between treatment A and treatment B in a 50 patients study is barely statistically significant (P-value 0.0492). If recruitment had stopped at 40 patients, the difference would not have been statistically significant.
How Big of an Effect Is It?
The number needed to treat (NNT) facilitates judging the impact of a study’s findings. NNT represents the reciprocal of absolute risk reduction and reveals the number of patients that in clinical practice should be treated to see the desired outcome.
Thank you for your attention
There is now time for all your questions!
Reference:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3380258/