Introduction

This chapter briefly discusses cancer clinical trials and systematic reviews of the cancer literature, together with the concept of critical appraisal and evidence-based medicine (EBM). Much of literature appraisal requires a common sense, but systematic, approach to ascertaining answers to key questions about potential biases in the design, conduct and reporting of a study or review.

Appraising reports of cancer clinical trials

The difficulty in appraising clinical trials lies in assessing the strength of the evidence. On cursory reading the author of the report may seem to present a strong case for the trial, but you need to see beyond the rhetoric, the eminence of the author or institution/trials group and the status of the journal publishing the paper, to assess the evidence and to see if it is strong enough to stand on its own. Very strong evidence may be enough to influence practice by itself, but more commonly the evidence is weaker and needs support from further studies.

What are the important elements when assessing reports of trials?

By far the most important elements are the design and conduct of the trial. The methods of analysis, while still important, are generally less likely to lead to inappropriate conclusions than inappropriate design and management of a trial. This chapter is based on the methods recommended by various EBM groups (Box 26.1).

Because there is such a huge volume of medical literature published each year, it is important to take a systematic approach to what you appraise thoroughly (Box 26.2). The first step in appraisal is to screen the paper to see if it is worthy of careful reading. It may be possible to answer these screening questions on reading the title and abstract.

Step 2 – Appraising a paper reporting a trial

There are a number of crucial questions that need to be answered (Box 26.2):

1 Was there an appropriate comparison group?

‘Though most of the facts were familiar to me, I had not sufficiently appreciated their relative importance, nor their connection to each other.’

Watson in Silver Blaze

The main aim is that the control group be identical to the intervention group in everything apart for the intervention itself.

Concurrent or historical controls?

When a new therapy is being tested some investigators will give the new treatment to all the patients and will then compare the outcomes with those obtained from the records of similar patients treated in the past – so-called historical controls. Any such comparison is fraught with danger as factors other than the new treatment may have change over time.

Did the authors use randomization?

The purpose of randomization is to ensure, as far as possible, that all factors (known and unknown) that may influence the treatment outcome are balanced between the treatment groups. This is done to reduce the risk of a chance bias. Randomization requires the use of a random device; this is normally a table of random numbers. Systematic allocation; i.e., alternating between treatments, odd or even birth date or hospital number is not an acceptable method, though it is sometimes referred to as pseudo-randomization. Since the researchers know which treatment each person is allocated to before they consent, selective allocation can occur, which will skew the results. Hence the process used for randomization must ensure that neither the trial subjects nor the investigator can influence the treatment arm each person ends up in (‘allocation concealment’).

Despite their pre-eminence, RCTs do, however, have their limitations. They are usually carried out in a narrowly defined research setting. Protocols require careful selection of patients with specific characteristics. There may be a proportion of patient eligible who chooses not to participate. Because of these factors, patients in an RCT may not be typical of patients in routine health practice.

2 Was the study based on a pre-specified protocol?

A protocol written before the trial starts is a prerequisite to good research and some journals are now recommending that the original protocol be submitted with the final paper to ensure that there was such a protocol and to identify any deviations from the original design. Any deviation from the original study design can result in skewed observation of clinical benefit.

When reading a paper bear in mind the following elements of trial design:

Was there a clearly defined research question?

A good study has limited objectives that are based on a biological hypothesis. Studies with no focus and without a clearly defined question are in danger of becoming a ‘fishing expedition’ with multiple comparisons.

The chances of multiple testing increases when there is numerous subgroups or endpoints or when data are reported on multiple side effects. This is particularly problematic when such comparisons are unplanned.

Did the study have appropriate end points?

Most RCTs in cancer therapy concentrate on reporting response rates (a surrogate end point of clinical benefit), disease free survival and overall survival. However, accepting that cure is out of the reach of many patients with cancer, information on symptom control and quality of life are often of major importance; they are frequently ignored in the primary research.

Optimal cut points and the problem with multiple comparisons

‘While the individual man is an insoluble problem, in aggregate he becomes a mathematical certainty. For example you can never foretell what any one man will do, but you can say with some precision what an average number will be up to. Individuals vary, but percentages remain constant.’

Sherlock Holmes, in The Sign of The Four

Where there is a continuous outcome measure, it is common to simplify analysis by defining one or more points that are used to characterize risk. The way that these points are selected may have a major effect on the analysis.

Subgroup analysis

Subgroup analyses are fairly common in trials; they use the data from a study to compare one endpoint across different subgroups. There are three major problems with this approach:

• The comparison is not randomized.

• The comparison is less precise as there are numbers of patients and those reaching end points is smaller.

• The power calculations for the size in a study may be invalidated by a large number of subgroups.

Where a subgroup is found to behave differently, you should consider if there is a plausible biological mechanism for this and whether other trials have found a similar finding. Where there is a statistical analysis, this should be done as a formal test of interaction. Strong empirical evidence suggests that post hoc subgroup analyses often lead to false positive results (Example Box 26.2).

Example Box 26.2
International study of infarct survival

Sleight P. Subgroup analyses in clinical trials: fun to look at – but don’t believe them! Curr Control Trials Cardiovasc Med 2000;1(1):25–27.

Analysis of subgroup results in a clinical trial is surprisingly unreliable, even in a large trial. This is due to a combination of reduced statistical power, increased variance and the play of chance. Reliance on such analyses is likely to be erroneous. Plausible explanations can usually be found for effects that are, in reality, simply due to the play of chance. When clinicians believe such subgroup analyses, there is a real danger of harm to the individual patient.

In order to study the effect of examining subgroups the investigators of the ISIS trial, testing the value of aspirin and streptokinase after MI, analysed the results by astrological star sign. All of the patients had their date of birth entered as an important ‘identifier’. They divided population into 12 subgroups by astrological star sign. Even in a highly positive trial such as ISIS-2, in which the overall statistical benefit for aspirin over placebo was extreme (p < 0.00001), division into only 12 subgroups threw up two (Gemini and Libra) for which aspirin had a non-significantly adverse effect (9% ± 13%).

ISIS-2 was carried out in 16 countries. For the streptokinase randomization, two countries had non-significantly negative results, and a single (different) country was non-significantly negative for aspirin.

There is no plausible explanation for such findings except for the entirely expected operation of the statistical play of chance. It is very important to realize that lack of a statistically significant effect is not evidence of lack of a real effect.

3 Did the investigators stick to the protocol?

Clinical research is rarely predictable, so changes or alterations to a protocol are often needed. However, they should be clearly reported, as major deviations may reduce the reliability of the data.

Only gold members can continue reading. Log In or Register to continue

Share this:

• Click to share on Twitter (Opens in new window)
• Click to share on Facebook (Opens in new window)

Related

Related posts:

Palliative care Management of common radiotherapy side effects Cancer in pregnancy Breast cancer Principles of systemic therapy Cancers of the musculoskeletal system

Stay updated, free articles. Join our Telegram channel

Join