How to Read a Paper: The Basics of Evidence-Based Medicine (9 page)

Secondary research is composed of the following

• Overviews
  , which are considered in Chapter 9, may be divided into
  

  a.
  (non-systematic) reviews
  , which summarise primary studies;
  

  b.
  systematic reviews
  , which do this using a rigorous, transparent and auditable (i.e. checkable) method;
  

  c.
  meta-analyses
  , which integrate the numerical data from more than one study;
  

• Guidelines
  , which are considered in Chapter 10, draw conclusions from primary studies about how clinicians should be behaving;
  
• Decision analyses
  , which are not discussed in detail in this book but are covered elsewhere [6], use the results of primary studies to generate probability trees to be used by both health professionals and patients in making choices about clinical management;
  
• Economic analyses
  , which are considered briefly in Chapter 12 and in more detail elsewhere [7], use the results of primary studies to say whether a particular course of action is a good use of resources.
  

Question Three: Was the research design appropriate to the question?

Examples of the sort of questions that can reasonably be answered by different types of primary research study are given in the sections that follow. One question that frequently cries out to be asked is this: was a randomised controlled trial (RCT) (see section ‘Randomised controlled trials’) the best method of addressing this particular research question, and if the study was not an RCT, should it have been? Before you jump to any conclusions, decide what broad field of research the study covers (see Box 3.2). Once you have done this, ask whether the study design was appropriate to this question. For more help on this task (which people often find difficult until they get the hang of it) see the Oxford Centre for Evidence-Based Medicine (EBM) website (www.cebm.ox.ac.uk).

Box 3.2 Broad fields of research

Most quantitative studies are concerned with one or more of the following:

• Therapy
  : testing the efficacy of drug treatments, surgical procedures, alternative methods of service delivery or other interventions. Preferred study design is randomised controlled trial (see section ‘Randomised controlled trials’ and Chapters 6 and 7).
  
• Diagnosis
  : demonstrating whether a new diagnostic test is valid (can we trust it?) and reliable (would we get the same results every time?). Preferred study design is cross-sectional survey (see section ‘Cross-sectional surveys’ and Chapter 8).
  
• Screening
  : demonstrating the value of tests that can be applied to large populations and that pick up disease at a pre-symptomatic stage. Preferred study design is cross-sectional survey (see section ‘Cross-sectional surveys’ and Chapter 8).
  
• Prognosis
  : determining what is likely to happen to someone whose disease is picked up at an early stage. Preferred study design is longitudinal survey (see section ‘Cross-sectional surveys’).
  
• Causation
  : determining whether a putative harmful agent, such as environmental pollution, is related to the development of illness. Preferred study design is cohort or case–control study, depending on how rare the disease is (see sections ‘Cross-sectional surveys’ and ‘Case reports’), but case reports (see section ‘The traditional hierarchy of evidence’) may also provide crucial information.
  
• Psychometric studies
  : measuring attitudes, beliefs or preferences, often about the nature of illness or its treatment.
  

Qualitative studies are discussed in Chapter 12.

Randomised controlled trials

In an RCT, participants in the trial are randomly allocated by a process equivalent to the flip of a coin to either one intervention (such as a drug treatment) or another (such as placebo treatment—or more commonly, best current therapy). Both groups are followed up for a pre-specified time period and analysed in terms of specific outcomes defined at the outset of the study (e.g. death, heart attack, and serum cholesterol level). Because,
on average
, the groups are identical apart from the intervention, any differences in outcome are, in theory, attributable to the intervention. In reality, however, not every RCT is a bowl of cherries.

Some papers that report trials comparing an intervention with a control group are not, in fact, randomised trials at all. The terminology for these is
other controlled clinical trials
—a term used to describe comparative studies in which participants were allocated to intervention or control groups in a non-random manner. This situation may arise, for example, when random allocation would be impossible, impractical or unethical—for example, when patients on ward A receive one diet while those on ward B receive a different diet. (Although this design is inferior to the RCT, it is much easier to execute, and was used successfully a century ago to demonstrate the benefit of brown rice over white rice in the treatment of beriberi [8].) The problems of non-random allocation are discussed further in section ‘Was systematic bias avoided or minimised?’ in relation to determining whether the two groups in a trial can reasonably be compared with one another on a statistical level.

Some trials count as a sort of halfway house between true randomised trials and non-randomised trials. In these, randomisation is not performed truly at random (e.g. using sequentially numbered sealed envelopes each with a computer-generated random number inside), but by some method that allows the clinician to know which group the patient would be in
before he or she makes a definitive decision to randomise the patient
. This allows subtle biases to creep in, as the clinician might be more (or less) likely to enter a particular patient into the trial if he or she believed that this individual would get active treatment. In particular, patients with more severe disease may be subconsciously withheld from the placebo arm of the trial. Examples of unacceptable methods include randomisation by last digit of date of birth (even numbers to group A, odds to group B), toss of a coin (heads to group A, tails to group B), sequential allocation (patient A to group 1; patient B to group 2, etc.) and date seen in clinic (all patients seen this week to group A; all those seen next week to group 2, etc.) (Box 3.3) [9] [10].

Listed here are examples of clinical questions that would be best answered by an RCT, but note also the examples in the later sections of this chapter of situations where other types of studies could or must be used instead.

• Is this drug better than a placebo or a different drug for a particular disease?
  
• Is a new surgical procedure better than the currently favoured practice?
  
• Is an online decision support algorithm better than verbal advice in helping patients make informed choices about the treatment options for a particular condition?
  
• Will changing from a diet high in saturated fats to one high in polyunsaturated fats significantly affect serum cholesterol levels?
  

Box 3.3 Advantages of the randomised controlled trial design

1.
Allows rigorous evaluation of a single variable (e.g. effect of drug treatment versus placebo) in a precisely defined patient group (e.g. post-menopausal women aged 50–60 years).

2.
Prospective design (i.e. data are collected on events which happen
after
you decide to do the study).

3.
Uses hypotheticodeductive reasoning (i.e. seeks to falsify, rather than confirm, its own hypothesis; see section ‘Three preliminary questions to get your bearings’).

4.
Potentially eradicates bias by comparing two otherwise identical groups (but see subsequent text and section ‘Was systematic bias avoided or minimised?’).

5.
Allows for meta-analysis (combining the numerical results of several similar trials) at a later date; see section ‘Ten questions to ask about a paper that claims to validate a diagnostic or screening test’).

RCTs are often said to be the gold standard in medical research. Up to a point, this is true (see section ‘The traditional hierarchy of evidence’), but only for certain types of clinical questions (see Box 3.2 and sections ‘Cohort studies’, ‘Case-control studies’, ‘Cross-sectional surveys’ and ‘Case reports’). The questions that best lend themselves to the RCT design all relate to
interventions
, and are mainly concerned with therapy or prevention. It should be remembered, however, that even when we are looking at therapeutic interventions, and especially when we are not, there are a number of important disadvantages associated with randomised trials (see Box 3.4) [11] [12].

Remember, too, that the results of an RCT may have limited applicability as a result of exclusion criteria (rules about who may not be entered into the study), inclusion bias (selection of trial participants from a group that is unrepresentative of everyone with the condition (see section ‘Whom is the study about?’)), refusal (or inability) of certain patient groups to give consent to be included in the trial, analysis of only pre-defined ‘objective’ endpoints which may exclude important qualitative aspects of the intervention (see Chapter 12) and publication bias (i.e. the selective publication of positive results, often but not always because the organisation that funded the research stands to gain or lose depending on the findings [9] [10]). Furthermore, RCTs can be well or badly managed [2], and, once published, their results are open to distortion by an over-enthusiastic scientific community or by a public eager for a new wonder drug [13]. While all these problems might also occur with other trial designs, they may be particularly pertinent when an RCT is being sold to you as, methodologically speaking, whiter than white.

Box 3.4 Disadvantages of the randomised controlled trial design

Expensive and time-consuming, hence, in practice,

• many RCTs are either never carried out, are performed on too few patients or are undertaken for too short a period (see section ‘Were preliminary statistical questions addressed?’);
  
• most RCTs are funded by large research bodies (university or government-sponsored) or drug companies, who ultimately dictate the research agenda;
  
• surrogate endpoints may not reflect outcomes that are important to patients (see section ‘Surrogate endpoints’).
  

May introduce ‘hidden bias’, especially through

• imperfect randomisation (see preceding text);
  
• failure to randomise all eligible patients (clinician only offers participation in the trial to patients she/he considers will respond well to the intervention);
  
• failure to blind assessors to randomisation status of patients (see section ‘Was assessment ‘blind’?’).
  

There are, in addition, many situations in which RCTs are unnecessary, impractical or inappropriate:

RCTs are unnecessary

• when a clearly successful intervention for an otherwise fatal condition is discovered;
  
• when a previous RCT or meta-analysis has given a definitive result (either positive or negative—see section ‘Probability and confidence’). Arguably, it is actually
  unethical
  to ask patients to be randomised to a clinical trial without first conducting a systematic literature review to see whether the trial needs to be carried out at all.
  

RCTs are impractical

• where it would be unethical to seek consent to randomise (see section ‘A note on ethical considerations’);
  
• where the number of participants needed to demonstrate a significant difference between the groups is prohibitively high (see section ‘Were preliminary statistical questions addressed?’);
  

RCTs are inappropriate

• where the study is looking at the prognosis of a disease. For this analysis, the appropriate route to best evidence is a longitudinal survey of a properly assembled
  inception cohort
  (see section ‘Cross-sectional surveys’);
  
• where the study is looking at the validity of a diagnostic or screening test. For this analysis, the appropriate route to best evidence is a
  cross-sectional survey
  of patients clinically suspected of harbouring the relevant disorder (see section ‘Cross-sectional surveys’ and Chapter 7);
  
• where the study is looking at a ‘quality of care’ issue in which the criteria for ‘success’ have not yet been established. For example, an RCT comparing medical versus surgical methods of abortion might assess ‘success’ in terms of number of patients achieving complete evacuation, amount of bleeding and pain level. The patients, however, might decide that other aspects of the procedure are important, such as knowing in advance how long the procedure will take, not seeing or feeling the abortus come out, and so on. For this analysis, the appropriate route to best evidence is
  qualitative research methods
  (see Chapter 12).
  

All these issues have been discussed in great depth by clinical epidemiologists, who remind us that to turn our noses up at the non-randomised trial may indicate scientific naiveté and not, as many people routinely assume, intellectual rigour [11]. You might also like to look up the emerging science of
pragmatic
RCTs—a methodology for taking account of practical, real-world challenges so that the findings of your trial will be more relevant to that real world when the trial is finished [14]. See also section ‘What information to expect in a paper describing a randomised controlled trial: the CONSORT statement’ where I introduce the Consolidated Standards of Reporting Trials (CONSORT) statement for presenting the findings of RCTs.