Read Core Topics in General & Emergency Surgery: Companion to Specialist Surgical Practice Online
Authors: Simon Paterson-Brown MBBS MPhil MS FRCS
Core Topics in General & Emergency Surgery: Companion to Specialist Surgical Practice (5 page)
A systematic review is an overview of primary studies carried out to an exhaustive, defined and repeatable protocol.
There has been an explosion in the published medical literature, with over two million articles a year published in 20 000 journals. The task of keeping up with new advances in medical research has become quite overwhelming. We have also seen that the results of trials in the same subject may be contradictory, and that the underlying message can be masked. Systematic reviews are designed to search out meticulously all relevant studies on a subject, evaluate the quality of each study and assimilate the information to produce a balanced and unbiased conclusion.
22
One advantage of a systematic review with a meta-analysis over a traditional subjective narrative review is that by synthesising the results of many smaller studies, the original lack of statistical power of each study may be overcome by cumulative size, and any treatment effect is more clearly demonstrated. This, in turn, can lead to a reduction in delay between research advances and clinical implementation. For example, it has been demonstrated that if the original studies done on the use of anticoagulants after myocardial infarction had been reviewed, their benefits would have been apparent much earlier.
23
,
24
It is obviously essential that both the benefit or any harm caused by an intervention becomes apparent as soon as possible.
Unfortunately, as in reported trials, not all reviews are as rigorously researched and synthesised as one would hope and are open to similar pitfalls as RCTs. The Cochrane Collaboration has sought to rectify this and has worked upon refining the methods used for systematic reviews. It has consequently produced some of the most reliable and useful reviews, and its methods have been widely adopted by other reviewers. The Cochrane Collaboration advises that each review must be based on an explicit protocol, which sets out the objectives and methods so that a second party could reproduce the review at a later date if required.
Because of the increasing importance of systematic reviews as a method of providing the evidence base for a variety of clinical activities, the methods are discussed in some detail below. There are several key elements in producing a systematic review.
Develop a protocol for a clearly defined question
Within a protocol:
the objectives of the review of the RCTs must be stated;
eligibility criteria must be included (e.g. relevant patient groups, types of intervention and trial design);
appropriate outcome measures should be defined.
In the Cochrane Collaboration, each systematic review is preceded by a published protocol that is subjected to a process of peer review. This helps to ensure high quality, avoids duplication of effort and is designed to reduce bias by setting standards for inclusion criteria before the results from identified studies have been assessed.
Literature search
All published and unpublished material should be sought. This includes examining studies in non-English journals, grey literature, conference reports, company reports (drug companies can hold a lot of vital information from their own research) and any personal contacts, for personal studies or information. The details of the search methodology and search terms used should be specified in order to make the review reproducible and allow readers to repeat the search to identify further relevant information published after the review. The most frequently used initial source of information is MEDLINE but this does have limitations. It only indexes about one-third of all medical articles that exist in libraries (over 10 million in total),
25
and an average search by a regular user would only yield about one-fifth of the trials that can be identified by more rigorous techniques for literature searching.
26
It also has a bias towards articles published in English. Other electronic and indexed databases should also be searched, but often the only way to ensure that the maximum number of relevant trials are found, wherever published and in whatever language, is to hand search the journals. This is one of the tasks of the Cochrane Collaboration through a database maintained at the Baltimore Cochrane Centre.
One must also be aware, however, that there is a potential for ‘publication bias’. Trials that are more likely to get published are those with a positive result rather than a negative or no-effect result,
27
and are also more likely to be cited in other articles.
28
Evaluating the studies
Each trial should be assessed to see if it meets the inclusion criteria set out in the protocol (eligibility). If it meets the required standards, then the trial is subjected to a critical appraisal, ideally by two independent reviewers, to ascertain its validity, relevance and reliability. Any exclusions should be reported and justified; if there is missing information from the published article, it may be necessary to attempt to contact the author of the primary research. Reviewers should also, if possible, be ‘blinded’ to the authors and journals of publication, etc. in order to minimise any personal bias.
The Cochrane reviewers are assisted in all these tasks by instructions in the Cochrane Handbook
14
and through workshops at the Cochrane Centres.
29
Synthesis of the results
Once the studies have been graded according to quality and relevance, their results may be combined in an interpretative or a statistical fashion. It must be decided if it is appropriate to combine some studies and which comparisons to make. Subgroup or sensitivity analyses may also be appropriate. The statistical analysis is called a meta-analysis and is discussed below.
Discussion
The review should be summarised. The aims, methods and reported results should be discussed and the following issues considered:
quality of the studies;
possible sources of heterogeneity (reasons for inconsistency between studies, e.g. patient selection, methods of randomisation, duration of follow-up or differences in statistical analysis);
bias;
chance;
applicability of the findings.
As with any study, a review can be done badly, and the reader must critically appraise a review to assess its quality. Systematic errors may be introduced by omitting some relevant studies, by selection bias (such as excluding foreign language journals) or by including inappropriate studies (such as those considering different patient groups or irrelevant outcomes). Despite all precautions, the findings of a systematic review may differ from those of a large-scale, high-quality RCT. This will be discussed below in relation to meta-analysis.
A meta-analysis is a specific statistical strategy for assembling the results of several studies into a single estimate, which may be incorporated into a systematic literature review.
30
Here we must make the distinction that the term ‘meta-analysis’ refers to the statistical techniques used to combine the results of several studies and is not synonymous with systematic review, as it is sometimes used.
A common problem in clinical trials is that the results are not clear-cut, either because of size or because of the design of the trial. The systematic review is designed to eliminate some of these problems and give appropriate weightings to the best- and worst-quality studies, regardless of size. Meta-analysis is the statistical tool used to combine the results and give ‘power’ to the estimates of effect.
Meta-analyses use a variety of statistical techniques according to the type of data being analysed (dichotomous, continuous or individual patient data).
14
There are two main models used to analyse the results: the fixed-effect model (logistic regression, Mantel–Haenszel test and Peto's method) and the random-effect model. The major concern with fixed-effect methods is that they assume no clinical heterogeneity between the individual trials, and this may be unrealistic.
31
The random-effect method takes into consideration random variation and clinical heterogeneity between trials. In the presentation of meta-analysis, a consistent scale should be chosen for measuring treatment effects and to cope with the possible large scale of difference in proportions, risk ratios or odds ratios that can be used.
Trials can have many different components
21
and therefore a meta-analysis is only valid if the trials that it seeks to summarise are homogeneous: you cannot add apples and oranges.
32
If trials are not comparable and any heterogeneity is ignored, the analysis can produce misleading results.
Figure 1.1
shows an example of this from a meta-analysis of 19 RCTs investigating the use of endoscopic sclerotherapy to reduce mortality from oesophageal varices in the primary treatment of cirrhotic patients.
33
Each trial is represented by a ‘point estimate’ of the difference between the groups, and a horizontal line showing the 95% confidence interval (CI). If the line does not cross the line of no effect, then there is a 95% chance that there is a real difference between the groups. It can be seen that in this case the trials are not homogeneous as some of the lower limits of the CIs are above the highest limits of CIs in other trials. Such a lack of homogeneity may have a variety of causes, relating to clinical heterogeneity (differences in patient mix, setting, etc.) or differences in methods. The degree of statistical heterogeneity can be measured to see if it is greater than is compatible with the play of chance.
34
Such a statistical tool may lack statistical power; consequently, results that do not show significant heterogeneity do not necessarily mean that the trials are truly homogeneous and one must look beyond them to assess the degree of heterogeneity.
‘Meta-analysis is on the strongest ground when the methods employed in the primary studies are sufficiently similar that any differences in their results are due to the play of chance.’
30
Figure 1.1
An example of a meta-analysis of 19 randomised controlled trials investigating the use of endoscopic sclerotherapy to reduce mortality from oesophageal varices in the primary treatment of cirrhotic patients.
Reproduced from Chalmer I, Altman DG. Systematic reviews. London: BMJ Publishing, 1995; p. 119. With permission from Blackwell Publishing Ltd.
Views on the usefulness of meta-analyses are divided. On the one hand, they may provide conclusions that could not be reached from other trials because of the small numbers involved. However, on the other hand, they have some limitations and cannot produce a single simple answer to all complex clinical problems. They may give misleading results if used inappropriately where there is a biased body of literature or clinical or methodological heterogeneity. If used with caution, however, they may be a useful tool in providing information to help in decision-making.
Figure 1.2
shows a funnel plot of a meta-analysis relating to the use of magnesium following myocardial infarction.
35
The result of each study in the analysis is represented by a circle plotting the odds ratio (with the vertical line being at 1, the ‘line of no effect’) against the trial size. The diamond represents the overall results of the meta-analysis with its pooled data from all the smaller studies shown. This study
36
was published in 1993 and showed that it was beneficial and safe to give intravenous magnesium in patients with acute myocardial infarction. The majority of the studies involved show a positive effect of the treatment, as does the meta-analysis. However, the results from this study were contradicted in 1995 by ISIS-4, a very large RCT involving 58 050 patients.
37
It had three arms, in one of which intravenous magnesium was given to patients suspected of an acute myocardial infarction. The results are marked on the funnel plot and show that there is no clear benefit for this treatment, contrary to the results of the earlier meta-analysis.
Figure 1.2
A funnel plot of a meta-analysis relating to the use of magnesium following myocardial infarction. Points indicate values from small and medium-sized trials; the diamond is the combined odds ratio with 95% confidence interval from the meta-analysis of these trials and the square is that for a mega trial.
Reproduced from Egger M, Smith GD. Misleading meta-analysis. Br Med J 1995; 310:752–4. With permission from the BMJ Publishing Group Ltd.
Some would say that this is one of the major problems with using statistical synthesis. An alternative viewpoint is that it is an example of the importance of ensuring that the material fed into a meta-analysis from a systematic review is researched and critically appraised to the highest possible standard. Explanations for the contradictory findings in this review have been given as:
32
,
35
publication bias, since only trials with positive results were included (see funnel plot);
methodological weakness in the small trials;
clinical heterogeneity.
