How to Read a Paper: The Basics of Evidence-Based Medicine (33 page)

Question Two: Whose viewpoint are costs and benefits being considered from?

From the patient's point of view, he or she generally wants to get better as quickly as possible. From the Treasury's point of view, the most cost-effective health intervention is one that returns all citizens promptly to taxpayer status and, when this status is no longer tenable, causes immediate sudden death. From the drug company's point of view, it would be difficult to imagine a cost–benefit equation that did not contain one of the company's products, and from a physiotherapist's point of view, the removal of a physiotherapy service would never be cost-effective. There is no such thing as an economic analysis that is devoid of perspective. Most assume the perspective of the health care system itself, although some take into account the hidden costs to the patient and society (e.g. as a result of work days lost). There is no ‘right’ perspective for an economic evaluation—but the paper should say clearly whose costs and whose benefits have been counted ‘in’ and ‘out’.

Question Three: Have the interventions being compared been shown to be clinically effective?

Nobody wants inexpensive treatment if it doesn't work. The paper you are reading may simply be an economic analysis, in which case it will be based on a previously published clinical trial, or it will be an economic evaluation of a new trial whose clinical results are presented in the same paper. Either way, you must make sure that the intervention that ‘works out cheaper’ is not substantially less effective in clinical terms than the one that stands to be rejected on the grounds of cost. (Note, however, that in a resource-limited health care system, it is often sensible to use treatments that are a little less effective when they are a lot less expensive than the best on offer!).

Question Four: Are the interventions sensible and workable in the settings where they are likely to be applied?

A research trial that compares one obscure and unaffordable intervention with another will have little impact on medical practice. Remember that standard current practice (which may be ‘doing nothing’) should almost certainly be one of the alternatives compared. Too many research trials look at intervention packages that would be impossible to implement in the non-research setting (e.g. they assume that general practitioners will own a state-of-the-art computer and agree to follow a protocol, that infinite nurse time is available for the taking of blood tests or that patients will make their personal treatment choices solely on the basis of the trial's primary outcome measure).

Question Five: Which method of analysis was used, and was this appropriate?

This decision can be summarised as given here (see section ‘How can we help ensure that evidence-based guidelines are followed?’).

a.
If the interventions produced identical outcomes ⇒ cost-minimisation analysis.

b.
If the important outcome is unidimensional ⇒ cost-effectiveness analysis.

c.
If the important outcome is multi-dimensional ⇒ cost–utility analysis.

d.
If the outcomes can be expressed meaningfully in monetary terms (i.e. if it is possible to weigh the cost–benefit equation for this condition against the cost–benefit equation for another condition) ⇒ cost–benefit analysis.

e.
If a cost–benefit analysis would otherwise be appropriate but the preference values given to different health states are disputed or likely to change ⇒ cost–consequences analysis.

Question Six: How were costs and benefits measured?

Look back at section ‘How can we help ensure that evidence-based guidelines are followed?’, where I outlined some of the costs associated with my appendix operation. Now imagine a more complicated example—the rehabilitation of stroke patients into their own homes with attendance at a day centre compared with a standard alternative intervention (rehabilitation in a long-stay hospital). The economic analysis must take into account not just the time of the various professionals involved, the time of the secretaries and administrators who help run the service, and the cost of the food and drugs consumed by the stroke patients but also a fraction of the capital cost of building the day centre and maintaining a transport service to and from it.

There are no hard and fast rules for deciding which costs to include. If calculating ‘cost per case’ from first principles, remember that someone has to pay for heating, lighting, personnel support and even the accountants' bills of the institution. In general terms, these ‘hidden costs’ are known as overheads, and generally add an additional 30–60% onto the cost of a project. The task of costing things like operations and outpatient visits in the UK is easier than it used to be because these experiences are now bought and sold at a price that reflects (or should reflect) all overheads involved. Be warned, however, that unit costs of health interventions calculated in one country often bear no relation to those of the same intervention elsewhere, even when these costs are expressed as a proportion of GNP.

Benefits such as earlier return to work for a particular individual can, on the face of it, be measured in terms of the cost of employing that person at his or her usual daily rate. This approach has the unfortunate and politically unacceptable consequence of valuing the health of professional people higher than that of manual workers, homemakers or the unemployed, and that of the white majority higher than that of (generally) lower paid minority ethnic groups. It might therefore be preferable to derive the cost of sick days from the average national wage.

In a cost-effectiveness analysis, changes in health status will be expressed in natural units (see section ‘How can we help ensure that evidence-based guidelines are followed?’). But just because the units are natural does not automatically make them appropriate. For example, the economic analysis of the treatment of peptic ulcer by two different drugs might measure outcome as ‘proportion of ulcers healed after a 6-week course’. Treatments could be compared according to the cost per ulcer healed. However, if the relapse rates on the two drugs were very different, drug A might be falsely deemed ‘more cost-effective’ than drug B. A better outcome measure here might be ‘ulcers which remained healed at one year’.

In cost–benefit analysis, where health status is expressed in utility units, such as QALYs, you would, if you were being really rigorous about evaluating the paper, look back at how the particular utilities used in the analysis were derived (see section ‘How can we help ensure that evidence-based guidelines are followed?’). In particular, you will want to know whose health preference values were used—those of patients, doctors, health economists or the government.

Question Seven: Were incremental, rather than absolute, benefits considered?

This question is best illustrated by a simple example. Let's say drug X, at £100 per course, cures 10 out of every 20 patients. Its new competitor, drug Y, costs £120 per course and cures 11 out of 20 patients. The cost per case cured with drug X is £200 (because you spent £2000 curing 10 people), and the cost per case cured with drug Y is £218 (because you spent £2400 curing 11 people).

The
incremental
cost of drug Y—that is, the extra cost of curing the extra patient—is NOT £18 but £400, as this is the total amount extra that you have had to pay to achieve an outcome over and above what you would have achieved by giving all patients the cheaper drug. This striking example should be borne in mind the next time a pharmaceutical representative tries to persuade you that his or her product is ‘more effective and only marginally more expensive’.

Question Eight: Was the ‘here and now’ given precedence over the distant future?

A bird in the hand is worth two in the bush. In health as well as money terms, we value a benefit today more highly than we value a promise of the same benefit in 5 years' time. When the costs or benefits of an intervention (or lack of the intervention) will occur sometime in the future, their value should be
discounted
to reflect this. The actual amount of discount that should be allowed for future, as opposed to immediate, health benefit, is pretty arbitrary, but most analyses use a figure of around 5% per year.

Question Nine: Was a sensitivity analysis performed?

Let's say a cost–benefit analysis comes out as saying that hernia repair by day-case surgery costs £1500 per QALY, whereas traditional open repair, with its associated hospital stay, costs £2100 per QALY. But, when you look at how the calculations were performed, you are surprised at how cheaply the laparoscopic equipment has been costed. If you raise the price of this equipment by 25%, does day-case surgery still come out dramatically cheaper? It may, or it may not.

Sensitivity analysis, or exploration of ‘what-ifs’, was described in section ‘Validating diagnostic tests against a gold standard’ in relation to meta-analysis. Exactly the same principles apply here: if adjusting the figures to account for the full range of possible influences gives you a totally different answer, you should not place too much reliance on the analysis. For a good example of a sensitivity analysis on a topic of both scientific and political importance, see this paper on the cost-effectiveness of statin therapy in people with different levels of baseline risk for cardiovascular disease [11].

Question Ten: Were “bottom line” aggregate scores overused?

In section ‘How can we help ensure that evidence-based guidelines are followed?’, I introduced the notion of cost–consequences analysis, in which the reader of the paper can attach his or her own values to different utilities. In practice, this is an unusual way of presenting an economic analysis, and, more commonly, the reader is faced with a cost–utility or cost–benefit analysis that gives a composite score in unfamiliar units which do not translate readily into exactly what gains and losses the patient can expect. The situation is analogous to the father who is told, ‘your child’s intelligence quotient is 115', when he would feel far better informed if he were presented with the disaggregated data: ‘Johnny can read, write, count, and draw pretty well for his age’.

Conclusion

I hope this chapter has shown that the critical appraisal of an economic analysis rests as crucially on asking questions such as, ‘where did those numbers come from?’ and ‘have any numbers been left out?’ as on checking that the sums themselves were correct. Whilst few papers will fulfil all the criteria listed in section ‘Ten questions to ask about an economic analysis’ and summarised in Appendix 1, you should, after reading the chapter, be able to distinguish an economic analysis of moderate or good methodological quality from one which slips ‘throwaway costings’ (‘drug X costs less than drug Y; therefore it is more cost-effective’) into its results or discussion section.

References

1
Drummond M, Jefferson T. Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party.
BMJ: British Medical Journal
1996;
313
(7052):275.

2
Jefferson T, Demicheli V, Mugford M.
Elementary economic evaluation in health care
. London: BMJ Books, 2000.

3
Donaldson C, Mitton C.
Priority setting toolkit: guide to the use of economics in healthcare decision making
. Oxford, John Wiley & Sons, 2009.

4
McDowell I, Newell C, McDowell I.
Measuring health: a guide to rating scales and questionnaires
. New York: Oxford University Press, 2006.

5
Bradley C, Speight J. Patient perceptions of diabetes and diabetes therapy: assessing quality of life.
Diabetes/Metabolism Research and Reviews
2002;
18
(S3):S64–9.

6
Bache I. Measuring quality of life for public policy: an idea whose time has come? Agenda-setting dynamics in the European Union.
Journal of European Public Policy
2013;
20
(1):21–38.

7
Fairclough DL.
Design and analysis of quality of life studies in clinical trials
. Boca Raton: CRC Press, 2010.

8
Phillips D.
Quality of life: concept, policy and practice
. Oxon: Routledge, 2012.

9
Young T, Yang Y, Brazier JE, et al. The first stage of developing preference-based measures: constructing a health-state classification using Rasch analysis.
Quality of Life Research
2009;
18
(2):253–65.

10
Henderson C, Knapp M, Fernández J-L, et al. Cost effectiveness of telehealth for patients with long term conditions (Whole Systems Demonstrator telehealth questionnaire study): nested economic evaluation in a pragmatic, cluster randomised controlled trial.
BMJ: British Medical Journal
2013;346:f1035.

11
Jha V, Modi GK. Cardiovascular disease: the price of a QALY—cost-effectiveness of statins in CKD.
Nature Reviews Nephrology
2013;
9
:377–9.