How to Read a Paper: The Basics of Evidence-Based Medicine (35 page)

Evaluating papers that describe qualitative research

By its very nature, qualitative research is non-standard, unconfined and dependent on the subjective experience of both the researcher and the researched. It explores what needs to be explored and cuts its cloth accordingly. As implied in the previous section, qualitative research is an in-depth, interpretive task, not a technical procedure. It depends crucially on a competent and experienced researcher exercising the kind of skills and judgements that are difficult, if not impossible, to measure objectively. It is debatable, therefore, whether an all-encompassing critical appraisal checklist along the lines of the Users' Guides to the Medical Literature for quantitative research could ever be developed, although valiant attempts have been made [3] [4, 10] [13]. Some people have argued that critical appraisal checklists potentially detract from research quality in qualitative research because they encourage a mechanistic and protocol-driven approach [14].

My own view, and that of a number of individuals who have attempted, or are currently working on, this very task, is that such a checklist may not be as exhaustive or as universally applicable as the various guides for appraising quantitative research, but that it is certainly possible to set some ground rules. Without doubt, the best attempt to offer guidance (and also the best exposition of the uncertainties and unknowables) has been made by Dixon-Woods and her colleagues [15]. The list that follows has been distilled from the published work cited elsewhere in this chapter, and also from discussions many years ago with Dr Rod Taylor, who produced one of the earliest critical appraisal guides for qualitative papers.

Question One: Did the paper describe an important clinical problem addressed via a clearly formulated question?

In section ‘Three preliminary questions to get your bearings’, I explained that one of the first things you should look for in any research paper is a statement of why the research was carried out and what specific question it addressed. Qualitative papers are no exception to this rule: there is absolutely no scientific value in interviewing or observing people just for the sake of it. Papers which cannot define their topic of research more closely than ‘we decided to interview 20 patients with epilepsy’ inspire little confidence that the researchers really knew what they were studying or why.

You might be more inclined to read on if the paper stated in its introduction something like, ‘Epilepsy is a common and potentially disabling condition, and a significant proportion of patients do not remain fit-free on medication. Antiepileptic medication is known to have unpleasant side effects, and several studies have shown that a high proportion of patients do not take their tablets regularly. We therefore decided to explore patients’ beliefs about epilepsy and their perceived reasons for not taking their medication'.

As I explained in section ‘What is qualitative research?’, the iterative nature of qualitative research is such that the definitive research question may not be clearly focused at the outset of the study, but it should certainly have been formulated by the time the report is written!

Question Two: Was a qualitative approach appropriate?

If the objective of the research was to explore, interpret or obtain a deeper understanding of a particular clinical issue, qualitative methods were almost certainly the most appropriate ones to use. If, however, the research aimed to achieve some other goal (such as determining the incidence of a disease or the frequency of an adverse drug reaction, testing a cause-and-effect hypothesis, or showing that one drug has a better risk–benefit ratio than another), qualitative methods are clearly inappropriate! If you think a case–control, cohort study or randomised trial would have been better suited to the research question posed in the paper than the qualitative methods that were actually used, you might like to compare that question with the examples in section ‘Randomised controlled trials’ to confirm your hunch.

Question Three: How were (a) the setting and (b) the subjects selected?

Look back at Box 12.1, which contrasts the
statistical
sampling methods of quantitative research with
theoretical
ones of qualitative research. Let me explain what this means. In the earlier chapters, particularly section ‘Whom is the study about?’, I emphasised the importance, in quantitative research, of ensuring that a truly random sample of participants is recruited. A random sample will ensure that the results reflect, on average, the condition of the population from which that sample was drawn.

In qualitative research, however, we are not interested in an ‘on-average’ view of a patient population. We want to gain an in-depth understanding of the experience of particular individuals or groups, and we should, therefore, deliberately seek out individuals or groups who fit the bill. If, for example, we wished to study the experience of women when they gave birth in hospital, we would be perfectly justified in going out of our way to find women who had had a range of different birth experiences—an induced delivery, an emergency Caesarean section, a delivery by a medical student, a late miscarriage, and so on.

We would also wish to select some women who had had shared antenatal care between an obstetrician and their general practitioner, and some women who had been cared for by community midwives throughout the pregnancy. In this example, it might be particularly instructive to find women who had had their care provided by male doctors, even though this would be a relatively unusual situation. Finally, we might choose to study patients who gave birth in the setting of a large, modern, ‘high-tech’ maternity unit as well as some who did so in a small community hospital. Of course, all these specifications will give us ‘biased’ samples, but that is exactly what we want.

Watch out for qualitative research where the sample has been selected (or appears to have been selected) purely on the basis of convenience. In the above-mentioned example, taking the first dozen patients to pass through the nearest labour ward would be the easiest way to notch up interviews, but the information obtained may be considerably less helpful.

Question Four: What was the researcher's perspective, and has this been taken into account?

Given that qualitative research is necessarily grounded in real-life experience, a paper describing such research should not be ‘trashed’ simply because the researchers have declared a particular cultural perspective or personal involvement with the participants of the research. Quite the reverse: they should be congratulated for doing just that. It is important to recognise that there is no way of abolishing, or fully controlling for, observer bias in qualitative research. This is most obviously the case when participant observation (see
Table 12.1
) is used, but it is also true for other forms of data collection and of data analysis.

If, for example, the research concerns the experience of adults with asthma living in damp and overcrowded housing and the perceived effect of these surroundings on their health, the data generated by techniques such as focus groups or semi-structured interviews are likely to be heavily influenced by what the
interviewer
believes about this subject and by whether he or she is employed by the hospital chest clinic, the social work department of the local authority, or an environmental pressure group. But because it is inconceivable that the interviews could have been conducted by someone with no views at all and no ideological or cultural perspective, the most that can be required of the researchers is that they describe in detail where they are coming from so that the results can be interpreted accordingly.

It is for this reason, incidentally, that qualitative researchers generally prefer to write up their work in the first person (‘I interviewed the participants’ rather than ‘the participants were interviewed’), because this makes explicit the role and influence of the researcher.

Question Five: What methods did the researcher use for collecting data—and are these described in enough detail?

I once spent 2 years doing highly quantitative, laboratory-based experimental research in which around 15 h of every week were spent filling or emptying test tubes. There was a standard way to fill the test tubes, a standard way to spin them in the centrifuge, and even a standard way to wash them up. When I finally published my research, some 900 h of drudgery was summed up in a single sentence: ‘Patients’ serum rhubarb levels were measured according to the method described by Bloggs and Bloggs [reference to Bloggs and Bloggs' paper on how to measure serum rhubarb]'.

I now spend quite a lot of my time doing qualitative research, and I can confirm that it's infinitely more fun. My research colleagues and I have spent some 15 years exploring the beliefs, hopes, fears and attitudes of diabetic patients from the minority ethnic groups in the East End of London (we began with British Bangladeshis and extended the work to other South Asian and—later—other ethnic groups). We had to develop, for example, a valid way of simultaneously translating and transcribing interviews that were conducted in Sylheti, a complex dialect of Bengali which has no written form. We found that participants' attitudes appear to be heavily influenced by the presence in the room of certain of their relatives, so we contrived to interview some patients in both the presence and the absence of these key relatives.

I could go on describing the methods we devised to address this particular research topic, but I have probably made my point: the methods section of a qualitative paper often cannot be written in shorthand or dismissed by reference to someone else's research techniques. It may have to be lengthy and discursive because it is telling a unique story without which the results cannot be interpreted. As with the sampling strategy, there are no hard and fast rules about exactly what details should be included in this section of the paper. You should simply ask, ‘have I been given enough information about the methods used?’, and, if you have, use your common sense to assess, ‘are these methods a sensible and adequate way of addressing the research question?’

Question Six: What methods did the researcher use to analyse the data—and what quality control measures were implemented?

The data analysis section of a qualitative research paper is the opportunity for the researcher(s) to demonstrate the difference between sense and nonsense. Having amassed a thick pile of completed interview transcripts or field notes, the genuine qualitative researcher has hardly begun. It is simply not good enough to flick through the text looking for ‘interesting quotes’ to support a particular theory. The researcher must find a
systematic
way of analysing his or her data, and, in particular, must seek to detect and interpret items of data that appear to contradict or challenge the theories derived from the majority. One of the best short articles on qualitative data analysis was published by Cathy Pope and Sue Ziebland in the
British Medical Journal
a few years ago—look it out if you're new to this field and want to know where to start [16]. If you want the definitive textbook on qualitative research, which describes multiple different approaches to analysis, try the marvellous tome edited by Denzin and Lincoln [2].

By far the commonest way of analysing the kind of qualitative data that is generally collected in biomedical research is
thematic analysis
. In this, the researchers go through printouts of free text, draw up a list of broad themes and allocate coding categories to each. For example, a ‘theme’ might be patients' knowledge about their illness and within this theme, codes might include ‘transmissible causes’, ‘supernatural causes’, ‘causes due to own behaviour’, and so on. Note that these codes do not correspond to a conventional biomedical taxonomy (‘genetic’, ‘infectious’, ‘metabolic’, and so on), because the point of the research is to explore the interviewees' taxonomy, whether the researcher agrees with it or not. Thematic analysis is often tackled by drawing up a matrix or framework with a new column for each theme and a new row for each ‘case’ (e.g. an interview transcript), and cutting and pasting relevant segments of text into each box [13]. Another type of thematic analysis is the constant comparative method—in which each new piece of data is compared with the emerging summary of all the previous items, allowing step-by-step refinement of an emerging theory [17].

Quite commonly these days, qualitative data analysis is performed with the help of a computer programme such as ATLAS-TI or NVIVO, which makes it much easier to handle large datasets. The statements made by all the interviewees on a particular topic can be compared with one another, and sophisticated comparisons can be made such as ‘did people who made statement A also tend to make statement B?’ But remember, a qualitative computer programme does not analyse the data by autopilot, any more than a quantitative programme like SPSS can tell the researcher which statistical test to apply in each case! Whilst the sentence ‘data were analysed using NVIVO’ might appear impressive, the GIGO rule (garbage in, garbage out) often applies. Excellent qualitative data analysis can occur using the VLDRT (very large dining room table) method, in which printouts of (say) interviews are marked up with felt pens and (say) the constant comparative method is undertaken manually instead of electronically.

It's often difficult when writing up qualitative research to demonstrate how quality control was achieved. As mentioned in the previous section, just because the data have been analysed by more than one researcher does not
necessarily
assure rigour. Indeed, researchers who never disagree on their subjective judgements (is a particular paragraph in a patient's account really evidence of ‘anxiety’ or ‘disempowerment’ or ‘trust’?) are probably not thinking hard enough about their own interpretations. The essence of quality in such circumstances is more to do with the level of critical dialogue between the researchers, and in
how
disagreements were exposed and resolved. In analysing my early research data on the health beliefs of British Bangladeshis with diabetes, for example, three of us looked in turn at a typed interview transcript and assigned codings to particular statements [18]. We then compared our decisions and argued (sometimes heatedly) about our disagreements. Our analysis revealed differences in the interpretation of certain statements that we were unable to fully resolve. For example, we never reached agreement about what the term
exercise
means in this ethnic group. This did not mean that one of us was ‘wrong’ but that there were
inherent
ambiguities in the data. Perhaps, for example, this sample of interviewees were themselves confused about what the term
exercise
means and the benefits it offers to people with diabetes.