While Archimedes does, not infrequently, get all concerned about invasive fungal infections, this post is not of the issue of beta-D-glucan testing, or problems of azole interactions. Instead, its a swipe at the problem of how, given a transparent system of asking questions, acquiring information, and appraising the evidence we can come to such differences when we get to applying this. Why do we find it so tricky to break our clinical practice mould? 

Well, it may be just something entrenched deeply in our psychological make up as humans. A fascinating read is provided by “ Biases in the interpretation and use of research results.”  which details studies giving groups of politically partisan university students contradictory articles, flawed in identical ways, and finding the interpretation of the evidence was almost always to strengthen their initial positions. The interviews and comments reflected how, despite academic training, people found favour in the methodology which supported their views.

This supports our observations of how two people can look at the same data and see the opposite conclusion, and we can use it to hold fast to the ideas of a framework of critical appraisal that can objectively demonstrate risks of bias. Sadly, it doesn’t quite get as far as managing to help us see beyond the mould we work in … but perhaps acknowledging the mental furrows we work in is the first step to moving beyond them?

Reference: RJ MacCoun. Biases in the interpretation and use of research results. Annu Rev Psychol
1998: 49; 259-87.

Well … it never happened in antiquity.  Archimedes got killed by an invading soldier.

But back to Archi in the ADC – should we abandon this column as the articles are not ‘proper’ systematic reviews, or is a good short-cut review nearly as good as a 12-month long ‘full’ review?

This is, as you would no doubt guess, something that there have been a number of publications about! A recent email discussion on the evidence-based-health listserve on the issue was summarised by Jon Brassey of tripdatabase (of which Archi has repeatedly spoken favourably).

The bottom line is that most rapid reviews are more limited, have less transparent methodologies, are uneven in quality and yet come up with qualitatively the same answer as proper big reviews. So until we have even better evidence comparing the short-cut and the ‘full’ systematic reviews and shows us that the short-cuts are dead-ends, Archi’s staying put for a while longer.

I am regularly faced with questions comparing two management approaches, and sometimes struggle to work out if the data which supports them shows that one thing is better, one thing is maybe not better, but not worse, that the two things are the same, or that we can’t really tell what the differences might be. Technically, I am looking to see if one way is superior, non-inferior, equivalent, or if the data is just too scarce to tell.

It can be helpful to try to work out what you really need to know, when addressing this sort of dilemma. If you need to know if strategy A is definitely* better than B, then you’ll be wanting a confidence interval (CI) of the risk ratio, of whatever outcome you’re considering, that doesn’t include 1.

If you need know anything else, you need to define what is a clinically meaningful difference. It’s challenging to decide this, but say you conclude that a severity score that differed by 2 points was important, then this is your clinically meaningful difference. For equivalence of treatments, you want the difference to not reach 2 points between the therapies. For non-inferiority, you want to make sure the 95% CI new treatment is not going to give you a severity score doesn’t include “-2″.  If the confidence intervals stretch wider, especially if they cross the line of no effect, then you’re data are just too few to get an answer. (Or “underpowered” if you want to sound posh.)

Armed with these definitions, you can decide when you compare things if you will be satisfied with superiority, non-inferiority or equivalence. Or you can do, as I sometimes end up doing, whatever we usually do.

* definite 95% of the time

What can you do when a ‘gold standard’ isn’t actually that good at diagnosing a condition? It can be terribly problematic in interpreting sensitivity and specificity – for example comparing polymerase chain reaction diagnosis of microbiological infection with culture results. The ‘false positive’ may actually reflect real, and otherwise missed, diagnosis, and the ‘false negatives’ a failure of the old standard to identify someone who isn’t really unwell.

One thing to hold onto is that, at their core, most ‘diagnoses’ are a short-hand for a similar group of pathologies leading to a similar set of outcomes. What is ‘bronchiolitis’? What is ‘leukaemia’? It may be that with some conditions, a new diagnostic test needs to be evaluated as a prognostic marker or risk stratification aid before emerging as a new diagnostic criterion (it’s worth reviewing the story of the Philadelphia chromosome and leukaemia in this regard). With many tests giving continuous outcome values, it becomes potentially more meaningful to think of them as a graded indicator rather than a positive or negative result.

With more thinking along these lines, it can become tricky to really split hairs between prognosis and diagnostic tests, and may be worth considering them all as predictive factors. How you wish to interpret them, as risk ratios, or sensitivity/specificity, then depends on how you need to use them.

A while ago Archimedes reviewed the benefits of using ‘pre-appraised’ search resources, short-cuts to the best methodological quality evidence to answer clinical questions. The favoured database of many, PubMed [www.pubmed.gov] has now receieved a new addition to the range of resources on offer.

The main database still houses the ‘clinical queries‘ entry page [www.pubmed.gov/clinical], where you can select the focus of your question and use proven filters to cut to the best papers to answer your enquiry. There is also a new collection, ‘PubMed Health‘,[www.pubmed/gov/health] that draws together guidelines, full text of many systematic reviews, and independent appraisals of may more reviews, in order to improve our access to rapidly usable information. Using this as a source of search information should be tried out right at the top of your search hierarchy - looking for ‘summaries’ and  ’synopses’, and ‘systematic reviews’ before having to rely on ‘single studies’.

As with all search resources, you’ll need to play and see how well it delivers on the questions in your own subspeciality before incorporating it into your everyday practice, but this may be the PubMed version you’ve been waiting for.

Hot on the heels of this great Archimedes on whether or not you should routinely do an LP in infants with a urinary tract infection, comes another publication, covered with a fairly critical review in Journal Watch.
What’s fascinating here is both “sides” drawing a conclusion that they can’t draw.
The authors took 392 infants aged 30 to 90 days who had had an LP, identified from a microbiology lab database, and found 57 who had evidence of a UTI.  They found that one of these 57 had disseminated sepsis.  They noted that this infant had other clinical features of sepsis, and conclude that routine LP is not needed in children aged 30 to 90 days who have UTI and no other features of sepsis.
Here’s where I think they go wrong:  They then go on to state that abnormal urinalysis has a negative predictive value for meningitis of 98.2%.  This might be arithmetically correct – I’ve not checked – but it is wrong to do.  The reason for this is that one of the groups contains just one patient, and you should be very careful of any statistical conclusion from just one patient.  What if this was the only patient from the next thousand?  Or what if it was the first of a series of ten patients about to turn up in the analysis?
In Journal Watch, the commentator concludes the reverse is true; that the children aged 30 to 90 days with UTI remain at high risk and so should have LP. However, if we assume that this study has underestimated the numbers of menigitis even fourfold, that’s still 53 LPs that were unnecessary.  Some of those – at some point – will cause harm.
Who should we believe?  The truth probably lies somewhere between the two, and may be articulated as follows:
- in children with UTI, the younger the child, the higher the risk of disseminated sepsis
- at very young ages, it is likely that a prescriptive rule of “always do an LP regardless of clinical signs” may be safest, but as children get older, clinical assessment should gain ascendency
- the conclusion from the Archimedes article, which is: “Between 0% and 2% of infants under the age of 3 months with urinary tract infection have co-existing bacterial meningitis” stands unchanged with this addition to the literature.

My clinical bottom line?  Think about doing an LP.  Think very carefully, and then follow your clinical judgement, based on the clinical progress of the child.

Probiotics are everywhere these days. They are supposed to prevent all kinds of diseases, from infectious to immunological to allergic. Some of the claims have strong evidence, some not. A pilot study by Youngster I, et al, in which the role of probiotics before immunisations is studied, is yet another positive discovery, but there are several limitations that force us to be cautions and not overly excited about it.

The study is a small prospective double blind randomised trial from an urban medical center in Israel. Healthy infants aged 8-10 months were recruited. Fifty six infants were randomised to receive either a probiotic combination of four microorganisms or cornflour, matched in appearance and texture. A sachet a day was given orally for 5 months (from 10 to 15 months of age). At 12 months, infants received the MMRV vaccine. At 15 months, the authors measured if protective antibody levels were achieved. During all the duration of the study, a coordinator contacted the parents every two weeks to ask for compliance and side effects.

In short, there was no significant difference between groups in either the proportion of infants achieving protective levels of antibodies, or the number of side effects. When the authors used a composite outcome they observed a trend in favour of probiotics. Considering the proportion of samples tested where the titre of specific IgG did not achieve protective levels for each of the four diseases of the MMRV vaccine, the p value almost reached statistical significance: 15 infants in the placebo group versus 8 in the probiotic group (p = 0.052). The same happened when assessing adverse events: 11 infants in the placebo group experienced “any adverse effect” versus 4 in the probiotic group (p = 0.051).

Composite outcomes can be misleading. Other limitations, such as the small sample size and the unclear allocation concealment also bring uncertainty the results of this pilot trial. So, there may be a role for probiotics when vaccinating, but right now we need a lot more evidence.

Did you know that asprin following MI doesn’t work for those with Gemini and Libra star signs? No, it’s true*. The ISIS-2 trial, which deomonstrated the mortality benefits for anti-platelet agents after myocardial infarction with a p<0.00001 only showed benefit for people born in ten of the twelve signs of the zodiac. So if you believe statistics, and randomised trials, then you could save 1/6th of the antiplatelet bill by not giving it to this lot.

The problem of subgroup analysis is fraught with statistical, philosophical, and I would argeu emotional problems. We all want to find out not just if something works, but in whom does it work best, or worst. In oncology, we want to reduce the toxic effects of our therapies. (Even if at times this doesn’t seem to be the case.) With expensive biological agents in rheumatology, we want to keep the heathcare budget in balance yet maximising the benefit to patients. And to do this we look at subgroups.

The ‘obvious’ problem is that if you take 20 subgroups, you’d expect 1:20 ( which is 5 in 100, which is 0.05 …) to be ‘significant’. The less obvious thing is that chance may throw up three spurious association in 20 subgroups, around 1 in 7.
A defence is that ‘it’s biologically plausible’. Perhaps. But set yourself a challenge. Pick a treatment, and split the patients. For example, acute asthma and magnesium sulphate infusions. Now, imagine it works better in the more poorly patients. Come up with an explanation. Flip it – it works better in less poorly patients. Explain that. When I’ve tried it on wards, I’ve found most of us can do this in under a minute.

Subgroups are exploratory. They suggest. They hint. If repeated and repeated, they may be true. However, you should hold tight when you see the next subgroup analysis and remember that the true* explanation may actually be astrological after all.

* OK – so ‘true’ may be a lie.

… and other Bohemian aphorisms …

There is a quite brilliant paper from the under-advertised PLoS One which shows how, in the are of incubation periods for respiratory disease, Truth By Citation is quite strikingly different than the reality of the evidence. The networks of citations demonstrate how repetition, sometime but not always with a citation, leads to a ‘truth’ emerging which does not reflect the real picture of the evidence.

Truth, beauty, and absinthe

This paper joins a similar mass of information which demonstrates how information about prognostic biomarkers are dominated by the few studies which show remarkably strong associations, and rarely reference the systematic reviews that place the studies in context.
And there is are still the classic example of sudden infant death and sleeping position.

It’s these gentle reminders of the need for an overview of the evidence, and understanding that our current view of reality is just that – our current view – that persuade me that despite a decade of Archimedes there is still a requirement for us to encourage each other to seek the evidence for our actions, and then change if the evidence tells us we’re doing it wrongly.

Obviously, I’m excluding the rather large proportion of my workload where the presence of white cells in the CSF indicate metastatic disease … but in normal children, if you did an LP on a child after a seizure and got a total white cell count of 19, would you be treating for meningitis?

I have to say, I would. But there are a team from Australia that are considering the question: until then, what will you be doing?