A good quality systematic review should identify and synthesise all the available evidence, for a particular question, through meta-analysis. Conclusions can then be made about the effect of the intervention on the outcome. As, in theory, all the available evidence is gathered and assessed, surely the conclusions from the meta-analysis must be the truth and we can then apply this to practice?
Well…..not quite. The transition from putting the conclusions of a systematic review into guideline development is not quite as simple. We need to assess the quality of the evidence presented and its application to practice.
The GRADE system is a systematic, standardised approach to assessing the quality of the evidence in a systematic review. It assesses the study design, limitations of detailed design and execution (the risk of bias), inconsistency, indirectness and the applicability of the evidence, imprecision and publication bias. Limitations in any of these categories can lower the quality of the evidence. As well as rating down, we can also rate up the quality of the evidence. This can be done if the estimate of effect from the meta-analysis is very large, if there is a dose-response gradient, or if the meta-analysis demonstrates a direction of effect despite all possible confounding factors that would expect to cause the effect to go in the opposite direction. With all this considered, the evidence is then graded as ‘high’, ‘moderate’, ‘low’, or ‘very low’.
To illustrate the process, we will use the example of a recent Cochrane systematic review meta-analysis of randomised controlled trials that assessed the use of topical ointment or cream to prevent invasive infection in preterm infants in high-income countries, compared to routine cares (Cleminson, McGuire 2014). Meta-analysis found a borderline statistically significant higher incidence of invasive infection: typical RR 1.20 [95% CI 1.01, 1.42], RD 0.05 [95% CI 0.00, 0.09], 6 trials, 386 infants
We used the GRADE system to assess the quality of the evidence:
| GRADE Category of Assessment | Downgrade quality of evidence | Comments |
| Study design | No | -Only randomised controlled trials were included in the review |
| Risk of bias | Serious (-1) | -Lack of blinding of caregivers and clinicians in all trials-Unclear and inconsistent random sequence generation in one trial
-Incomplete outcome data in one trial |
| Inconsistency | Inconsistency (-1) | – There was evidence of moderate heterogeneity (I² = 36%) that was not explained by pre-specified subgroup analyses.-Individual estimates favoured both intervention and control |
| Indirectness | No | |
| Imprecision | Serious (-1) | -Broad confidence interval |
| Publication bias | No | – There were not sufficient trials to explore symmetry of funnel plots as a means of identifying possible publication or reporting bias. |
No criteria were met that would upgrade the quality of this evidence. Overall, the quality of the evidence is down-graded to ‘very low’. This lowers our confidence in the estimate of effect and therefore we are less confident in the result that use of topical oils in preterm babies to prevent invasive infection may actually slightly increase the risk of invasive infection.
So, a meta-analysis of several trials assessing the effect of an intervention on a particular outcome may find the results in favour/against the use of the intervention BUT the quality of the evidence is poor so we must err on the side of caution when applying the results to guideline development.
The quality of the evidence presented to guideline developers is one of the factors that they consider when making recommendations for practice. If the quality of the evidence is low, then this can weaken the strength of the recommendation. To increase our confidence in the probable magnitude of the effect, and therefore increase the quality of the evidence presented, further well designed research studies are required. These may then support the initial findings of the systematic review, or they may in fact demonstrate an alternative direction of effect. We then incorporate the findings of further research into future ‘updates’ of the systematic review and re-assess the quality of the evidence.
In Summary:
- The demonstration that an intervention can result in a significant effect on an outcome is different than our confidence in these effects.
- We can use the GRADE system to assess the quality of the data
- The overall quality assessment can guide our confidence in the findings
- It is important to consider this when translating the findings into recommendations for practice and into what context they are considered.
For further information, see http://www.gradeworkinggroup.org
Academic Clinical Fellow
York Teaching Hospital
Centre of Reviews and Dissemination, University of York