{"id":738,"date":"2013-10-08T21:03:04","date_gmt":"2013-10-08T20:03:04","guid":{"rendered":"https:\/\/blogs.bmj.com\/adc\/?p=738"},"modified":"2013-10-09T09:17:36","modified_gmt":"2013-10-09T08:17:36","slug":"guest-blog-the-end-of-systematic-reviews","status":"publish","type":"post","link":"https:\/\/blogs.bmj.com\/adc\/2013\/10\/08\/guest-blog-the-end-of-systematic-reviews\/","title":{"rendered":"Guest Blog: The end of systematic reviews?"},"content":{"rendered":"

\"\"So the titles intentionally provocative and NOT the brainchild of the post’s author (@JRBTrip of @TRIPdatabase) … but Jon has provided us at the Archives with a paediatric-orientated version of the new TRIP rapid-review system. Read on to find out more, and comment \/ tweet us your thoughts … Bob Phillips for @ADC_BMJ<\/p>\n

Trip Rapid Review system.<\/h2>\n

The Trip Database is a free, EBM-focussed, search engine.\u00a0 Think Google but for those interested in high quality evidence.\u00a0 Amongst other content types (e.g. guidelines, CATs) we include systematic reviews, possibly the internet\u2019s largest collection of systematic reviews.\u00a0 Systematic reviews are a vital component of evidence-based healthcare.\u00a0 However, they are not without problems, and I have written extensively on the topic<\/a>.\u00a0 One of the biggest problems being that they take way too long (nearly two years) which means that many are out of date when launched.\u00a0 Also \u2013 given the huge workload \u2013 there are not enough of them meaning many (?most) areas of clinical practice are not covered. It is from this thinking that Trip created it\u2019s recently released rapid review system.\u00a0 It\u2019s certainly not intended to replace systematic reviews, it\u2019s more of a proof of concept, a curio, something of interest!\u00a0 I wanted to show that, with very little money and some imagination you can produce something really interesting (it\u2019s certainly that) that might alter the perception of people involved in systematic review production (which it might).\u00a0 But, there are lots of issues with the system \u2013 the main one being it hasn\u2019t been validated (but we\u2019ve shown some positive early results<\/a>).<\/p>\n

So, what does it do and how does it work? It\u2019s really quite simple, we used machine learning to train the system to recognise when an abstract reports the intervention has a positive effect and when it has a negative effect.\u00a0 This took an age, requiring over 500 abstracts being added to the training system, each read and marked as being positive or negative. Scarily, most machine learning training tasks require thousands of examples, so there\u2019s still further learning\/teaching required.\u00a0 As such the system occasionally has problems and we require a human to tell us the answer.<\/p>\n

So, what does the system look like?\u00a0 When you go to the Trip site (www.tripdatabase.com<\/a>) to the right of the search box are three buttons, Trip Rapid Review is on the bottom (underneath the PICO search button).\u00a0 You simply click on that and two search boxes appear.\u00a0 One is for the population of interest (e.g. childhood epilepsy) and the other for the intervention (e.g. melatonin)\u00a0 Once you entered these you press \u2018Search\u2019 and Trip goes off and returns articles that match the search terms and that are likely to be clinical trials. \u00a0 The user then scans the returned trials and selects the ones that match their intention and press the \u2018Analyse\u2019 button.\u00a0 In less than ten seconds a score will be shown.\u00a0 The maximum score an intervention can get is +1 and the minimum -1.<\/p>\n

The score if generated based on a number of reasonably sounding – to me – assumptions!\u00a0 Each trial is given a score of +1 (if positive) and -1 (if negative) and we then adjust these scores based on sample size (we use machine reading for this).\u00a0 If the trial is large (>1000 participants) we leave the score untouched.\u00a0 If the trial is medium (100-999 participants) we reduce the score to +\/-0.5 and if it\u2019s small (<100) the score becomes +\/-0.25.\u00a0 The adjustments are based on the broad principle that the smaller the trial the more unreliable to results are.\u00a0 We then add up all the scores and divide by the number of trials \u2013 giving an average score.<\/p>\n

The biggest issue we have is what do the scores mean?\u00a0 I\u2019m fairly happy with the notion that the closer it is to +1 the more likely the intervention is to be effective.\u00a0 But at what point does it become effective and at what point is the result unclear?\u00a0 For instance, is a score of 0.2 good or bad?\u00a0 I think much depends on the number of trials and the typical size of trials, but these are all things that\u2019ll need to be worked through.<\/p>\n

To illustrate all the above points I\u2019ve created a screencast of me working through the example of melatonin inchildhood epilepsy.\u00a0 If a picture paints a thousand words, how many a screencast?<\/p>\n