I spent the first half of this weekend with a legendary friend in Bristol, who puts me to shame every time I see him with his literary and life broad-scope. He waved Intelligent Life, The Economist's side journal, at me as a 'must read', with particular reference to an article on a Brazilian tribe's approach to medicine.
On towards home, and to Papa Purshouse, who is an engineer and extremely squeamish; our worlds rarely collide (apart from a memorable episode during my intercalation when I was doing a module on biomechanics). So it was with great surprise that his favourite light read, The Economist, was waved in my face soon after rocking up at my parents' doorstep.
And lo, The Economist defined my weekend unexpectedly with the question - How Truthful is Science?
My background interest in Open Access has driven my curiosity into how we read information, and how that information comes to be something you are able to read in the comfort of your home/lab/office. Whilst I learned how to critically appraise and analyse literature at medical school, it's people like Ben Goldacre who have made me question the information in front of me in a more global way. The cover feature in the Economist this week notes that the vast majority of findings in scientific research studies have been found to be unreproducible, and reiterates what Ben Goldacre has been saying for a number of years about positive result publication bias. The article notes that reviewers rarely reanalyse the data, and that there is no culture or appetite for data replication by other research groups to validate data. Peer review on submitting a paper is seen as the gold standard of quality assurance. And yet, it doesn't seem to be working. John Bohannon recently tested this theory by submitting a fake article on an invented study - it was accepted by 157 out of 304 journals.
It is funny to reflect on this as a junior medical doctor with research aspirations. The Economist article is right - one is looking for results. Often, a negative result is seen as a hurdle to overcome rather than a result in itself. I cannot think of many memorable studies I have read with a negative result (apart from the whole bevacizumab 'disease free survival versus overall survival' debacle). But at my stage, the obsession with publications from those around me is almost suffocating. Of course, we want to do it properly and accurately. I fear the peer review process as much as the next person. But once it's out in script, it is Job Done. The idea of going back over someone else's work instead of moving on to the next thing is unthinkable. In the Economist article, they talk about a lab that offers to validate your results independently - but who pays for this? Individual scientists? And where is the value-added in the system for this?
My key reflection on this is the lack of value on peer review. Peer review is done for free and is an expected part of the job for any established scientist. And yet, on the biography page of most scientists, all that is listed is their list of publications. There is no note about the number of articles they have peer reviewed. In the financial squeeze, scientists have to focus on what will keep them employed - and at the moment, that's research output rather than a really cracking review.
So back to the Intelligent Life article, where a journalist follows a Brazilian tribe and observes their health care rituals. He reflects on how he brings his own prejudices to the situation; how can these methods possibly work? And yet they do. I guess overall, we assume the system we know is best. But, much like the argument in favour of open access, perhaps we need to think beyond the devil we know to find a more accurate way of answering science's big questions.
On towards home, and to Papa Purshouse, who is an engineer and extremely squeamish; our worlds rarely collide (apart from a memorable episode during my intercalation when I was doing a module on biomechanics). So it was with great surprise that his favourite light read, The Economist, was waved in my face soon after rocking up at my parents' doorstep.
And lo, The Economist defined my weekend unexpectedly with the question - How Truthful is Science?
My background interest in Open Access has driven my curiosity into how we read information, and how that information comes to be something you are able to read in the comfort of your home/lab/office. Whilst I learned how to critically appraise and analyse literature at medical school, it's people like Ben Goldacre who have made me question the information in front of me in a more global way. The cover feature in the Economist this week notes that the vast majority of findings in scientific research studies have been found to be unreproducible, and reiterates what Ben Goldacre has been saying for a number of years about positive result publication bias. The article notes that reviewers rarely reanalyse the data, and that there is no culture or appetite for data replication by other research groups to validate data. Peer review on submitting a paper is seen as the gold standard of quality assurance. And yet, it doesn't seem to be working. John Bohannon recently tested this theory by submitting a fake article on an invented study - it was accepted by 157 out of 304 journals.
It is funny to reflect on this as a junior medical doctor with research aspirations. The Economist article is right - one is looking for results. Often, a negative result is seen as a hurdle to overcome rather than a result in itself. I cannot think of many memorable studies I have read with a negative result (apart from the whole bevacizumab 'disease free survival versus overall survival' debacle). But at my stage, the obsession with publications from those around me is almost suffocating. Of course, we want to do it properly and accurately. I fear the peer review process as much as the next person. But once it's out in script, it is Job Done. The idea of going back over someone else's work instead of moving on to the next thing is unthinkable. In the Economist article, they talk about a lab that offers to validate your results independently - but who pays for this? Individual scientists? And where is the value-added in the system for this?
My key reflection on this is the lack of value on peer review. Peer review is done for free and is an expected part of the job for any established scientist. And yet, on the biography page of most scientists, all that is listed is their list of publications. There is no note about the number of articles they have peer reviewed. In the financial squeeze, scientists have to focus on what will keep them employed - and at the moment, that's research output rather than a really cracking review.
So back to the Intelligent Life article, where a journalist follows a Brazilian tribe and observes their health care rituals. He reflects on how he brings his own prejudices to the situation; how can these methods possibly work? And yet they do. I guess overall, we assume the system we know is best. But, much like the argument in favour of open access, perhaps we need to think beyond the devil we know to find a more accurate way of answering science's big questions.
No comments:
Post a Comment