Saturday, 31 August 2013

American-based scientists tend to big it up.

Fanelli D, Ioannidis JP. US studies may overestimate effect sizes in softer research. Proc Natl Acad Sci U S A. 2013 Aug 26. [Epub ahead of print]


Many biases affect scientific research, causing a waste of resources, posing a threat to human health, and hampering scientific progress. These problems are hypothesized to be worsened by lack of consensus on theories and methods, by selective publication processes, and by career systems too heavily oriented toward productivity, such as those adopted in the United States (US). Here, we extracted 1,174 primary outcomes appearing in 82 meta-analyses published in health-related biological and behavioral research sampled from the Web of Science categories Genetics & Heredity and Psychiatry and measured how individual results deviated from the overall summary effect size within their respective meta-analysis. We found that primary studies whose outcome included behavioral parameters were generally more likely to report extreme effects, and those with a corresponding author based in the US were more likely to deviate in the direction predicted by their experimental hypotheses, particularly when their outcome did not include additional biological parameters. Non-behavioral studies showed no such "US effect" and were subject mainly to sampling variance and small-study effects, which were stronger for non-US countries. Although this latter finding could be interpreted as a publication bias against non-US authors, the US effect observed in behavioral research is unlikely to be generated by editorial biases. Behavioral studies have lower methodological consensus and higher noise, making US researchers potentially more likely to express an underlying propensity to report strong and significant findings.


So non-behavioural science such as genetics are no so easy to big-up. There it is difficult to manipulate the results such as say reading a genetic code, but behavioural sciences such as drug tests are woolly and one can pick different ways to analyse the results. 

The results in behavioural studies had more extreme positive biases, suggesting that people are sexing-up their results by over extending their analysis.

Furthermore the work that had US-based authors (remember the US is a magnet for the World's best scientists) showed a greater tendency for the more extreme examples to occur. Therefore they are more likely to find that Whacky drug effect.

Why? Probably pressure to get that all important result. This can be the make or break for the people doing the work. Dull results...no grants. It can be the survival of the fittest.

I suspect in a few years this regional influence will decrease as more and more pressure is heaped on global scientists to deliver the goods and get those eye catching headlining papers.

Taylor-made for non-reproducible studies.

22 comments:

  1. There is a tendency in this country to look upon the USA as having the answer to all our health problems. When I was diagnosed, I was encouraged to write to my doctor cousin in New York. An American, he trained here because we had the best doctors. I didn't write, my consultant was Prof MacDonald.

    ReplyDelete
    Replies
    1. As this country (UK) has decimated its pharmaceutical industry R & D we will have to look to other countries for the answers

      Delete
    2. Yes agreed, but we still have to trust the Doctors and scientists in this country. It isn't just about Pharmacology. My point was as patients, the idea that the answer is in the USA is not always the case.

      Delete
  2. More likely it is the journal's that allow bad studies to be published.If there was a better review process maybe you would have fewer Zambonied studies making it to publication.

    ReplyDelete
    Replies
    1. There may be some element of reality here, certain journals are interested in stuff that is media sexy and may not stand the test of time.

      As to the review process sometimes it can appear ace but other times it is like having your teeth pulled. Waited a week for a review for one paper waited 4-5 months for another...which is unbelievable and very very frustrating. I hope the referees are barred from refereeing anyone elses paper.....this was simply unacceptable.

      If I were an editor (I have too much to do to want to do this) the first thing I would do is have anonymous papers. I think too much gets through just because of the name,and perceived power, of the author.

      I did one that was pretty much junk but it was published.....turns out the author was a section editor of the journal.

      That leads to point two maybe you should never publish in a journal in which you are editor....It stops any comments of being an in-house journal...Wonder if ProfG has thoughts on this :-)

      On another point stopping stuff being published means much information is lost. Better that we see the rubbish stuff, because you often get to realize, what is rubbish and what is not and importantly what avenues to go down and which ones to avoid.

      If someone had failed to repeat the Zamboni study very quickly, then the dogma may not have formed.


      Delete
  3. I cannot believe how MouseDoc has censored my comment on account of it being critical of the way America operates. My gosh, what a feeble and pathetic figure you are, Dr Baker.

    You are no different from the Tory nasties ruining this country. You have omitted an intellectually reasoned POV merely on account of it upsetting American readers: the same folk that don't believe in free health care.

    I pity you.

    ReplyDelete
    Replies
    1. Dear "America can kiss it"

      You say "You have omitted an intellectually reasoned on account of it upsetting American readers"

      No I decided to spam them it because it was some form of xenophobic rant that has nothing to do with MS.

      How can Atomics bombs, Hollywood, Syria and the name you hide behind show intellectual reasoning and important how this is relevant to MSers I don't know.

      There are differences in health care systems between countries and I thought about selectively postings bits of the post that were relevant.
      But frankly didn't have the time.

      People in America likewise post about their views about our health care system, which is not without its problems. However if is wrapped up with a load of nonsense then it risks being put in the bin

      The Tory Nasties that you are ranting about have respected democracy, which the US extoles. Maybe we took a vote and democracy won and your rant was removed.

      Maybe I am American maybe I am Blue, maybe Green and take offence at being called blue

      Delete
    2. Nonsense Dr Baker, you turned your back on democracy by unilaterally deciding to remove my comment. It was you that was spineless and undemocratic by being perturbed at the prospect of my criticisms offending Americans. Perhaps you have a vested interest in America.

      I am very disappointed and, yes, you are as small-minded as David Cameron. You should run as a Tory councillor in 2015.

      Delete
    3. As a neutral observer I suggest you go to Hyde Park's Corner or elsewhere to do politics. Stop doing it here on a MS blog. Thank you.

      Delete
    4. If you have democracy you don't always get the right result

      Maybe I have a vested interest in America, but I think that is irrelevant here, it happens to be a place where an eighth of the WorldsMSers live.

      Naturally I can understand your disappointment given the time spent writing the deleted rant

      Spineless....yes..I do not post in my real name...but watch out the CIA could be watching: -).

      Unilaterally deciding...well...you comments fulfilled criteria for deletion, but I will recall blog-parliament tomorrow and take a free vote...post, bin, redact. Its sat in spam waiting for comment!

      So both G and now myself have been equated to David Cameron in the last couple of weeks.

      I wonder if I will be censured for responding?

      Delete
    5. PS What makes you think the blog is a democracy? :-)

      Delete
    6. Blog parliament! Who knew?

      Delete
    7. "America can kiss it" = Dr Dre II

      Delete
    8. Who knew?
      All administrators have access

      The vote was unanimous-SPAM...see MD2 below.

      As ProfG said the rants levelled at the US could equally be leveled at UK but two hundred years earlier.

      anyway enough said

      Delete
  4. Everything is bigger in America and not always what it seems. You have to wade thru the flood of information. Don't forget to get your immunizations:-).

    ReplyDelete
  5. Huh?

    I read through this abstract (and the following commentary), and I really don't understand the point of any of it.

    ReplyDelete
    Replies
    1. go to BBC Radio 4 and listn to Inside Science
      http://www.bbc.co.uk/programmes/b038ynzc
      (not sure if this works if don't have UK IP address)

      Research by Professor John Ioannidis suggests the reality is falling short of the ideal. He talks to Alice Roberts about bias in softer science disciplines, and how having an American on the team leads to more exaggerated claims for the results


      It says that studies on behaviour and particularly those from USA have a higher chance of bias towards to reporting highly significant positive data....expanding the subtext so they more likely to show stuff works, maybe when it does not really

      Delete
  6. Really this just smacks of U.S. bashing. Andrew Wakefield is British. Dr. Zamboni is Italian. Hwang Woo-suk is from South Korea. Maybe it's not just Americans showing publication bias.

    ReplyDelete
  7. I did not write the article, it was brought to my attention as being of interest and was published in an American Journal.

    Andrew wakefield has nothing to do with MS, thankfully, although I did waste bits of my life on him on his theory was around the vasculature, Zamboni gets a fair amount of column inches as do stem cells.

    Working on an reporting behaviour studies which I do occasionally on this blog, it is important that it is clear that there can be over exaggeration of the importance of stuff it helps you to digest. We spend half of our time scratching our heads understanding stories, this speaks. Was the "salt story" real-eat salt gt autoimmunity or is it over exaggeration? Does eating blueberries cause neuroprotection?

    This is stuff you have to realise when you hear media claims for cure this and cure that...and this you do hear. This adds to warnings about the EAE brigade of which I am one, which gives you cure a week.

    So when you hear the stuff that sounds outrageous take a step back...it may just be.

    Oh course it is not just American Labs (they may not be Americans after all as many people working in US are of non-US origin) that over egg...this study just indicates they egg more than most!

    If the truth hurts put a plaster on it....However, you have to admit it is topical:-).



    ReplyDelete
  8. Blog Debate Over, Votes cast
    The Ayes to the em..Right have it

    Post by "America Can Kiss it" confined to Spam

    So with Democracy you do not always get what you want.
    I am sure politicians on both sides of the pond are very aware of this.

    ReplyDelete
  9. One of the functions of the blog is to engage the public with things that we do. Therefore you may get more posts on vitamin D, viruses and mir. Mir what? are they they are micro RNA that are epigenetic factors.... You say you have not published on this subject. Correct.

    I know but we have spent the last year getting you up to speed so when it comes you understand.

    Another important aspect is the failure to translate.
    Why do so many animal studies not produce human drugs.

    I am in the middle of writing a paper and one aspect I thought pertinent is over exaggeration of the importance of the result by the scientist and overinterpreting the significance of the result by the clinician.

    THIS IS A PERFECT EXAMPLE FOR THIS VIEW, CITATION ON ITS WAY

    ReplyDelete

Please note that all comments are moderated and any personal or marketing-related submissions will not be shown.