Tuesday, 13 May 2014

Research Waste...Too much Quantity where's the Quality?

Ioannidis JPA et al. Increasing value and reducing waste in research design, conduct, and analysis.The Lancet, 2014: 383:166 - 17
Correctable weaknesses in the design, conduct, and analysis of biomedical and public health research studies can produce misleading results and waste valuable resources. Small effects can be difficult to distinguish from bias introduced by study design and analyses. An absence of detailed written protocols and poor documentation of research is common. Information obtained might not be useful or important, and statistical precision or power is often too low or used in a misleading way. Insufficient consideration might be given to both previous and continuing studies. Arbitrary choice of analyses and an overemphasis on random extremes might affect the reported findings. Several problems relate to the research workforce, including failure to involve experienced statisticians and methodologists, failure to train clinical researchers and laboratory scientists in research methods and design, and the involvement of stakeholders with conflicts of interest. Inadequate emphasis is placed on recording of research decisions and on reproducibility of research. Finally, reward systems incentivise quantity more than quality, and novelty more than reliability. We propose potential solutions for these problems, including improvements in protocols and documentation, consideration of evidence from studies in progress, standardisation of research efforts, optimisation and training of an experienced and non-conflicted scientific workforce, and reconsideration of scientific reward systems.

The recommendations were

  • Make publicly available the full protocols, analysis plans or sequence of analytical choices, and raw data for all designed and undertaken biomedical research
  • Monitoring—proportion of reported studies with publicly available (ideally preregistered) protocol and analysis plans, and proportion with raw data and analytical algorithms publicly available within 6 months after publication of a study report
  • Maximise the effect-to-bias ratio in research through defensible design and conduct standards, a well trained methodological research workforce, continuing professional development, and involvement of non-conflicted stakeholders
  • Monitoring—proportion of publications without conflicts of interest, as attested by declaration statements and then checked by reviewers; the proportion of publications with involvement of scientists who are methodologically well qualified is also important, but difficult to document
  • Reward (with funding, and academic or other recognition) reproducibility practices and reproducible research, and enable an efficient culture for replication of research
  • Monitoring—proportion of research studies undergoing rigorous independent replication and reproducibility checks, and proportion replicated and reproduced

Ten options to improve the quality of animal research
Protocols and optimum design

  • Creation of a publicly accessible date-stamped protocol preceding data collection and analysis, or clear documentation that research was entirely exploratory. 
Use of realistic sample size calculations
  • Focus on relevance, not only statistical efficiency
Effect-to-bias ratio
  • Random assignment of groups
  • Incorporation of blind observers
  • Incorporation of heterogeneity into the design, whenever appropriate, to enhance generalisability
  • Increase in multicentre studies
  • Publishers should adopt and implement the ARRIVE (Animal Research: Reporting In Vivo Experiments) guidelinesWorkforce and stakeholders

Programmes for continuing professional development for researchers

Reproducibility and reward systems
  • Funders should increase attention towards quality and enforce public availability of raw data and analyses

If you put your head above the parapet, people are out there often will take a pot-shot. Sometimes these shots can be fair-do's and often quite destructive and hurtful. Sometimes truth hurts, sometimes people talk mushroom food.

The link to this paper was provided in response to one of our posts, maybe by an MSer or maybe by a have-a-go Researcher. There is a need to up the quality of some papers, so researchers have a read and for the non-researchers if you read science papers have a think about some of the point to help decide if the contents are quality or not.

These are opinions and people will not agree with all of them, including me. There were many comments to this paper made here (click). However, there is a point. As you can see on the Blog there are vast differences in the quality of studies reported, including many clinical studies. Many animal studies are not intended to spawn clinical trials, but some are. This is perhaps where the authors of the comments are aiming the majority of their comments and so get some views on aspects of animal work wrong. However, canning non-reproducible stuff early saves lots of money in the long run and stops us providing false hope.

There are now thousands of journals, many online only and as they can charge for publication, an income generator. This means that they need to fill pages. As such quantity appears to be the name of the game and quality can go out of the window. As such much research with go nowhere and will not translate into human benefit.

What is quality? This will mean different things to different people, some we can agree is lower quality than other studies. However some of the apparent quality work published in quality journals is never reproducible and we see this time and time again. This is the nature of Science, however the referees for (quality) journals would do well to read and adhere to some of the comments, and then quality would go up if more quality control is put into the system.

However, time and time again we see that many studies involving MSers are too small to give a definitive answer, so it needs to be done again wasting years in the process. 

Should this be supported or should researchers be encouraged to do it right first time. I have heard one Genetic researcher state that unless 10,000 samples are looked at then the data will not tell us much. The literature is full of genomic studies involving less than 10,000 samples.

I think TeamG are not holier than though and sometimes one strikes a balance between information, logistics and cost. Small studies can inform whether it is cost effective to do a larger study. It does however, waste time.