I just completed the “Reproducible Research” course as part of the Signature Track Certification in Data Science I mentioned in July. While this subject is not completely new it certainly is a timely topic with Science recently releasing a research paper called “Estimating the reproducibility of psychological science” which found that “mean effect size (r) of the replication effects …was half the magnitude of the mean effect size of the original effects.” Additionally, only “thirty-six percent of replications had [statistically] significant results.”
The authors try to state things in the most positive light but the media has picked up on “over half of psychology studies fail reproducibility test.” But the direction of the results for the most part did not change, just the magnitude of the relationship. Still, it is enough to potentially raise doubts about all research that relies on statistical evidence.
What is reproducibility in scientific research? First, consider what the ideal would be—replication. Every high school chemistry class does experiments that are repeatable, regardless of the classroom, teacher, students and classroom equipment. This is the highest standard, to replicate an experiment, with new data, investigators and even new methods, to obtain the same results as the original.
But replication is generally not feasible—think of large clinical trials—so reproducibility is the next best goal. The idea is simply for original researchers to make their code, data and methods available to others. How did this topic gain notice? In the complex world of “omics” and trying to create clinically useful gene-expression tests Duke researchers published results that received widespread criticism from the field but did not review their work until three years later when the National Cancer Institute and the Institute of Medicine got involved.
The IOM report Evolution of Translation Omics: Lessons Learned and the Path Forward describes a set of recommendations for reproducible, safe research. These include using test and train data or secondary samples, “locking” computer algorithms and further recommendations specific to various study designs. Another aspect they emphasize is the clinical/biological interpretation and significance of the results. Stats 101: statistical significance is not practical significance.
So, back to psychology and reproducibility. Thinking too much: self-generated thought as the engine of neuroticism is a recently published opinion piece that claims a link between negative thought and neuroticism (which seems plausible) and neuroticism and creativity. They write: “a key feature of creative thought is the ability to generate solutions to problems that are distinct from the traditional way the problem is solved. This hypothesis is supported by the stereotype of the brooding, tortured, genius, as well as a variety of empirical evidence.”
But one of their supporting empirical pieces found: “Except for bipolar disorder, individuals with overall creative professions were not more likely to suffer from investigated psychiatric disorders than controls.” (Emphasis mine). This opinion paper has been widely picked up in the mainstream press with some outlets calling it a study. In fairness, portions of the article are about their base hypothesis that neuroticism is the result of "a tendency to engage in negatively hued SGT." And they use brain imaging data in their research which is an increasingly popular trend in the psychological and social sciences.
Interestingly, Skeptic magazine just published a review of the book The Insanity Hoax: Exposing the Myth of the Mad Genius by Judith Schlesinger who argues against this myth as being detestable: “the patronizing caricature of the mad creative and how it devalues the artistic product” as well as untrue; with the desirable traits of genius such as tenacity, self-discipline and confidence being the opposite of neurotic traits. And like the others, she cites psychological studies to support her assertions.
My favorite sentence from the review and where we leave the topic, for now:
It's telling that there is no "mad baker" theory of the culinary arts.
Jesse Sharp is an expert in the analysis of health care data. Passionate about data and the ethics of analysis he writes on topics related to medicine, public health and statistics. More...