At the recent OSCON 2012 convention, Kaitlin Thaney, Manager of External Partnerships at Digital Science, shared in her talk a fresh approach to scientific research. Her views offer a way out of the stagnation of many research fields, where the rewards system (how researchers get evaluated for job positions, raises and promotions) and the economics of funding discourage researchers from openly sharing their data and tools with larger communities and the public.
Here is a set of remarkable quotes from her talk:
We are still dealing with some of the archaic principles carried over even from the sixteen hundreds...
Thaney is referring to the retrograde practices that are still used in scientific publishing, particularly regarding the form and content of scientific papers. While the rest of the world has embraced the Internet, the large majority of scientific publications are still using this format: 10 pages of single spaced writing with occasional figures, pictures and tables. Overall, it is static and utterly insufficient in it's ability to enable anyone to properly evaluate the work, much less attempt to replicate it.
We are locked in old mechanisms, when you think about how science is rewarded, the golden ticket in this case is still the scientific paper. Go back to 1665, the notion of writing something down on a piece of paper, and transferring knowledge that way...
The irony that the Web was invented at CERN with the explicit purpose of accelerating the dissemination and sharing of scientific information, adds insult to injury. There are certainly a few progressive journals, such as PLoS ONE, that have embraced the idea of online-rich publications, but the large majority of the scientific community is stuck in the publishing practices of the 1700s.
“...traditions last not because they are excellent, but because influential people are averse to change and because of the sheer burdens of transition to a better state.”
—Cass Sunstein
When it comes to what our future was supposed to be [it's] 'where is my jetpack?' My day to day is: 'Where is my data?'...when it comes to academic research.
Thaney is alluding to the realization that some of the expectations we had a decade ago about our future, didn't come to be. We certainly don't have those jetpacks to commute to work, nor the flying cars in our garages, nor the dinner in a pill. However, when it comes to academic research, the missed future comes down to fundamentals. We still don't share data across research groups (except for some rare bright cases), and even more worrisome, in many cases researchers do not know where their own data is, certainly not the data of the experiment they ran last year.
Some of the most expensive research that has been done is managed by post-it notes and poorly annotated excel spreadsheets...
Thaney highlights the fact that very few labs have a systematic revision control system to store and classify their own data, and even fewer have formal databases or combinations of databases and laboratory information systems. Even in the rare case that systematic data-management exists, the data is not available for sharing outside of the organization that captured it. In the absence of such basic data tools, Thaney follows up with a fundamental question:
Where is my ability to reproduce experiments?
This may come as a surprise to any external observer of the scientific process, but the unfortunate reality is that the majority of labs are not able to replicate their own experiments, much less facilitate the replication of those same experiments by others. And Thaney points rightfully to the source of the problem: the fact that the incentives for academics are tied to the number of papers they publish and not to the actual relevance of the research, nor to the level by which their results can be reproduced by independent groups. Researchers are asked for novelty, not for correctness of results. The outcome of this is that about 90% of scientific papers are not reproducible.
At the core, funding agencies, scientific publishers and academic institutions have confused the profession of researcher with the profession of inventor. By using metrics reserved for inventors to evaluate their researchers, they convert scientific workers into “paper writers”, preventing researchers from becoming what they should have been: agents of discovery.
Scientists can be really stubborn human beings and if the incentives are not there, you are not going to see the behavioral adoption... [Science] is one of the most important enterprises that we have and it is time that we redefine performance.
By applying the hallmarks of open source communities—openness, transparency, meritocracy, and reproducibility—to these problems, we have the opportunity to restore the scientific method in scientific research.
The work that Thaney does today at Digital Science is focused on providing tools that will enable researchers to restore the practice of reproducibility in their daily work, sharing data with peers and the public, and accelerate research and discovery for all.
3 Comments