Imagine it is 1912, but that the Titanic is fitted with an underwater radar system. Imagine that it senses an iceberg so large that even the captain can understand that by the law of conservation of momentum, the ship will be stopped in its path. Should the captain use the radar information to inform the decision to alter course, or should the captain ignore it because radar is merely an invention of science therefore prone to exaggeration and false findings?
The New Yorker Magazine has just published an immensely popular article titled "The Truth Wears Off -- Is there something wrong with the scientific method?" The article reports several examples of scientific findings that appeared to be significant when first published, but when tested over time, demonstrate weaker and weaker results. Zyprexa is a second-generation anti-depressant that showed great promise in clinical trials in the nineteen-nineties. By 2001, Zyprexa earned more revenue than Prozac, and it remains Eli Lilly's top-selling drug.
According to the article, recent studies of these second-generation anti-depressants show that the therapeutic power of the drugs appears to be steadily waning, down to less than half of that documented in the first trials. It reports that many researchers began to argue that the expensive pharmaceuticals weren’t any better than first-generation antipsychotics, which have been in use since the fifties. And it quotes John Davis, a professor of psychiatry at the University of Illinois at Chicago, as saying “In fact, sometimes they now look even worse." How could such drugs be approved if the FDA is using the scientific method, which requires independent reproducibility of results?
Quoting the article:
But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.
In 2005, a paper was published titled "Why Most Published Research Findings Are False," and it has become that author's most-cited work. If we paradoxically accept its findings as true, what reasonable interpretation should we give to President Obama's inaugural promise to "restore science to its rightful place"? This is a serious question If we read the science on global climate change, it reads like a radar screen flashing RED ALERT about the impending iceberg of environmental collapse. Should we heed the warnings that more than 95% of all climate science papers report, or should we maintain course, confident that all these predictions are nothing more than statistical aberrations and gamesmanship?
Over the New Year's holiday I had a chance to watch the movie "Fair Game" [official website: www.fairgame-movie.com], which is based on Valerie Plame Wilson's book "Fair Game: My Life as a Spy, My Betrayal by the White House". (It also incorporates material from Ambassador Joe Wilson's book "The Politics of Truth: Inside the Lies that Led to War and Betrayed My Wife's CIA Identity".) In that movie, the subject of truth is examined in many contexts. There are the 16 words that George W. Bush uttered during his January 28, 2003 State of the Union address ("The British government has learned that Saddam Hussein recently sought significant quantities of uranium from Africa."), words accepted as true enough to authorize the invasion of Iraq. There is the op-ed piece that Ambassador Wilson wrote on July 6, 2003 ("What I didn't find in Africa"). There is a dramatization of the discussion between CIA and the Office of the Vice President about intelligence and whether the seriousness of a potential scenario (Iraq acquires fissile nuclear material) should be allowed to influence the assessment that such facts are in evidence. Finally, the movie shows how the selective use of source information, without proper process, led to a clearly erroneous assessment of facts regarding Iraq's nuclear capabilities, not to mention the possibility that erroneous decisions were made in light of those facts.
It is estimated that the NIH spends more than $30B/year on medical research, and that the CIA spends more than $40B/year on intelligence activities. If "most published research findings are false," would we all be better off in a pre-scientific world? How are we to make policy and investment decisions, be they question of which armies to raise against rogue nations or which drugs to take against rogue cells?
The New Yorker reports that in a forthcoming paper, Jonathan Schooler recommends the establishment of an open source database, in which researchers are required to outline their planned investigations and document all their results. This is an interesting prospect, especially because of the studies I've read about open source software.
The IT industry spends $1.5T/year knowing full well that $500B/year is being wasted on software and systems that will never make it to production, or if they do, bad software quality will "challenge" them with schedule slips, missing features, and bugs significant enough to interfere with operational capabilities. The #1 reason for choosing open source software in the enterprise is "quality as compared with proprietary software". In a series of studies published by Coverity, open source software has achieved on average (across more than 250 projects, more than 55 million source lines of code (SLOC)) 100x lower defect density than proprietary software. Is any of that true? Or is it just a bunch of scientific nonsense?
I believe that the answers to these questions are among the most important we face today. What is science? What can it know? What can it teach us? How should we make decisions based on that information? I'm excited to discover that others believe that open source, which has been inspired by the scientific method, may yet be called upon to rescue science from those who merely try to confirm pre-conceive hypothesis. Can open source prove itself to be valuable? That would be quite a feat. But honestly, I see no better course.
5 Comments