We're so good at medical studies that most of them are wrong

ars technica:

We're so good at medical studies that most of them are wrong

It's possible to get the mental equivalent of whiplash from the latest medical findings, as risk factors are identified one year and exonerated the next. According to a panel at the American Association for the Advancement of Science, this isn't a failure of medical research; it's a failure of statistics, and one that is becoming more common in fields ranging from genomics to astronomy. The problem is that our statistical tools for evaluating the probability of error haven't kept pace with our own successes, in the form of our ability to obtain massive data sets and perform multiple tests on them. Even given a low tolerance for error, the sheer number of tests performed ensures that some of them will produce erroneous results at random.

Statistical validation of results, as Shaffer described it, simply involves testing the null hypothesis: that the pattern you detect in your data occurs at random. If you can reject the null hypothesis—and science and medicine have settled on rejecting it when there's only a five percent or less chance that it occurred at random—then you accept that your actual finding is significant.

The problem now is that we're rapidly expanding our ability to do tests. Various speakers pointed to data sources as diverse as gene expression chips and the Sloan Digital Sky Survey, which provide tens of thousands of individual data points to analyze. At the same time, the growth of computing power has meant that we can ask many questions of these large data sets at once, and each one of these tests increases the prospects than an error will occur in a study; as Shaffer put it, "every decision increases your error prospects." She pointed out that dividing data into subgroups, which can often identify susceptible subpopulations, is also a decision, and increases the chances of a spurious error. Smaller populations are also more prone to random associations.

And we've got to the point that everybody stops listening when they hear "a new study has shown . . .".

We shouldn’t. Science (and medicine is based on science) advances not on the basis of one study, but on the basis of many studies. Groundbreaking research is important, but what is more important is comparing those results to what is known and seeing if the results can be reproduced. Those which can’t be reproduced get weeded out, and gradually a consensus forms about what is known. Its how science advances, medical science included.

[quote=the article]…Shaffer seemed to suggest that we simply have to recognize the problem and communicate it with the public, so that people don’t leap to health conclusions each time a new population study gets published. Medical researchers recognize the value of replication, and they don’t start writing prescriptions based on the latest gene expression study—they wait for the individual genes to be validated.

I think this identifies the real problem. The media is hyping studies before the results have been reproduced. This hyping of the latest medical discovery is good for TV ratings, I suppose, but its lousy medicine and it confuses the public.

Fascinating article. I took a baby statistics course (very basic) for my doctorate, and it was eye-opening with regard to the gaps in what statistics can and cannot tell researchers. Yet the media are very excited to report "new findings" with little, if any, regard to a study in context of other studies or the need to be replicated, etc.

I'm no mathematician or statistician, but I think the need to create new statistical equations is fascinating -- a whole new frontier of science!


Wow, so many topics this touches upon.

The field of statistics as a whole is a remarkable story. While there has been work that could be described as ‘statistical’ done for centuries, it is literally only in the last century that any unification as a single disipline has been acknowledged. The field has grown exponentially, and now with the ubiquitousness of technology and powerful computers, its growth continues perhaps even superexponentially.

I found the statement that faulted statistics interesting, and would disagree. One can think of statistics as a discipline concerned with describing signals and noise. In fact, it gives us incredibly powerful tools to do just that. But the fallacy present in much of modern thinking is that we concentrate on describing the signal, and so long as we describe the noise to some extent, it will go away. Umm, no. You could describe it exactly, and it still won’t go away.

No, this seems like a tendency to not use statistics well. The problem then is, when one uses it well, one cannot always make as many bold statements. And answers of “maybe” doesn’t make for riveting journalism.

About doing “so many tests”, I always tell my scientific colleagues to prioritize their questions they want answered and be willing to not ask some of lower priority. My rationale is, the more questions one asks, the more opportunities one has to be wrong.

Speaking of statistics:


That is on my desktop at work. A classic. :)

I don’t know if there is a perfect answer to any problem, or we wouldn’t have problems, would we? I think the thing is, we have high expectations of a lot of companies and doctors and strategists; yet, we don’t ever take into accordance the fact that we are all merely human. What we can do is deal with the lot that we have, and there are a lot of medical and insurance companies who are doing just that. And that’s an awesome thing to see. There’s one insurance company that I think has a christian based name and it’s called Golden Rule . It’s all about the do unto others as you would have others do to you. Anyway… I think it’s pretty cool that we are making the most of what we’ve got!

Thanks so much for sharing this article. Regarding the topic of scientific results:

There are set of methods of inquiry used for scientific experiments. One of those methods consists of requiring the results and data to be publicly reviewed and replicated. Before anything can be deemed “scientific knowledge,” it must be submitted to the scientific community for criticism and empirical testing by others. Other scientists must be able to attempt the same experiment and obtain the same results. (Remember that these processes do not necessarily guarantee that the information is correct, but rather, they serve as a minimal criterion.) This criterion can be especially helpful for a consumer who needs to distinguish a claim as either pseudoscience or scientific knowledge.

The following article addresses how the continuous change of scientific technology (i.e., even just the updating of computer software) and the use of computational tools can interfere with being able to precisely repeat an experiment (when attempting to reproduce results).


For example, in some cases, the data resides in proprietary databases. Some of this work may run up against the issues of data preservation, as older information may reside on media that's no longer supported or in file formats that are difficult to read.

Another example is that when using computational analysis, it is not always possible to repeat the exact same experiment, because not all of the necessary information and procedures can be seen. The raw material of computational analysis can be a complex mix of public information and internally generated data.

A further example is the use of software tools. An analysis pipeline may involve dozens of specialized software tools chained together in series, each with a number of parameters that need to be documented for their output to be reproduced. Like the data, some of these tools are proprietary, and many of them undergo frequent revisions that add new features, change algorithms, and so on. Some of them may be developed in-house, where commenting and version control often take a back seat to simply getting software that works. Finally, even the best commercial software has bugs.

Does the continuous advancement of scientific technology and the arrival of computational tools means that it's time for a major revision of the scientific method?

The mass media is a very poor source of news when it comes to new scientific discoveries. More often than not, the science reporting gets it wrong

I agree that it is not the numbers that are the problem, but the way those numbers are being presented in order to further political and economic interests.

The net result is that science itself is becoming marginalized from our decision making process. There is only so many times that science can be used as a political and corporate tool before cynicism sets in.

Heisenberg was correct. (See Physics-uncertaincy principle).

I’ll try to keep the thread going.
Tobacco smoke is a class A carcinogen- Studies done could only get to an 80% confidence level that it is a carcinogen. In order for a medication to receive approval for distibution it must reach the 95% confidence level. If you are a statistician or a scientist you know the jump in statistical certainty from 80 to 95% is huge. The whole new health care bill (I use lower case 'cause its about control not about “health care”). It doesn’t even include the “children” like obama siad it would! They will use statistics to deceive the people but use the propoganda of hope and change to try to ram it through.And keep ramming it through the people who actually work and contribute to our society.

Thanx. Maybe for each discipline they need a fossilised OS & db systems to go with it.

I just read an article in the NY Times Bk Review, archivists are having a similar problem with literature. Salman Rushdie donated some old Apples containing early drafts, notebooks, &c to a library & in his will John Updike left Harvard fifty 5.25 disks. Gee, thanx. :confused: Now all they need is a ?86 running DOS X or Windows* Y*.

Didymus, I think this is the article you mentioned:

I am surprised this issue hasn’t been better resolved. I remember the issue being raised back in the 1980s, when personal computer standards were changing rapidly. It was wondered if historians 100 years from now would be able to access computer media.

From the NYT article:

Located in Silicon Valley, Stanford has received a lot of born-digital collections, which has pushed it to become a pioneer in the field. This past summer the library opened a digital forensics laboratory — the first in the nation.

The heart of the lab is the Forensic Recovery of Evidence Device, nicknamed FRED, which enables archivists to dig out data, bit by bit, from current and antiquated floppies, CDs, DVDs, hard drives, computer tapes and flash memories, while protecting the files from corruption. (Emory is giving the Woodruff library $500,000 to create a computer forensics lab like the one at Stanford, Ms. Farr said.)

It seems strange that the issue has taken so long to be addressed.

your post is so good.you share nice tips in this post which are helpful for me also others.thanks for taking time to discus this topic and share.
Alina's List:thumbsup:

Religion is basically a cultural system creates long lasting meaning. Anyhow whatever your religion is, In all cases, You must try to respect other religions. As no Religion permit to say anything bad against any particular Religion…
columbia star
charleston sc newspaper

This seems to be off topic.

Jud 1:3 Beloved, being very eager to write to you of our common salvation, I found it necessary to write appealing to you to contend for the faith which was once for all delivered to the saints.

Yeah, have to admit, when I hear about some new ‘wonder’ treatment found in a study to cure such and such condition coming down the pipeline, I think I’m not going to be the first to give it a try. I’ll let someone else have that honor. That is one of the big selling points on the past though - people like to try the “new” product. New generates sales.

"There are three kinds of lies: lies, damned lies and statistics"
--Mark Twain (although attributed to various people before him)

DISCLAIMER: The views and opinions expressed in these forums do not necessarily reflect those of Catholic Answers. For official apologetics resources please visit www.catholic.com.