F your significance, I want magnitude
Now, provided we can do this a few more times to confirm p<0.05, I am become Death, the destroyer of worlds
Recently the aducanumab approval, via the implied trillion-dollar US taxpayer cost for a useless drug, has made a lot of people suspicious of modern pharmacology. There's a small chance those people might even realize this is already happening with e.g. oncology drugs. But I think there's a deeper issue behind this approval; Behind the fact that billions of dollars were spent investigating such a boring drug.
Significance tests, and statistics as a whole, are an instrument to look at weak signals, the kind that should only concern a specialist building up evidence towards something greater.
Significance tests should have as little to do with experiments as possible. An experiment that requires statistics is infinitely inferior to one that doesn't. The ideal experiment proves a "causal" relationship, or at least approaches that, by showing that no matter how many times we do a thing, the result is always the same.
Such low hanging fruit is gone, we are now looking for weak correlations or for signals at scales much smaller than ours (i.e. ones which we can't measure directly). This is fine, this is how we got the last 100 years of science.
But once statistics are used to assess the outcome of a science-based intervention, something in our collective reasoning has gone stupidly wrong.
Nuclear physics used a series of experiments that relied upon statistical analysis in order to generate a bunch of probabilistic rules at a level lower than those where we have direct sensory access. This in itself is fine, since those rules could be combined into practical human-scale (god-scale?) applications.
You don't need a significance test for a hydrogen bomb, or a nuclear reactor, or even for waterproof polymer. The effects of these applications are so amazing that it becomes pointless to inquire into them being random effects. We are unlocking abilities that we didn't know physical reality allowed for, the very idea of wondering about their significance makes no sense.
In order words, the significance is pointless once you reach a certain magnitude.
Nor is it hard to have an impactful outcome. Medicine used to have great impacts all the time, smallpox vaccines protected very well against a disease with 50% mortality, penicillin and insulin were commonly used to bring people from the cusp of death.
Psychology too used to be able to provide findings that were useful, not just significant. Modern psychology as prototyped by Kahneman and Tversky was used as a falsification tool for deeply rooted fallacies about how humans operate. In this paradigm, a single observation of nonsensical behaviour was enough to negate harmful misconceptions such as economic rationality.
But when people are desperate they tend to cling to any vague correlation for hope, snake oil salesmen prey on this desperation, and thus evidence-based medicine is born.
The idea of evidence-based medicine in itself is not bad, but it doesn't fix the underlying problem of useless medicine. Even if we have very high confidence that a drug does "a bit more than jack shit" to treat a condition, it's still "just a bit more than useless". Conversely, when an intervention is so strong as to look miraculous, it's sufficient to gather a few trustworthy observations to be confident about its efficacy.
A "real" medical intervention of the 21st century wouldn't require much statistical analysis to announce it's coming, It would be heralded by the 80-year-old running 120-mile marathons looking not a bit different than the day he turned 20. it would be heralded on the front page of youtube in a video showing a quadriplegic man racing a Tesla using his exoskeleton. Observable to every child when they hear stories about "flu" and "colds" and "infections" and "herpes" from their parents when they ask why their school gives them a vaccine pill at the beginning of each year.
Real scientific discovery is hard, it requires strong models that are seemingly useless and might only pay off in dozens of hundreds of years.
I should make this clear, I'd never advocate the idea of scrapping "useless" science. On the contrary, we ought to embrace it. We have to welcome the fact that a lot of useless shit will get done, and we have to try and hold that useless stuff to as much rigour as possible using statistics and 3rd party monitoring.
But when society asks for the scientific establishment to pay its dues, it ought to answer with undeniable tools. Significance testing has to stay a mechanism internal to the method of discovery, used to validate sub-claims. The end results have to be able to speak for themselves.
Published on: 2021-09-13
