Regression to the mean when facing the tails

There's a strange phenomenon I've observed when paying attention to my thoughts and behaviour. It's so odd I need to find some external examples before postulating it, just to check. The best way I can describe it is a tendency to behave normally (more so than usual)when faced with very unusual situations. Despite the more unique and reasoned behaviour we'd usually exhibit seemingly being designed for just such an unusual situation.

i - Finding God on your deathbed

It's been often noted that the older people get, the more religious they become. This is not unique to western countries, the trend holds even in Islamic countries (though the research is a bit lacking).

The answers to why this happens are conflicting. I default to the inevitability of death leading people to adopt beliefs to sueth the psychological pain. But I'm sure others argue that the main driver here is that wisdom accumulates with age, and following a religion is prerequisite on wisdom. Yet others may say that those people were "religious all along", but that the business and indeterminacy of youth means that the belief only collapses into action at an older age. Regardless, I think we can all agree the real (pedantic?) answer is multi-faceted and complex, probably including all of the above and many more.

However, I want to remark upon this phenomenon as an example of regression to the mean. I speculate that, on average, young people go through their day with a high degree of uncertainty about death, consciousness, the "meaning of life", a personally "harmful" yet societally "good" superego, ethics and other such subjects that most religions address or disolve. By "high level of uncertainty" I don't mean they don't think about them, quite the opposite, they think about them a lot and have an insane range of opinions around the subjects throughout their youth. Going back to your culture's religion is often a case of "regression to the mean", you abandon carving your own existentialist path and default to what most people are you are kinda-doing.

The easiest way to dissolve this observation is an eve-psych explanation that has something to do with youth and will of exploring and rebelling. Maybe throw in a study about how traits on an aged brain are correlated with being a tory. But this is an explanation that dissolves any young-old behaviour difference, so even if true, it's boring and generic enough as to not provide any satisfaction.

More importantly, the questions which religions address become much more pertinent as you age:

Death? Seems much more relevant after you got wheeled in for a surgery with a 5% fatality rate or had a bout of covid that placed you in the ICU. Experiences which are more probable in the old and the cumulative chance of which can only go up with age (i.e. the chance of having had a life-threatening condition by age X+Y must be bigger than that of having a life-threatening condition by age X)
Ethics? Not very relevant when the most important questions it addresses is "should I break up with Sally because Bob is hotter?" or "should I become vegetarian?". Much more relevant when it addresses "Should I increase the chances of the company making it through the recession, or take a risk but note fire half my staff during a period when finding a job is impossible?" or "Should I continue outing with my buddies 4 nights a week, or spend that time reading to and playing with my kids before bed". The more power over others we have, the more relevant ethics becomes, and power tends to grow with age.
The meaning of life? Seems like more of a pressing question when you deeply feel the inevitability of death (see item 1)
The superego? Seems like more of a pressing thing when you become aware of how much of your life it dictates, and you see an option to leave a nagging spouse, 1.5 kids and 0.7 pets to go enjoy hookers and coke in Columbia for the rest of your life. (also, see item 2)
Consciousness? ... Not sure what's the deal with this one, but people seem much more obsessed with it as they age. The evidence coming from physicists seems undeniable.

This fits a pattern of "Difficult, powerful and unusual decisions/experiences/thoughts leading to a regression to the mean in one's thinking". I might be getting ahead of myself by saying "fits a pattern" here, because I've only given examples around one subject (religion), and a subject which I am fairly detached from (and I presume the same can be said about you, probabilistically young-leaning reader).

ii - Software development

A similar pattern is one that I've noted in myself and many others that work on writing software. Namely that, as you work more, you end up regressing more towards "standard" tooling.

Two years ago I really disliked the idea of docker and thought of it as technically misguided, I even disliked things like snaps. The alternatives I'd propose were standardized environments and aiming to use as many statically compiled packages as possible.

Whether or not my ideas were good is unimportant, they turned out to work just fine while developing two successful projects. I still think that approach is an ideal to aim for, I've not changed my reasoning about it much. Yet when faced with my most recent project, i.e. one where I have to lead a larger team, that has higher stakes and is more complex... I ended up dropping them in favour of docker.

This is something I've noted with many other beliefs I had about programming, that the more I'm in a position to implement them, the more I get scared and regress to the "standard" solution.

I don't think I'm alone here and I think this is a trend that gets worst the more influence you have. Consider who is the demographic that ends up being the software architects and CTOs of large companies, it's usually very good developers, the kind that like niche languages and have weird, strong and usually very good opinions.

Yet somehow large company codebases are not the kind of place where I would go if I were looking for projects in a theoretically sound but not yet very popular language with a compiler that double as a category theory-focused theorem prover.

It might have been that every important engineering manager in the world hated Haskell. But that makes no sense if promotions in the tech world are based on merit (strongly correlated with passion for programming), intelligence-signalling and sophistry that would lead us to expect the exact opposite. So why did Haskell never make it big ?. Here Haskell just a stand-in for "niche approach to problem-solving that is theoretically sound but unpopular and not so widely explored".

And here the argument of "age" doesn't apply too much, because while it might be a factor, tech is the one field where ageism is not a factor, or if it is, most would argue it favours the young. Tech workers aren't just regressing to the ways of their forefathers because they are getting old.

So why are the smartest programmers leading companies using the most boring of technologies?

The "standard" approaches are usually the best solutions.
Progressing into important role requires political favour from the "old guard", so you end up having to think like them.
You might be able to use non-standard solutions, but the people you manage won't.

My reply to those are:

See everything in this blog under the label "programming" for why I will disregard that hypothesis.
Conflicts with the fact that most major tech companies are startups. Some with the founding technical team having the average age of a juvenile frathouse.
The selling point of the non-standard solution is that they will save time and work, so why not use them anyway and hire a smaller and more capable team?

iii - Beauty

I've noted this trend, once again, in myself first. But in hindsight, it's something I observed in many other people that write. Namely that the most beautiful things are the hardest to describe in a way that's appealing.

I can make the history of smart pointers sound vaguely fun and interesting, I can conduct vivid descriptions of mundane scenes, yet ask me to describe the most beautiful vistas I've ever seen and I will lack words.

Recently, I was hiking the cliffs of La Gomera and saw what might have been the most beautiful scenery I've ever encountered; after a lot of post-factual thinking about it the only thing I could say was that "It's the kind of place where I'd go to talk with the gods".

... Which even I will admit is fairly pompous, cliche and uninspiring.

This is not limited to writing either, it applies to photography. The "best" photographs are of rather mundane scenes or objects, limited in scope, commonplace, just very well orchestrated and lit.

Try to capture a truly awe-inspiring scenery, the kind that makes you scared and joyful at the same time, and you're left short. Anyone who's tried to take photos of anything that really impressed them can probably attest to this, and must always back up the photos with a disclaimer about how much better it looked in reality. People taking stills of food never have to such disclaimers, they can perfectly capture and elevate the essence of a granola bar with their lense.

We are usually able to find clever compliments about quirks a person has, but try to compliment the things we really care about in a friend or lover and you come of sounding rather cliche. Not just in the subject matter, but in the way it's expressed. There's a reason why the best muses are imaginary.

Maybe nothing describes this phenomenon best than the philosophical era when people talked about "the sublime", which boils down to certain types of awe at the sheer power, beauty, and scale of nature. Anyone who's seen the Alpine valleys which inspired these philosophers can attest to their shock. But somehow they manage to take one of the most interesting feelings one encounters and make it sound beyond boring and uninspiring.

Indeed, I ought to stop talking about the subject of beauty myself, least I flood this article with so many cliches as to be too ashamed to publish it.

iv - War

Finally, war might be the most interesting of these phenomena. More specifically conscription, and the fact that people allow it to happen with so little opposition.

Consider the fact that, at the end of the day, escaping conscription is not that difficult. Defecting to the enemy under the right circumstances with a few documents or even just the intent of helping is often welcomed and might be rewarded with a pardon and a ticket to somewhere nice, or at least a cushy bureaucratic position in the intelligence or propaganda machine.

Avoiding conscription is risky, but historically much less so than fighting a war. But maybe you want a more concrete example, fine.

The US is a country where less than half the population has voted in the mid-term election. So I assume most people are at best neutral on the whole American identity, cruising by, could be better, could be worst.

Even more so, it's a country that's stereotyped by its citizens being independent, free-thinking, deeply concerned about ethics, open to new experiences, deeply Christian, not quick to give in to threat and freedom-loving.

Yet, only 50 years, ago, about 2.2 million randomly selected young men were introduced into an experiment with two choices:

In the default choices, there's a 10% chance of a horrible death, a much larger chance of mutilation while working as slaves (i.e. under threat of torture or execution for not following orders) in hellish conditions with their main task seemingly being that of murdering innocent civilians, mainly women and children, and/or poisoning and burning their land.

In the alternative choice, there is a <1% of being summoned to a US court and facing the choice of being sent to prison for a few years or moving to Canda.

So how many people dodged the draft for the Vietnam war? ~570,000 out of 2,200,000 dodged the draft.

This is outright insane, even with very bad information, even with tons of propaganda, even if someone is not the brightest, the calculus here is so heavily biased towards dodging the draft that I'd expect the draftees to be case studies for whether or not poor decision making should be classified as a psychiatric condition worthy of involuntary confinement.

I should make it clear I'm not saying the US was wrong to fight Vietnam, or that what they did was unethical given the circumstances. The only claim you must accept is that most people fighting were not ideologically-bound enough for said ideology to outweigh their self-interest or moral code

So why didn't more people dodge the draft? Especially in the later stages when what was happening was public knowledge, why didn't everyone refuse to fight?

The only reason I can think of is that war is a strange time. Most people never encountered a war or a draft, nor ever thought about what they ought to do when drafted.

When faced with this odd and seemingly dangerous situations most people regressed to the mean, they did what they were told, they stopped thinking and went along with what authority figures told them.

v - Regression to the mean when facing the tail

To put it more broadly, I'd expect "unusual" traits to manifest under "unusual" situations. I'd expect interesting scenarios to lead to interesting behaviour. I'd expect difficult problems to lead to creative solutions.

But I look at myself, I look around, I look at history, and I see the opposite. The more unusual a situation, the more "usual" and predictable our behaviour becomes.

Granted, this is all anecdotal, some people get confused when I write these kinds of articles, thinking that I'm trying to make an evidence-based statement about the world. I'm just putting words on a phenomenon I hope some of you have already noticed, but never spared much thought towards. If you don't really think that what I'm describing above is "a thing" based on observing your own mind and behaviour, it might well not be, or at least not for yourself.

Leaving that aside though, if you can grant me that this regression to more "normal" than usual behaviour when faced with weird situations, it becomes interesting to think about why it happens and whether it's good.

I can see this generating a working model of my mind which is quite unusual from the one I'd have assumed. One where I manifest and generate "reasonable" and "unique" behaviour exactly in the situations where I need it least. Those where I already have fairly good working patterns, or where the new behaviour is of low impact.

One last programming anecdote.

I used to enjoy writing what I'd call "modern metaprogramming-focused functional-inspired C++" and Rust in my spare time and thought that if only you could write software in them from the ground up you'd have an incredible advancement over everyone else in terms of maintainability and performance. Back when that behaviour and those thoughts couldn't be put in practice. Now I've had 1+ years of work with serious possibilities of bringing it into practice, and maybe if I suggested putting things on hold for 2 months to change our codebase to Rust I'd get shut down even after doing my best to argue the point. But what scares me is that I haven't even felt the need to suggest it as a passing thought.

Unless I make a huge effort to decouple myself from my immediate memory, my regression to normal behaviour makes sense to "me".

I strongly believe that my technical choices, for example, are the pragmatic choice that will lead to the best outcomes due to a variety of factors: such as availability of people that know the language and the coding patterns used, the mature library ecosystems, avoiding refactoring overhead and learning overhead, not introducing extra languages to the codebase, the... BUT WAIT! screams me from the past.

I've devolved into thinking like the exact people I was rebelling against, I started my path because I thought "I can probably do this better than these old hacks" which lead me to a point where... I've become one of the "old hacks".

One question to be asked is if this is a good thing or not? Maybe regression to the mean in weird circumstances if fine. Maybe this whole "having a distinct personality and rational beliefs and refined behaviours" thing is something only meant for very usual situations. But, if so, I am way too self-centred to consider this.

The question then becomes one of how to stop this for things that matter most. Ultimately I don't care that much about programming language choices, I never quite did.

But I care about e.g. escaping enslavement when war comes or getting out of blue chips or state-backed currency when the system cracks, or not falling into the "1.5 kids 0.7 dogs 2.2 spouses 1.4 mortgages" style of living that seems so soul-crushing, or not abandoning my transhumanist ideals when death gets closer.

Published on: 2021-02-08

Reactions: