Software as an innovation gap parasite

I want to propose to you a concept I will call the "innovation gap". I will define it as the time between the availability of new technologies and knowledge to n>=1 people and the moment when it is readily available to over 5% of the global population, or to over 50% of the population that could benefit from the innovation.

Around 1850, a bunch of inventors were creating relatively cost-effective incandescent light bulbs. Let’s assume that by 1900 most people in the US and Europe were benefiting from this invention, having their own bulbs and the ability to buy and install them if so desired.

That means 50 years of effort closed the innovation gap through a combination of technological advances related to the innovation itself (better light bulbs) and unrelated changes: cheaper transport, better public opinion, aggressive sales tactics, government subsidies for public electricity.

Some innovation gaps close much faster: Someone finds the Spectre security vulnerability on Intel CPUs. The kernel developers start trying to fix it, once a fix (the innovation) is ready, you have 10 kernel developers benefiting from it, 5 days later that number is now 1000 (people testing stuff), 10 days later it’s been pushed to all Linux rolling releases, and 20 days later basically every distro has it. That means in 20 days the innovation gap has been bridged.

Essentially the innovation gap is filled by:

Related industries coming of age
Mass production
Marketing/Sales/Distribution

At least in the past, there’s an argument to be made that it wasn’t the scientists, designers and engineers that made the innovation who reaped the financial rewards. Instead, it was the likes of the “robber barons”, people that were very good at “sales” - the most liberal definition of sales possible, the one that includes family-connection, bribes, lying, skirting laws and overall acting like an immoral twat.

The “last steps” of innovation, the part where you have to make it popular, is often the hardest, since you need one form or another of societal buy-in. When you’re creating the innovation, you are dealing with nature, when you’re closing the gap, you must deal with people.

There are different ways the gap can be bridged.

It can be filled in, as is the case with light bulbs. Manufacturers sell to large distributors and then things flow down the supply chain. Most of the money is spent making light bulbs better and cheaper, relatively little money is spent on sales, on advertising and on bribing regulators.
The gap can also be bridged by monopolies, stopping anyone else from making the connection. This is the case with things like new monoclonal antibodies designed to fight various strains of cancer cells. The general principles and innovation based on which any monoclonal antibody can be created is patent-free and was paid for by the British and American public, but there’s a regulatory gap worth a few billions for getting a new one approved, and once one does get approved, it’s granted a long-term monopoly where the price can be as high as 10,000x production cost.
Lastly, I think the innovation gap can be infected, this might be worse than both alternatives. What do I mean by this? I mean that the innovation gap can have so many people trying to fill it that it never closes, it’s constantly bustling with activity, thousands of entities use it to feed and are fighting each other for short-term resource dominance, rather than competing to build the best possible bridge.

Infected gaps are usually memetic gaps

It’s hard to find examples of this last category, most of them would revolve around scalping like sales practices. Until recently this kind of phenomenon was rare; you could only see it in social movements like politics and religion, since purely parasitic behavior is only possible when the innovation gap is purely memetic.

You can have one person producing something and one thousand salesmen selling it, and you could argue that behavior is somewhat parasitic, since it leads to price inflation and no real improvement. But the end result is the innovation gap being closed, even if inefficiently. However, the worst case scenario for innovation gap parasitism, in the case of memetic innovations, is that the gap actually gets wider and ultimately might never close.

I think a good example of this is religion, how various people around the world all reached this “enlightened” state of bliss, happiness and love for other people. Yet, in trying to iron-out the details of how to get it across to people, the core of their message slowly got replaced by some version of “Allow this medieval hierarchy to take your money, start wars and molest your children because it leads to eternal happiness”.

But I think this example might be a bit too controversial, or maybe a bit too foreign. So instead, I think a good example is one of the many modern religion replacements/supplements: diets !

It seems like you could vastly improve the health and happiness of all Europeans and Americans via 4 very simple rules:

Avoid products with added disaccharide and monosaccharide when possible
Have gaps of at least 12 hours between periods of eating, ideally a bit more
Avoid products with fully or partially hydrogenated fats
Drink less alcohol

Those rules alone are not equivalent to any one of the 10001 fad diets being paddled around, but they will be almost as good as the best fad diet for the vast majority. They are easy to follow, no matter your budget, schedule or dietary knowledge. You can train someone to follow them in less than a week.

However, we don’t have every nutritionist and doctor spouting these 4 simple rules, or a similar set of criteria. Instead, you’ve got thousands of quacks peddling their own version of a complicated diet, with a catchy name, a few dogmatic “super foods” and some sort of mythology around how bad meat/sugar/fats/gluten/milk/eggs/methionine/etc. are.

But most of the quacks are probably right when they say their diet is “better”. Because the standard European and American diets are so horrible that anyone can create an improvement on them. But the fact that there are so many possible solutions, many of which can be quite profitable for however is peddling them, has led to an infection of the innovation gap.

An infection where most people stuck on the other side of the gap due to decision paralysis; watching the doctors argue about no carbs vs low carbs vs slow carbs or vegan vs pescatarian; not realizing they could just replace that 1L of sugar water they’re gobbling down with some plain water and get half the way to the other side of the gap.

Parasitic software

I think a lot of the software we make can be thought of as parasitic in this way. That is to say, programs that fill an easy-to-cross innovation gap and make it harder to navigate.

When you boil down what computers can do, it’s a rather limited amount of things. Store data, send messages, do math for us, entice us via peripheral devices with pretty audio-visual displays. However, most of the advancement required for computers to do math faster, store more data, send larger messages (think 4k video streaming vs email) and play higher quality video/audio lies mainly in the realm of hardware and material science.

Sure, there are computer scientists who helped adapt the mathematics of simulation software to specific hardware to make it much faster. However, I highly doubt that they were as mission-critical as the hardware developers. My favorite example, since I work in ML, is that is wasn’t a clever new backpropagation technique or tensor computation library that made neural network viable 10 years ago, it was faster GPUs.

But I digress; after all, going after the programmers working on basic science is hardly aiming for a low hanging fruit, it's like trying to argue college is useless and using Harvard grads as examples. I’m quite happy applying this theory to most software development, not all of it.

So let’s limit this to things which anyone can probably agree, that are easy to write and easy to replace:

Software that could be replaced by a spreadsheet and a week of training for the operator
Software that could be replaced by a mysql database, sqlworkbench as a GUI and a set of users with carefully selected privileges.
Anything that could be easily made using Wordpress
Anything that someone else already created, which you re-create with a small twist and a large shift in branding/marketing strategy. For example Wix trying to re-create and overtake Squarespace and Squarespace doing the same with Wordpress. Or an even better example - the endless social-media platform wars that were going on, before Facebook came out as dominant and decided to buy all successful competitors (e.g. Instagram).

In other words, tools that are created not to connect the consumer with an innovation, but to try and drain as much money from the consumer in connecting them with said innovation.

Software for addicting people

The goal of social media is not to provide instant multi-media sharing between friends, it’s to manipulate human’s primitive dopaminergic system into coming back as often as possible, for as long as possible, to watch and click as may ads as possible.

Similarly, the goal of most popular games is not to have an entertaining experience during your boring tram ride, it’s to convince you to spend money on loot boxes.

Look at the top 100 games on the google/apple store and I’d guarantee you 90%+ have “microtransactions” (a form of gambling, but dumber). Yet 100% of those games could just have a 1$ upfront cost, and they’d more than make a nice profit. Instead, they are trying to manipulate people into slowly paying them hundreds of dollars. Why have we reached this point? Well, complicated topic, but it’s not because game developers are evil people, it’s because the market is getting busier and it’s becoming ever harder to make money “fairly”.

Software made to be marketed

The goal of Squarespace was not to create a platform that’s better than Wordpress. It’s figuring out the correct “influencers” that are admired by everyone interested in content production, in order to get to their target audience first and loudest.

This is also how a lot of mobile games got in the sore state they are in now. The mobile market heavily favored those with a lot of advertising money, so the more budget you could throw at ads the better. But ads are expensive and they usually only work if your product is free… the rest is history.

Suffice to say, software made to be marketed is something that exists not because it’s “better” in any significant way, but because someone thought that the marketing strategy of the existing solution sucks, and given a small variation on the software, they could make it sell much better due to superior marketing.

In here I also count software that’s made to be marketed via the insider connections. For instance, bank {X} uses software {Y}, which like any software is not perfect, but does the job. However, the son of the CEO of bank {X} wants to become a ”businessman”… so daddy lends him a few millions to start a software company creating banking software…. Surprisingly enough, bank {X} is rather interested in buying it to try a replacement for {Y}. So we add another shitty banking software to the pile of shitty banking software, nothing gets solved, money gets shuffled around, everyone is happy.

Software for people that can’t use a computer

Inside the ever-festering innovation gap I also include any sort of application that is essentially a spreadsheet with pretty graphics. A Calendar? A to-do list? A game of battleships? Grading software? Accounting software? A sales management system? A ride sharing app? A bank’s whole infrastructure? All can, essentially, be replaced by a spreadsheet or a series of spreadsheets (Think excel, google sheets, libreoffice sheets)

But hold on a second. If everyone replaced almost all number-management apps with spreadsheets, you’d have to spent at least… a whole class in school teaching for everyone to be proficient at using spreadsheets.

Which, I guess, is correct… but we spend at least 20 times that amount teaching kids pointless tat. So, I don’t think the sacrifice is that great.

I’d go even further and claim that a lot of other software, from paid de-archiving tools to paid blue-light reducing software are essentially taking advantage of people’s naiveté, of their lack of knowledge about what a computer should be easily able to do and what open source software is.

But there’s no money to be made in making everyone computer literate; however, there’s dozens of trillions to be made by selling them “solutions” to their lack of knowledge… so, here we are.

But then again, maybe it’s a reductionist view, maybe a bunch of edge cases centered around malicious actors, local laws and time zone differences really makes the difference between needing 4 people and 4,000,0000 people. Maybe I’m just using my cold programmer’s eye to look at a “nice design” and labeling it as “brain hack to trick humans into clicking ads” rather than “generator of aesthetic enjoyment”.

Yet I think it could be interesting to look at some numbers, see how many programmers are working in industries that could be easily labeled as being part of the above categories.

According to SO’s survey of 71,000 developers, 2.6% of them are working in academic research. Another 2.8% are working in manufacturing; 2.8% are working on datacenter software, and 2.3% are working on telecommunications. Are some of the people working in these industries doing pointless worked that falls into the above categories? Probably. But let’s be very generous and say that, since all these industries contain “hard” software-related problems, all the engineers working here are actually innovating, not just re-iterating on bridges inside an innovation gap.

So now, let’s look at people working on addictive software, things that could be replaced by spread-sheets and html pages or thing that already exist and where open-source versions already exist. That’s people working in finance, banking, web development, social services, government service, healthcare service, education advertising, ecommerce, travel and marketing.

That would give us a 10.5/37 split; the rest of the people surveyed worked in vague areas, such as “Software development – other” (I assume game devs live here), “Information technology”, “SAAS”, “Data analytics”, “Consulting”… etc.

So let’s assume that these people follow the same split as above into innovative industries and industries that are infecting innovation gaps.

We’re left with a 22%/78% split, we’re left with 78% of programmers potentially doing nothing but harm to the world, fighting over who gets to connect consumers with hardware and widening the gap in the process.

But I digress, I think I’m jumping to conclusions here, I think I’m going over too many counter arguments. I’m not claiming the above numbers are true, or that this theory is true, but I think it’s a good self-critical point of view to adopt.

Also, please don’t take this as me trying to create an “other” here. The sole reason why this idea crossed my mind is because I worked and still work on many projects which I think fall in this category. If many programmers are really working on innovation gap parasites, I count myself among them.

If you enjoyed this article you might also like:

Published on: 2019-10-16

Reactions: