This blog is no longer active, you can find my new stuff at: george3d6.com, epistem.ink, ontologi.cc, and phenomenologi.cc
Stakes create good software
It surprises me to no end that founders and VCs, which pride themselves on being economically rational actors, can't imagine a world where everyone else is also a somewhat economically rational actor.
i - Maintenance cost disease
For most software companies and products, development costs seem to increase as they age. Even if the core focus of those companies or products remains unchanged, easy examples being Uber, eBay, Paypal, Lyft and Facebook.
I can present the paradox in a single drawing
(lines represent trends, not absolute values)
The hidden variable here is "team size". As these companies get older they accumulate engineers, but the quality of developers decreases.
The mean salary isn't decreasing due to salary cuts to the original team, it decreases because of the original team cashing out, retiring or starting their own companies, leading to a desperate attempt at replacing each of them with a lot of low paid interns.
This trend seems ubiquitous among large companies, the bit where their core product becomes worst is arguable, the bit where average salaries decrease is fairly indisputable if you include the shares early developers get as part of their compensation.
But why?
I'd expect a "healthy" version of the above graph to look closer to this:
The hidden variable, "team size", starts going down instead of up after the company matures. Salaries increase for longer and remain high because the original developers must be paid large amounts to stay in the team, even though most still probably leave after a while. Getting "replacements" for them is very expensive, they might need months or years of training to get up to speed (the codebase is old and complicated) and arguably have to be smarter than the original devs. After all, creating code is usually easier than understanding the code someone else created, I know many people that can do the former, but few that can do the latter.
ii - Dying a good death
The product has now matured. New features might get shuffled around, but overall you just have to sit back and watch as gradual improvements in hardware, network and compilers make your good product even better by catching bugs, improving performance and reducing server costs.
This is not to say sales, marketing and legal don't have an increased workload as the audience grows, but software, in principle, is the same thing, just add 100x the servers to your cluster and you're good to go.
Some might say that "scale" is the thing that causes dev team costs to go up, and that would be a confounder if people were to only design for scale when needed, but they aren't, so we use expensive IaaS providers, adopt serverless designs and run Kubernetes clusters from day 1. So I don't think the scale is the issue here, everything is already designed for scale. Furthermore, as I often like to point out, bigger machines can bring you a long way, you can fit 12TB of ECC DDR4 on a single instance and by the time I'm done writing this article those numbers might be 24 and 6, we are living at the end of Moore's law and it's amazing.
Maybe it's just a fundamental rule of everything that costs must go up forever, no matter how big or small the system, until it collapses. I'm rather optimistic and think the problem here is solvable and has been solved by a few companies, but it's that very solution that makes them boring.
You will see I use "decay" in the graph above to represent the final phase of a company, the one where usually the dev team grows exponentially with quality being sacrificed in exchange for quantity.
Despite the word I chose, I don't think this "decay" phase is necessarily bad. In the best of cases, it's synonymous with the product being forever enshrined in amber, continuing to be enjoyed by generations to come. At worst, its user base will very slowly dwindle as the market moves away from whatever generators of needs caused it's existence to being with.
A good example of a company going through a good "decay" period is Craigslist. A company making 694 million a year with 50 employees. It seems to be doing exactly what I'd have loved it to do if I ever invested in it, generating millions for me each year, playing it safe in a space it knows well, slowly expanding and improving. Will craigslist live forever? No. But even greed must know some limits, and if Craigslists is made defunct after generating dozens or hundreds of billions in revenue, I think everyone that started it will be fine with that.
Also note, only 500 millions of those are income, what's happening with 194 millions? Even if 50-80% go on something other than salaries, that's still an average salary of... oh god, I mean, I love my job and I'm not particularly inspired by craigslist and all... but... but anyway, I digress.
iii - How to make stake
I believe the reason that most companies are not going the Craiglist route is one of stake. Say you've got the funding team plus investors that have 90% of the stake in a venture, the rest is divided between late-stage investors and later employees. The funding team wants to cash out and stop working forever, which is fine, except that now only 10% of the stake in the success of the company is held by people working for the actual company.
So now you are left with two options:
a) Have workers that are interested in doing a good job for reasons other than the ultimate success of the business.
b) Make the stakes very high.
The first option seems to be what most public companies went with, and in principle is the reason why 2-men startups are able to wipe the floor with behemoths wielding hundreds of billions in assets. It's more or less the same problem that a government has in employing people, the lack of external incentives means that everyone is focused on gaming internal systems (e.g. trying to maximize political influence and department budgets at any cost), and those who aren't are soon kicked out of the game.
The second option seems to be what the Googles and Amazons of the world are doing, aiming to keep exponential growth going so that even if the stakes are very tiny relative to the company's size, employees still feel motivated by sharing in the product's success. This seems to have some flaws, for one, there's only so much growth a company like Google or Amazon can benefit from since their revenue is very tied into the overall economy, which doesn't seem to grow that quickly, if at all.
The third option seems to be the one that almost nobody is taking:
- Focus on what you did well the first time.
- Downscale (or at least keep the team small) to save on internal costs.
- Get very good developers and pay them (presumably) insane amounts to keep them motivated and make sure they never leave.
iv - Working without feedback
Why am I saying one must pay developers exorbitant amounts to keep them motivated? It seems that most are doing just fine with what is getting closer and closer to minimal wage nowadays.
This is true, developers are cheap, but also, somehow, companies find they must hire hundreds of them to upkeep a similar product that was originally maintained by a few, if not one. And for some reason, the really good ones seem to prefer to work as CEOs and CTOs, or get out of the game entirely and work in academia or become FIRE minimalists.
That is because low wages lead to corrupt incentives. Low wages means developers are not too concerned about the company going under, or being fired. They'd prefer for that not to happen, but they can find another job that pays them about as much, maybe they even welcome the change of setting.
The only way to make monetary and stake progress is going up the chain, i.e. into management. But that just reinforces the problem, since it means good developers get promoted into positions where most of their skills become progressively more useless.
If that picture seems wrong to you, reflect upon the last 20 years of software development consisting of hundreds if not thousands of teams of 1 to 3 loners achieving market dominance against well-organized teams of dozens of thousands of people with billions backing them.
So why don't companies have this model internally? Why wait for a startup to beat them when they could foster that kind of environment internally? Sure, Facebook can buy Instagram, but why spend dozens of billions buying Instagram instead of building it themselves.
Maybe it all boils down to fairly obvious insights about how humans think when they are "part of a team" and how people that "got there first" feel cheated if other people end up prospering equally with seemingly much smaller risks. More likely I'm wrong here and the current growth patterns are the best we could get, even given an ideal founding team.
v - Extra thoughts
Capitalism is great, it's the closest human have gotten to whatever magic the gods of love and peace bestow upon their clerics. It turns status-obsessed would-be mass murderers, your Ghnigish Khans, Alexanders the Greats and Stalins, into preference-utility maximizing specialists.
The one problem with capitalism is the wealth accumulation bit, it losses appeal without it, but the hierarchy of competition becomes hopelessly corrupt with it. Happily, enough death and taxes take care of that and have managed to help things move along ok for hundreds of years, granted, with a few hiccups, but hey, what system of running human affairs hasn't caused a genocide or two? We ought to look at the aggregate picture.
Corporations mirror capitalism, as above so bellow and all that jazz, in that the stakes available for any would-be participant get progressively lower. After all, if you buy in and work for google in the very beginning, you're getting shares that price in a great amount of risk. If you work for the google of today, the publically traded capital A google, there's no risk in taking shares as salary, you can sell them the next day for the equivalent cash amount, so no astronomical gains are possible.
This in itself is not an issue, startup developers might have the kind of risk profiles that make them happy to take X as wage and a low probability option at making a few millions as extra. But surely there is a "craigslist developer" profile that makes one inclined to just work for 3X, with no extra gamble.
When I say that a company should aim to "make billions in a few dozen years than die" many people will probably take this as a sign of failure. A business, they don't want to invest in.
I wonder if this boils down to people having much more "static" models of the future than I. I see "50 years from now" as so unimaginably different than whatever I "invest" in now might have some consequence to what or who I am then, but the causal chain is so convoluted all my best efforts at optimizing for the far future are essentially no better than random.
My "50 years from now" involves humans living to 150, most decision making and "reasonable" thought being outsourced to computers, a population boom that forces us to re-think the sustainability of our economy and shifting geopolitical power. And that's the boring stuff that seems easy to anticipate.
At least it seems obvious that the market has changed a lot in the last 50 years, so much so that how you invested hundreds of thousands 50 years ago pales in comparison to spending 1,000$ on buying big tech early in the game, or buying 100$ worth of bitcoin in 2011 (~ 4,000,000$ today).
Basically, I fail to comprehend a world model where one can make any real inference beyond "things will look really weird" more than a few years ahead. Some people seem to have a world model that predicts (they think) 100 years ahead. My opinion or your opinion of those people don't matter, what does matter is that they are probably much more prone to investing in already-mature companies long-term, thus they influence the tune those companies play to quite a lot.
A better writer than me once said: "demons are often mistaken for their summoners because they tend to wear their skins".
I wonder if people make a categorical mistake here where they classify the capital-A Google of today with the "don't be evil" Google olds, even though whatever that Google was is no more.
Many startups seem to act like summoners, performing the high-failure-rate ritual of trying to bring about a megacorporation. But does the megacorporation have anything to do with the startup, enough to think of them as a single entity, operating with vaguely the same incentives and principles? I certainly don't think so.
I recently discussed the ability of Hollywood to bring together a large number of experts in a short amount of time to build an impressive product (movie) and then leave a skeleton crew behind for small edits, sales and marketing.
The person I was speaking with thought this was an excellent idea and something that tech was missing out on. To some extent, I agree that this model would apply to tech.
The one thing I'm unsure of is who the "skeleton crew" ought to be, the marketing company of a movie can't add complexity and then demand more money to solve the prevailing problems. The movie is a finished product, marketing can't touch it. But a skeleton crew that maintains software changes the original product.
Also, it's unclear to me how "team of 10,000 experts" works in the case of software, it seems that most important and resilient software, even projects famous for having loads of contributors (e.g Linux), were in large part built by under a dozen people, often enough just one. Rust seems like an interesting exception.
The idea of startups going into "maintenance mode" instead of scaling though seems interesting, and might be much more customer-friendly. But I guess it doesn't sound as appealing to VC or employees.
On the whole, I think there's this problem of people not wanting to put a cap on things.
- Marriage is forever. [Reality: 8 years average]
- My dynasty will rule the roman empire for a millennium. [Reality: ~40 years average]
- I will be happy and contempt forever if I can get {X,Y,Z}. [Reality: Once desire is satisfied it comes back in a few seconds to hours]
So maybe it's the same thing with companies, "I will invest for the possibility of 10x yields" doesn't sound as sexy as "I will invest for the possibility of 100x yields". Even though, on average, what you get is 10x yields (when things go right), except for a few outliers.
Maybe achieving "good enough" when it comes to software companies is pointless. Maybe it's a game where the "win" condition is a shot at world dominance. Amazon is richer and more powerful (politically) than most countries in the world if it switches to drone delivery it might also be a software update away from having an army. I doubt AGI is going to be that big, but if it is, I think we all suspect who will control the first obviously superhuman thinker. SpaceX isn't the most valuable company, but it's got the best odds at being the dominant cosmopolitical player in the space age, with powers beyond all on Earth combined.
Maybe you either aim for Neuromancer-style corporation levels of power or go home. And whether or not you sell for 0, 1,000,000 or 100,000,000 USD is irrelevant as long as you don't reach the point where you are poised for world dominance.
Another thought is that programming practices can change towards more expensive approaches and you can't necessarily migrate back to the cheap option. Even if you think that plain html + jquery is enough for your webapp and hiring a dozen-strong team of people to write it in react is pointless, you won't find people that work with html + jquery anymore.
Another example, or at least stereotype that fits this, is old software that never gets refactored since it's very fragile and there's no ecosystem to refactor it into that doesn't include some drawbacks, drawbacks which may disturb the hard-fought-for equilibrium it has now reached.
A growth in specialization means you can't cut across having a very large team of specialists with one or two good generalists, since nobody wants to be a generalist anymore, or the people that do want that are crazy enough to almost always take the next step towards being founders.
On the whole, I don't know, but I'm hoping you can at least see my bewilderment at nobody else being bewildered or trying to find a reason for the current state of affairs.
If you enjoyed this article and want to dig more into how the ideas discussed here affect programming in the corporate world, and by extension programming as a whole, I'd recommend you follow up with my series on engineering's sociology.
It's only after I finished the article that I became aware of it, but Strange Loop Cannon has a series of article which contains some similar views on stagnation and degradation. So if you'd like a much lengthier, data-backed, economics-focused take on the above and more I'd suggest trying that, I quite enjoyed it.
I also have a followup to this article coming at some point, called "Mindsets for maintenance", which goes over some practical examples of how to maintain software without increasing costs, drops in quality or feeling miserable. If you're curious about reading that but don't want to subscribe feel free to tell me so via email and I'll ping it over to you when it's done.
Published on: 2021-02-15
