Is a trillion-dollars worth of programming lying on the ground ?

If you see a $20 bill lying on the sidewalk, today is your lucky day. If you see a $20 bill lying on the sidewalk in Grand Central Station, and you remember having seen the same bill a week ago, something is wrong. - Scott Alexander

This is kind of how I feel about the programming salary-gaps that I currently see all over the world. Let's go by the programming salaries data from Bloomberg because they were the only one kind enough to have a handy table with some.

enter image description here

At first glance, there's a bunch of countries where you can get a programmer to work for you at under 10,000$ a year (on average) and others where you can get a programmer to work for over 100,000$ a year (on average).

In places like the Bay Area and New York, making sums in the 100-150k range is fairly average yet drive 100km and those numbers are cut in half.

Disclaimer: Please don't take anything below as a "scientifically correct" analysis on the subject, take it more as a subjective rant making use of some easy to access data to occasionally verify it's hypothesis. For a full analysis of what I'm doing wrong bellow see the final heading.

1. Under an efficient market

If the market is all-knowing and all-optimizing, how can this be?

Programming jobs are silly easy to move around and most of them can be done remotely.

Even if you want to argue that remote work is inferior to office work, you're still left with the fact that the low-pay countries are low-income countries, and starting an office there is cheap as chips.

There must be some justification that's keeping IBM from firing all it's US staff and cutting costs by 90% + rent by hiring people in Vietnam.

Hypothesis 1: Confounded by regulations

The most obvious reason why this may be the case is that we are being confounded by regulations. Doctors in the US and UK are very expensive, that's not because nobody has noticed there are equally skilled doctors in other countries that charge 1/5th the price (or even algorithms that can do most of it for pennies). It's because practicing medicine in those countries requires to go through a regulatory gauntlet that would make Kafka feel like an unimaginative fraud. If you want to e.g. practice telemedicine from another country and give prescriptions that are valid in US or UK pharmacies, that gauntlet is now 10x the size.

Does this apply to programmers?

Not really, to my knowledge a company can hire programmers as contractors for anywhere in the world. Even if a company can't do that in its home country it can have a subsidiary company in a free-er country that does it.

There's probably some added legal hassle and maybe even accounting-related hassle, but it doesn't seem significant (arguably even less in some situation, since people from other countries can work as contractors rather than employees).

It's hard to find numbers to back up this claim, so feel free to contradict me here if you know any or have more experience on this front. But pending further evidence, my intuition tells me that this doesn't explain even a tiny bit of the gap.

Hypothesis 2: Confounded by skill

One obvious argument here is that the salaries could be confounded by skill. Maybe there is some mix of factors that causes some countries to have better programmers than other countries, so our country-specific data is confounded by that.

I'm not sure what all the factors are, but I think you can boil them down to something like intelligence + education. Being smart and being thought how to be a programmer seems to be the main prerequisite of being a good programmer.

We don't have good proxies for either "intelligence" or education on a global scale. However, we have decent~ish proxies if we go down to the EU level. The homogenized culture, standardization of school curriculums, and relative fairness in PISA tests administration means we can get a good sampling of mathematical ability for kids in the 15-16 age bracket from PISA.

enter image description here

Sourced from ourworldindata.org

Combining the 2012 PISA scores with the Bloomberg data we get data points for 23 countries. Normally I'm against using e.g. arbitrary correlation formulas to determine relationships, but let's start with that:

pearson -> 0.56 ,p~=0.005

spearman -> 0.52, p~=0.01

That's something but nowhere near an explanation.

Considering that "programmer" includes people that were educated 30+ years ago and PISA wasn't a thing back then this might not be very significant.

We can, however, take the average PISA score between 2000 and 2012 and see if the same correlation holds, this might indicate that, if we had some PISA score going back even further, the correlation would still be similar. Even if you don't want to make that assumption, it would account for more of the difference.

There are other issues with taking the average, but whatever, let's go ahead and see:

pearson -> 0.63 ,p~=0.001

spearman -> 0.69, p~=0.0002

So, if we consider the PISA score average for 12 years the correlation gets stronger, the generations that took the PISA test from 2000 to 2012 are now 24 to 36 years old.

So how much of the data are we missing here? Hard to tell.

It seems obvious that looking back further would only increase this correlation. Past 2000 we get closer and closer to communist and fascists~ish dictatorships around the low-pay countries, which are known for their stifling or intellectual life.

The IQ data in Europe tells a different story from the PISA tests, but the quality of that data is much more spurious. At any rate, I think that PISA is a test for more high-level reasoning skills coupled with things like conscientiousness and overall education level. IQ is... well, a rather arbitrary measure collected in a rather arbitrary way which I don't trust, especially not when generalized across different types of tests, cultures, and generations.

I think the optimistic view here is to claim that this is all more or less explained by skill, after all looking back increases the correlation's strength. Then again this would predict something like "Countries that had the highest PISA scores in 2012 have the highest-paid Junior software devs nowadays".

It's annoying to gather numbers of this but I'm going to arbitrarily pick 4 countries from payscale:

The PISA ~= skill ~= pay hypothesis doesn't seem to hold for this sample, since it would predict France and Sweeden (much lower PISA scores in 2012) having significantly lower salaries than Poland and Finland (much higher PISA scores in 2012).

But I'm not sure how reliable Payscale's data is, I err on the side of assuming +/-50% kinda error for their salary ranges which might invalidate the observation about Finland.

Might it be worth putting together a script to scrap all Junior dev salaries and compare with the PISA scores that the current generation of junior devs had (roughly those from 2013 to 2018)? Maybe, but I am lazy, sorry.

Hypothesis 3: Confounded by circumstance

Another theory that would explain the wage gap is the circumstances that force smart people to become programmers

A smart person living in Switzerland might go into programming if they like it, but they might also go into biotech, medical research, industrial chemistry, geophysics, abstract mathematics, or a host of other fields where they would find high pay and social status.

On the other hand, a smart person living in Russia has roughly two choices, programming or working for the state controller oil and gas giants. Caveats apply here, but I think it's a roughly accepted hypothesis that both status and pay is high in Eastern Europe for people working in the field compared to many other fields of intellectual labor. This applies to poor countries in general since salaries in IT track the international market fairly well compared to other fields.

So that means that out of e.g. 50% of Swiss people that are smart enough to be programmers, maybe 0.x actually likes coding enough to become one, however in e.g. Polland that number is 3*0.x. So you end up with people being "forced into" being a developer in the low-pay countries, whereas in the high-pay country you only get people that really love it. People that love something tend to do it better, so whatever is not explained by intelligence and education can be explained by weaker self-selection.

Does the data back up this hypothesis?

Note: I wasn't able to find data for Russia and I manually collected it from the above chart since I couldn't access the raw data.

The answer is a strong NO:

pearson -> 0.59 ,p~=0.008

spearman -> 0.66, p~=0.002

I'd expect an inverse correlation between % of the population employed in ICT and salary, instead, I find the opposite. In the better-paid countries, a lot of people are working IT, whilst in the worst-paid countries, it's much fewer.

It might be worthwhile doing an analysis where we normalize avg IT pay with avg wage and see how much this is influenced by the low-IT countries having the worst salaries in IT comparative to other fields.

Still, considering how strong the inverse correlation is I don't think we can rescue the kind of effect we are looking for out of this one.

Hypothesis 4: Confounded by language

Another hypothesis here goes something like:

Most programming companies are started by people from the US and the UK [citation needed] and basically, all quality resources on the subject are in English [citation needed].

International teams communicate in English [citation needed], so countries with fewer English speakers will have fewer programmers paid at international standards.

Based on the EPI data for 2019 for all of Europe we find and general English skill -> salary correlation of:

pearson -> 0.48 ,p~=0.023

spearman -> 0.50, p~=0.015

It's something.

One thing that I am curious about if PISA average 2000 to 2012 (our proxy for intelligence and education) the EPI (our proxy for language skills) will have good predictive power for per-country salary when put together.

I think this is a good way of figuring out if EPI and PISA are just looking at the same thing (intelligence + education) from different angles, or if together they form a very powerful polycausal predictive model and explain our quandary.

To have more data, I'm going to go ahead and assume and EPI of 80 for the UK and Ireland, a bit higher than the Dutch one. (Note: this would make the above correlation 0.43 - 0.03, 0.54 - 0.005)

Well, the answer boils down to... no. Using a percentage error function we get an error of ~42% using a Linear Regression estimator in a k-fold cross-validation (k=4). For reference, with an X of only 0s the error is ~81%.

So English + PISA is better than nothing at predicting this but they are not the whole story. Indeed, PISA alone yields ~40% error with a regression estimator, and English yields ~44% error with a gradient booster and ~61% with a regression.

Also, changing the models between Linear Regression, Gradient Booster and Decision Tree Regressors changes the error quite a lot. Which is what I'd expect to see if we are fitting on a lot of noise and getting lucky with some models but not with others. Granted, this is also cause by the lack of a validation-set based stopping methodology (i.e. have a validation set and stop training or adjust some hyperparameters to reduce overfitting when accuracy starts going down on that)

So this analysis tells us:

Hypothesis 5: Confounded by drugs

One hypothesis that always seemed interesting to me is that countries like the US and the UK prescribe a lot of stimulants to kids and adults. Maybe this wider use of stimulants gives them am edge in boring yet intellectually challenging tasks (see: most programming jobs).

My experience with stuff like modafinil says meh, but many of the smart people I know swear by them so I think it's worth taking a look.

I think it's worth a shot, I'm kind of desperate at this point. But it turns out there's basically zero good data on this and after looking through studies [1], [2], [3], [4], [5] I've concluded that the best possible data I can get, which might just be noise, anyway this is the best last-year usage data I could find referring to % of participants that used them:

Denmark - 2.5%, Germany - 2.2%, Great Britain- 3.9%, Spain - 2.4%, Sweden -2.6% (first study linked)

and

Belgium - 2.4%, Denmark - 0%,Germany - 2.5%, Slovakia - 3.5%, Spain - 0.8%, Great Britain - 3.1% (second study linked)

So basically the number varies so much depending on the cohort study and the methodology used that unless someone did a more in-depth study on the adult demographic we care about we couldn't tell anything.

Also, the sample of countries is too low, one might have better luck running this analysis using the much higher quality US Data and comparing that with per-state average salaries.

At any rate, I ran the correlations just out of curiosity and got no relation.

Hypothesis 6: PISA and something else

Keep in mind that PISA scores were a fairly good predictor of salaries and average going from 2012 to 2000 even more so.

Maybe there's something another thing which PISA isn't measuring which is part of the "skill" equation and when combined with PISA it would have much better predictive power.

Obvious candidates here are buzzwords like "independence" and "creativity" and "ability to take responsibility" and so on. However PISA scores and English exam scores are a fairly objective marker, these metrics aren't, even if someone did try to measure them.

That being said I think it's a hypothesis worth looking at if I ever find the time to dig through some scores related to creativity or create a methodology for deriving a creativity score (e.g. nr books published per capita + nr of bands + nr of YouTubers, adjust first two negatively for avg income and 3rd one positively... or something like that).

2. Under a corrupt market

Moving on from the right to the left of the economic axis we have a maximally corrupt market, which serves only to funnel money to the richest people at the expenses of the poorer at all levels and thus global inequality in anything (including salaries) should be correlated with how rich a country already is.

How do we identify a "rich oppressive country"? Hard to tell.

Is PPP per capita as good and indicator as any? Probably. Since I don't know how recent the Bloomberg data is, I will go ahead and use the GDP data from 2012, since that's when my PISA data is from.

Does this correlate with high programming salaries? Yes

pearson -> 0.92 ,p~=1.7e-10

spearman -> 0.89, p~=7.3e-9

What if we try to predict average programming salaries using PPP per capita.

Will, a linear regression yields a 21% error. More complex models overfit the data (since sklearn isn't fancy enough to internally validate the model during training) but given the fact that they all overfit and end up with worst accuracies + what we expect to see here is just a straight line anyway, I'll go ahead and assume that linear regression is about as good as we can get.

What if we add PISA scores? Maybe GDP is a factor that goes into the PISA score determination, but maybe not, maybe GDP (the oppressive market hypothesis) and PISA scores (determinant under the efficient market hypothesis) combine to explain all the difference.

Nope, nope, nope

Running the k=4 cross-validation with Pisa + GDP plugged into a linear regression model yield ~31% error, which is wors than GDP alone, so Pisa is just making our model overfit. I say Pisa is the confounder because, remember, Pisa alone yields a 40% error whilst GDP only a 21% error.

But obviously the maximally oppressive market has some flaws:

  1. GDP per capita is a confounder for everything from education, to English skills, to pct of the population with access to computers to lack of led in water, gasoline and building materials to less tax evasion, to "creativity"
  2. The reverse hypothesis (a highly-skilled population earn more money is a generator of high GDP per capita) can be equally true and seems more likely.

If the predictive power of GPD per capita was ~100% (i.e. correlation bordering 1, linear regression => <1% error) I would have admitted this might be a bit fishy.

As it stands, GPD per capita seems likely to be a proxy for other things we care about that could lead to justifiably higher salaries AND not covering the "mystery factor" that would let us fully explain this gap.

3. So what drives the salary gap in the real market?

Ok, so that was a fairly silly analysis to run, but as stated before, this is a "stream of thought let's see if we can see any interesting correlation on this relatively flawed map" kind of article, rather than me trying to draw any authoritative claims.

That being said I think that using this flawed map of the world we can assume, at least for Europe:

But since all of those 4 things are strongly correlated to each other.

GDP per capita has ~0.48 (p ~= 0.004) pearsonr ~0.6 (p ~= 8e-5) spearmanr correlation with english levels and ~0.52 (p~=0.001), ~0.67 (p=~e-5) with mean pisa socre 2012-2000.

Furthermore, the relationship dictated by GDP is no improved if we also factor in PISA scores or EPI ranking, and the relationship dictated by PISA is not improved if we factor in EPI. So the predictive power of those factors is not cumulative, their predictive components are presumably almost perfectly correlated (though a different analysis would be required to determine this conclusively).

That being said there are a few things I haven't considered which I think are important and getting numbers on these is hard:

1. Taxation in low-income countries

It's basically impossible to get any numbers on this but taxation often works differently in lower-income countries.

I happen to know a fair bit about various places in Eastern Europe and it seems to be common practice to hire people here as freelancers rather than employees.

Let's take Russia as an example, picked arbitrarily since I had to look into the matter recently for unrelated reasons.

A 52k USD salary (in terms of costs for the employer) results in only 34k reaching the employee. On the other hand, even a very conservative estimate wouldn't put a freelancer's taxes above 13% total, resulting in 45k being paid to the employee.

So it's understandable that many people, especially many highly paid individuals, prefer to take this route. Even if this route is not taken for national companies (where it's in a legal gray area) it's perfectly legal for foreign companies to hire freelancers instead of setting up a local branch to hire them as employees, so I'd wager most smart companies take this route.

Add to that practices like outright paying the employees part of their salary without declaring it (e.g. cash in hand) that's more acceptable in e.g. Poland, Italy or Portugal the UK, Germany or France.

Add to that the possibility to pay for programming work as IP instead of as a labor/service... and you've got yourself a real confounder on your hands.

So how skewed is our salary data due to things like these?

Also, considering the fact that the salary data isn't very solid to being with, this might just make it an impossibly unreliable mess for anything but the richest countries.

2. Management location dictates office location

Another issue that arises if you assume offices are necessary or at least provide some advantages, is the fact that they tend to be located in areas that are rich because they tend to be located in areas where the management resides.

Take as an example:

The principal Google Campus is located in Mountain View. Why? Because most of their C-suite was based there. So the upper management also started being based there. No qualified PM wants to be away from upper management, that's career suicide, nor do they want to be away from their team. Google has a shit ton of money, so they might not care that much is a PM wants to hire someone for 100k in Mountain View or wants to reallocate his team and hire someone for 70k somewhere else. Thus you end up with people earning 100k+ and living in RVs in Mountain View.

This effect ad-infinitum results in the shiniest startups and corporations being located in NYC, the Bay Area, and London. Then the next tier being located in areas that are tightly connected to those places or minor hubs of their own... etc, etc.

Subsequently, this makes it harder to stat an office anywhere else, because all the good programmers are in those hubs and their friends and families live there and they've gotten accustomed to the local "culture" and don't want to leave.

Even if a few programmers are disposed to work somewhere else, getting a large team going locally ends up being a coordination problem much more expensive to fix than just biting the bullet and hiring in one of the major tech hubs.

3. Most "good" programmers move to the high-income countries

Again, fairly hard to find data on this but a fairly obvious scenario that ties in with the point above.

To illustrate with a toy example:

We have mediocre programmers and they charge on average 30k/y, good ones that charge 50k and excellent ones that charge 80k.

Germany has a 500 mediocre, 300 good, 200 excellent split.

Switzerland has better education and thus gets 30 mediocre, 40 good, 30 excellent.

So Swizterland ends up having an average programmers salary of 53k and Germany of 46k.

But excellent programmers might appreciate working with other excellent programmers and with the kind of institutions that educate them. So maybe 1/4th move from Germany to Switzerland, thus the German average wage goes down to 44k and the Swiss one goes up to 62k.

The more excellent programmers you have in Swizterland the more likely they are to attract other excellent programmers and this becomes a self-perpetuating loop up until living costs become so high that the market is saturated.

Additionally, this might drive some mediocre Swiss programmers to move to Germany thus lowering its average salary.

It's not obvious that something like this is happening in Europe, but it's obvious that it's happening in the US. However, I invite you to browse Angellist and Linkedin and look for jobs in Switzerland and Germany... it does seem like the former contains a lot more of the challenging and prestigious kind while the later contains much more of the "build the 1000000th CRM for doing a slightly different thing than the other 1000000 CRMs" kind.

So you end up with a loop where good programmers are going from Germany to Swizterland and mediocre ones are taking the opposite route, this doesn't show up on immigration numbers. It's very hard to track economically, since measuring the profit companies derive specifically from the software is difficult. So it's just lost to us as noise

Keep in mind all of these points explain the GDP per capita salary relationship fairly well and could be cumulative with skill as indicated by PISA scores.

Conclusion

I'm about as confused as when I started about why the wage gap between countries exist.

I don't think I've found anything particularly new or interesting in my search here. It at least served to rearrange my thoughts on the subject.

If anything it's made me a bit more open to the assumption that you can probably explain away the wage gap with skill + tax irregularities + immigration alone. That is to say, you don't necessarily need any irrationality or unfairness in the market to explain the wage gap, but I haven't proven this, it's just an assumption I might test in the future.

One could argue a huge inefficiency still exists if you assume that offices are not useful and companies should always work&hire remotely (thus removing the cost of living from the picture).

But in that case, I would at least expect office working to go away before we see any dent in the wage gap. However, a lot of companies seem to be set on office work. Whether or not this is a good thing, I don't know, I lean strongly towards "no" but that's a post for another time.

Mistakes

Published on: 2020-06-03









Reactions:

angry
loony
pls
sexy
straigt-face
twitter logo
Share this article on twitter
 linkedin logo
Share this article on linkedin
 Fb logo
Share this article on facebook