Jupyter Notebook Analysis of Salary Data Spreadsheet | Hacker News Reader

Jupyter Notebook Analysis of Salary Data Spreadsheet

cdn.rawgit.com

91 pointstalloaktrees31 comments

It's interesting - the company I work for (non-US, consulting) did analysis of gender pay gaps internally and noted that on average, females get 70% of male pay. However, within the same role, the difference was just 3%. Their focus is now on flexible working and figuring out why women are excluded/not interested in particular roles.

That seems to be the current state of knowledge.

> Although additional research in this area is clearly needed, this study leads to the unambiguous conclusion that the differences in the compensation of men and women are the result of a multitude of factors and that the raw wage gap should not be used as the basis to justify corrective action. Indeed, there may be nothing to correct. The differences in raw wages may be almost entirely the result of the individual choices being made by both male and female workers.[21]

– U.S. Department of Labor as cited by Wikipedia https://en.wikipedia.org/wiki/Gender_pay_gap

And yet people will still throw out the ~76 cents on the dollar figure that has been disproven again and again. Even 3% may be statistically significant so I don't see why some people feel the need to be blatantly dishonest regarding the problem.

Yes people used horrible and biased stats all the time to prove their point. And most people believe it.

> Even 3% may be statistically significant so I don't see why some people feel the need to be blatantly dishonest regarding the problem.

Statistical significant doesn't mean that it is significant.

The issue isn't just about "choice" etc: it's also about how jobs that women predominantly do are lower in prestige and pay than jobs that men predominantly do.

The change in programming from being a low prestige, low salary job done by women to being a high prestige, high salary job done by men is sometimes cited an example of that. I don't know if there are flaws in that example, but I think the overall idea makes sense. I read an interesting article about this, but I don't recall the link. Sorry.

Perhaps its important to determine if there are flaws in the example before using it as an example.

Alternatively, perhaps it's worthwhile to suggest an idea that might be relevant to a topic, even if one doesn't currently have the time or inclination to explore it in depth. I mean, you're interested in the truth right? Not just the rhetoric necessary to argue a position you already hold.

But sure, go for it. Why would you want someone to suggest an alternative way to explore or think about a problem when you can respond with snark instead?

You might find life more fulfilling if you don't treat new information as inherently antagonistic.

While I agree that the current state of knowledge is that the pay gap is much smaller but existent (the report you link cites 4.8% to 7.1%) after adjusting for job placement etc., it's absolutely a matter of politics and not data that "there may be nothing to correct" and this is the result of "individual choices". If well-qualified women have a harder time being respected and selected for promotions than equally-qualified men, you'll see an effect like this. If well-qualified women know that it's not worth their time to compete against equally-qualified men, you'll also see a similar effect. If couples of ambitious women and ambitious men generally end up favoring the man's career path, you'll see something that looks a lot like "individual choices," but in aggregate is really the result of gender roles.

I do, however, agree that there may be "nothing to correct" in the sense of simply giving women raises to correct the apparent wage gap. If the wage gap is a side effect of an opportunity gap or trust gap or whatever else, we should be focusing on fixing that.

We have a backend position (primarily PHP) open at our company (am I allowed to pimp that outside the who's hiring thread?). It's been open for ~6 months or so now, and haven't had a single female applicant. Not, we haven't had any female interviews; we haven't had any female _applicants_.

Our company is almost 50/50 for male/female, with our engineering team being 100% male. At the last company I worked for in the Tampa area of FL, we had one female developer who was MTF, and we did get one female applicant to an open position, who wasn't hired because she was deemed too junior. This was for a junior position, and I think the CEO was just being unknowingly sexist.

How many applicants (roughly) have you had total? That would be a useful piece of information to put this anecdote in context.

Does your company's team page show the all male engineering team? Or is it otherwise possible for potential applicants to see that your team is all men? A homogeneous team sends signals to women and minorities that may have them ruling you out as a potential place to work.

Even if this was a problem for potential female candidates how could you possibly fix it? Hide the gender identity of your team from the public?

Anecdotally this is absolutely something women take into account when evaluating places to work.

The solution isn't to hide the problem but it does mean that the later you try to fix your diversity problems the harder they will be to fix. As for how to actually solve that problem, you might try looking at what others (especially women) have written on the subject already:

http://geekfeminism.wikia.com/wiki/HOWTO_recruit_and_retain_... http://www.hiremorewomenintech.com/

If I could get a clone of a job applicant and change their gender, I would want to track them all the way through the hiring process:

- Do they hear about the same jobs? Are there differences in their job-seeking networks?

- Do they apply for the same jobs? (Are they even interested in the same jobs? Does the same job with two different job descriptions get different ratios of applications?)

- Do their resumes differ in style or substance (despite identical backgrounds)?

- Do they get the same number of interview offers for the same positions?

- Is their post-interview experience different? (offers, of course, but also benefits discussions and pay negotiations)

- Do they accept identical offers at the same rates?

It would be really interesting to look at this data, but I wonder how easy it would be to gather.

I'm trying to collect a similar dataset for freelancer hourly rates on Reddit:

https://www.reddit.com/r/freelance/comments/4bsk14/freelance...

Tried it on Hacker News as well, but it didn't get any attention:

https://news.ycombinator.com/item?id=11337833

As best I can tell, that last graph would've been clearer as a bar graph and not a scatter plot (or at least better labels). The x axis merely indicates the location. So, "mean annual salaries by location (in kUSD)" might be a good title for the last one.

Also are all the salaries in USD? I know Europe might pay less in tech - but the bottom three paying locations being Sweden, Berlin and London seems strange to me. Perhaps those locations salaries are not being converted to USD...

I appreciate this if only as a great example of using jupyter notebook and pandas. Thanks!

github repo:

https://github.com/jrenner/hacker-news-salaries-data

What strikes me that salary~experience (both company and overall) correlation is basically nonexistent in this particular dataset, at least looking visually.

I had the same thought, but then you have to remember years of experience does not equal mastery.

I may have missed it but it looks like the plot from cell 'In [64]' hasn't converted from Euros and GBP to Dollars.

I've got 99 problems, but a boss ain't one!

Awesome! Think you should convert the currencies at current spot rates instead of just removing the currency sign though :)

It looks like all the analysis is just on base salary. It would be much more interesting if it included bonus and stock.

Interesting there seems to be a shift in the 0-2 year job length position. I wonder if this is because the common trend is to jump ship at 2 yrs, so folks are going out job hunting, and getting a competitive raise at their current position with the job offer in hand?

Most interesting to me was that the gender gap was so much more pronounced in SV than outside

non-SV male median: 97000

non-SV female median: 90000 (92% of male)

SV male median: 137120

SV female median: 99187 (72% of male!)

Unless there's some external factor here, things look pretty damning in SV...

Without looking at the distribution of job_title between male/female in SV and non-SV, these numbers are meaningless.

Not to mention, the sample sizes are quite skewed -- the quantiles might line up better with more data.

Log-log scales would really help. Otherwise - points in the low range are too dense to be useful (big differences in density make be hard to spot, or even - invisible).

Reply on news.ycombinator.com