After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.
GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my CoL adjustment is -40%, and I live in the United States!). They explicitly state on their hiring page that they prefer people from cheaper areas. [0] [1]
On top of that, just from their headcount, it sounds like they have way too many people. 160 for a technical company like this is hard to manage when the employees are local; it's even harder to manage when employees are remote. Again, for remote work, you need mature, highly-skilled staff with a strong history of successful self-direction. You're not going to get that when you're chopping their salaries in half based on their location. Bad local wages is a major reason to work remotely in the first place.
[0] https://about.gitlab.com/jobs/developer/ ; use the calculator at the bottom of the page
[1] https://about.gitlab.com/handbook/people-operations/global-c... ; gems include "you are required to notify us when you move and we may adjust your compensation up or down" and "[a]ll things being equal we will hire people in lower cost markets vs. higher cost markets."
I 100% think you are on the right track. The only people from my area that would take a GitLab salary just can't get a job elsewhere.
As it is, for me the calculator gives me roughly what I was making in Dallas, a bit less (at the top end) than I make now in New York, and a Lockheed entry-level salary if I were to move to Ft. Worth.
That's... aggravating. You'd think their system would treat all locations in the same Core Based Statistical Area as the same. Better yet, treat all locations in the same Combined Statistical Area as the same; in fact, the definition of a CSA is a set of adjacent CBSAs with interconnected commuting patterns.
And the data is all freely available in machine-readable form from the Census Bureau's website [0], so there's no excuse to not use it.
In fact, just this past week, as a personal project, I've written a bunch of code to parse a whole ton of census data, combining CBSA/NECTA data, data from Summary File 1 (i.e. population, sorted by a number of demographics), and Gazetteer data (for land area, so I can calculate population density). Over the weekend, I'm going to take a stab at including data from the American Community Survey too.
Sacramento, CA. Mean Salary (per latest BLS data) [0] for Software Developer, Applications: $107,540; for Web Developer: $78,050
Sacramento, CA Gitlab locality pay index: 0.39+0.25=0.51
Vs.
New York City, same BLS figures [1]: $108,770/$81,430
NYC Gitlab locality pay index: 1.00+0.25=1.25
Sac/NYC Salary Ratio (BLS): 0.989/0.958
Sac/NYC Salary Ratio (Gitlab): 0.512
Fundamentally, the rent index based methodology you are using is, I would guess, giving you below market pay almost everywhere that isn't NYC or San Francisco, assuming that you've hit the market wage for the skills you are targeting in your NYC baseline.
[0] http://www.globalpropertyguide.com/most-expensive-cities
I wonder if any of these indexes breaks down a city by zip code. A good analogy for here would be if you averaged all rent over the neighborhoods in Brooklyn then used that number for a bunch of candidates in Williamsburg.
It really penalizes living in a third- / fourth-tier startup hub, for example, vs either a first- / second-tier hub or a non-hub.
Taken from another perspective maybe it's an intentional filter on the candidate pool to specific cities or countries. That wouldn't seem to be consistent with most remote philosophies but it's something to consider.
Interestingly, Buffer's salary calculator is almost the opposite with a relatively small percentage difference between Nashville or Austin vs SF or NYC for instance.
Didn't notice it until after the edit window was closed.
Remote employees who are good at being remote have some very good options at this point of their career. Why the heck would they take less money simply because they had the foresight to move to a low cost of living location? Half the (stellar) people seeking remote work have built their entire lives around this fact - and have made very conscious decisions regarding their career and lifestyle.
Obviously it's working out for Gitlab, but I can't imagine any of my senior remote talent finding this acceptable in any way. I guess I could see it working for a couple years "converting" an engineer who is super excited to move into remote work vs. on-site. But beyond that, this policy seems extremely dangerous to me.
I'm not really sure what the justification for CoL adjustments is, beyond "we can get away with it". Apparently I'm worth paying $120,000 working from home in London, but if I move 70 miles south I'm worth half that. Not only does it make little sense logically, there's no way cost of living here is less than half that in London, so its not even calculated correctly.
Getting rid of the COL deduction by selecting NYC as my location actually makes it reasonable so something tells me not pinning the US salary floor to 1.0 is saving them money and simultaneously getting them lower quality developers.
By offering people medicore, local salaries, they are losing a massive opportunity to clean up on the great talent in remote areas.
Why would someone want to live in a low-end city (assuming they are mobile) - when they could live the same 'quality of life' in a much cooler place?
Assuming a degree of mobility - the whole point of living in a 2cnd tier place is that you can save a lot of money, which makes up for the '2cnd tierendess' of the place.
Montreal is 1/2 Boston. There are a good batch of great devs in Montreal. I could double my salary and move to Boston? I'd probably just do that. And still save more.
If they paid a 20% premium over other Montrealers, they'd be able to attract the best talent - and still save a lot of money over Boston/SF/NYC.
Paying 'regular market wages' for remote localities is not a winning proposition - it doesn't take advantage of the fact they are great startup, reasonably well financed etc..
But what neighbourhood you live in Boston is a choice.
Picking up and leaving a city is not a choice for most people.
Also the 'cost of everything except housing' is pretty consistent - everyone pays roughly the same a gallon of gas, and an identical basket of groceries.
The kicker is that I'm employed remotely by by a company based in a city whose CoL hit is even larger than the one I currently live in. The NYC range they give is plausible for me as someone who lives in a place that is much cheaper than NYC, but it's not exciting and wouldn't really motivate me to apply.
GitLab's numbers may look right on paper (and they do explain their methodology in the second link, discussing rent indexes, etc), but they're absolutely not comparable to what good people earn in the real world. Good talent commands good wages everywhere. A skilled remote worker is not going to come to GitLab for an average-for-their-area salary.
Even when working locally, talented people make much more than the average for their area. I know this because I've hired many good people and unless they're entry-level (meaning they're good but haven't had a chance to prove it yet), they don't start coming in the door until you're offering at least a 50% price premium over the reported median. In fact, we had a lot of bad candidates come in asking for around that much. The senior people I hired would make more than double the median for their nominal role. The stats on these government reports don't account for seniority, skill, niche demand, etc., and likely include some people that self-report as holding a title when they're really just aspiring to that position.
Once you're a mid-level dev, you should be pulling in a bare minimum of 80-85k, no matter where you live. I say this not as a bubble-dwelling San Franciscan (who surely finds such numbers laughable even for entry-level), but someone who has lived in various parts of "flyover country" his entire life. When local opportunities for at least this compensation are not forthcoming, go contract or remote.
There's no reason for good people to leave money on the table, and GitLab doesn't appear to get that.
They're paying talentless Wordpress dev prices for their mid-tier devs, by local standards. This is in a mid-sized midwestern US city. LOLWUT? No one decent in my city would accept a job there.
"In developing the compensation formula above, we looked at the compensation of our team members which had been set in the past (without the formula), and found out that there was a statistically significant correlation between compensation and the factors that are now in the formula. We purposefully chose to look for correlations with metrics that are probably causal and definitely relevant in people's lives (the rent!). This also has the advantage of letting us work with data that is readily available publicly, as opposed to trying to scour the web for market compensation rates for all roles in all locations. Perhaps surprisingly, there was a stronger correlation between compensation and rent index than with the more general cost of living index available through Numbeo (or the cost of living with rent index, for that matter); and so we moved ahead with the Rent Index."
And, for US locations, BLS data is readily available publicly. While that may not have been a suitable source for a worldwide formula, it would have at least been a suitable reality check on the validity of the formula you came up with to see if it reasonably approximated the variations in market rates within the US.
You're also right that we need the calculator to work worldwide, and I suspect that that may be part of the reason here that the calculator works well on that scale and then not always as well as it should on more local scales. We'll continue to explore how to get better on both scales; the challenge is always that we want to keep things as simple as possible also.
I've made two issues based on your comment here and the one where you shared specific data for Sacramento (thanks for that!):
- https://gitlab.com/gitlab-com/organization/issues/23 to cross check BLS data with the calculator.
- https://gitlab.com/gitlab-com/organization/issues/24 to address the global vs local question.
A) What's to stop me from getting hired in a 0.21 COL area and moving to San Francisco or Washington, DC? Are you going to fire me or pay me $20k a year because I moved?
B) My labor is not worth less and my work is not lower quality because I move to SE Asia or Eastern Europe. This is idiotic.
Nice to "write everything down"
They seem like nice folks, but as an experienced remote worker (with a history of both remote employment and remote contracting), I wouldn't bother to apply based on what I currently know.
I just checked the calculator for a major German city and I can say they won't be recruiting anybody away from the likes of Siemens. Rocket Internet maybe.
Their base is off of NYC, apparently, and they adjust based on location; from what I infer from that, if they are really middling for NYC, their modifiers are all wrong for the way market wages actually vary by location; in Sacramento, the adjusted range is below public sector pay for similar roles (salary alone, not considering benefits, etc.), much less private sector pay.
The thing is, their basic methodology is arguably somewhat sensible, but they could (at least in the US) use BLS numbers for job categories by location to get modifiers that actually reflect local markets and keep their pay consistent with respect to local markets. By choosing an index that evidently is a poor proxy for variations in market pay, they end up with pay that doesn't make sense.
That said, acknowledging that a lot of things are different from when I was starting out, I can't imagine walking into my first tech industry product management job as a remote worker.
I love the Gitlab product and what they have done to Git hosting. But, their salary structure is really screwed up.
Depending on which of the 5-6 applicable job titles listed by the Bureau of Labor Statistics's Occupational Employment Statistics [0] one chooses to apply, my salary is between 40% and 120% higher than the "average mean wage" (itself an inflated statistic) in my area.
This is true not just of myself but of all the good candidates I've hired. Good people don't work for average rates, especially not good people who are capable of doing remote work.
GitLab is doing itself a big disservice by targeting the "average" local wage.
There's a very important difference between: "SFO is very expensive so we have to pay people who live there more" and "we can pay people who don't live in SFO less"
This post is now fodder that can be used against me the next time I advocate for remote work at a company.
Edit: it took 10 minutes for two people to pipe up on Twitter, one specifically suggesting remote work made the ops failure more likely:
What's tone-deaf is the vitriol some people are spilling in this thread.
But you're right, the timing of this piece was probably not ideal.
...and it's not like any large web company hasn't had some howlers over the years.
Could you elaborate? I recently heard about GitLab probably less than a week ago and was under the impression they're a great company to work for.
Like any company, we have our own business operations issues too but it's still one of the best jobs I've ever had. We're pretty open about our flaws, to the point where the CEO even has a page that lists his out: https://about.gitlab.com/handbook/people-operations/ceo-pref...
I liked their transparency a lot too! But it's neither possible nor acceptable to keep six hours of data loss and eighteen hours of continuous downtime "to yourself".
Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)
This is almost completely on their admin staff, maybe other people aren't willing to say it, but I will. Test your backups. Or at least make sure they're non-zero in size. It should really be Operations 101.
Whether you do this automatically or manually by setting a reminder on your calendar once a week or even month, doesn't matter. Something this simple would have solved their entire issue. I do this and we run a much smaller shop than GitLab. Heck, if we were larger I'd have hot spare database servers in another datacenter in case the primary got nuked by disk failure, network outages or mistakes.
"Automated failover can be achieved with pacemaker alongside STONITH network management. Keep in mind that application servers need to be prepared for transitioning to the new network addresses.
In this situation you can also opt to synchronize the database via a database specific protocol instead of DRBD. In the documentation for each database you can find out more about the options for MySQL and the options for PostgreSQL." https://about.gitlab.com/high-availability/#filesystem-stora...
That's what lead to the problem. We were trying to fix it, but ran the wrong procedure on the primary instead of the secondary
Thus wiping the primary, instead of what might have been left on the secondary.
The secondary was already removed at that point.
Basically the procedure was the following:
1. Secondary falls behind too much, stops replicating. At this point you need to manually re-sync the whole thing
2. A re-sync requires an empty PostgreSQL data directory, so this data was removed
3. Re-sync doesn't work, leading to the other problems
4. At some point team-member-1 thought previous re-sync attempts left data behind, so team-member-1 wiped the data directory again to be sure; except team-member-1 ran this on db1.cluster.gitlab.com (the primary) instead of db2.cluster.gitlab.com (the secondary)
I'm just genuinely unsure what a better outcome would have been here. (It's certainly a process failure that no backups existed other than the manual 6-hour-old snapshot, but I'm not sure you can do much better than automating that.)
Maybe being fully remote helps to let stuff like this "slip through the cracks". (Not great phrasing, but I don't think anyone made a conscious decision "loosing X hours of data is ok", and nobody questioned what goal the current practices could (not) achieve.) I don't know, but it seems at least a question one might ask.
(Or are we just saying that they should have been taking backups every 15 minutes or hour or so?)
In the event of accidental deletion, the most recent snapshot will be restored and the transactions up to a certain timestamp will be replayed on top of it. If done correctly, this can prevent all but a few minutes of data loss.
AWS Aurora (and possibly other database-as-a-service providers) has this functionality built-in.
More frequent backups would of course help, if their impact is tolerable. They also need to be tested to actually exist and work.
At least LVM snapshots apparently are cheap enough that they can be done out-of-schedule just to get slightly newer data to staging, so they likely could have been done more often (but they probably weren't thought of as backups, which is why 24 hours seemed enough and nobody had a prepared plan to get back to production from them).
Similarly, Azure-side snapshots are mentioned as not enabled for the database hosts. Maybe that's not viable to do (they had performance issues with Azure, and I don't know how much overhead they case), maybe just something they forgot to set up.
Others have asked "why is there only one replica", which also seems like a good question (but my experience with database replication is close to non-existent, and maybe the higher load of more replicas would have caused other issues. Don't know.).
My point is more that I don't have the impression that the 24 hours are a figure that they arrived on by evaluating what "service level" they could achieve, but more a result of someone at some point setting up a backup with some interval. I don't think 24 hours is a figure where they can say "that's the best we currently can achieve", or even "that's what we planned for", but "that's what we have because that's what we have and nobody has paid closer attention". There is at the very least a cultural angle to it, which is why how they work could be relevant.
Then as someone else mentioned you can stream your replication log to storage elsewhere giving you the ability to do point in time restores.
Anyone with more experience with large databases have anything to add or any concerns with this sort of scheme?
If you look up the doc, they have like 6 different places where they looked for backups and each point is like "We should have these backups, checking for them now... woops, it hasn't been working."
The only reason GitLab's data loss wasn't even more disastrous is that an employee just so happened to take a manual snapshot 6 hours before the incident occurred.
Now, I'm sure we've all been in situations like that. I don't think it necessarily reflects on the individual technical employees of GitLab. I think it says more about the organizational and management resources at the top. It was their job to ensure that the appropriate resources were committed to data integrity, and they failed to do so.
90% of us probably work at places with similar problems, but publishing "the secret to [managing technical employees]" fresh on the heels of such a spectacular organizational failure is probably unwise.
> So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.
It's not true that having zero tested working backup/restore strategies happens to everyone. It's a catastrophic failure in process. Let's not blame any specific engineers, but it seems totally appropriate to blame the CEO (who was the subject of the puff piece!) and decrease your trust in the company appropriately.
writing things down is not the problem... failing to read the things they write down is the problem. they probably don't have time to read, because they are too busy always writing. they can't find what they need to read in the sea of endless wasteful and pointless drivel that never should have been written down in the first place. you're all idiots.
How is their remote working practices at all related to the database disaster?
For that matter, how are either of those related to source control and rebuilding from old source? What would rebuilding source help with when you just deleted your production DB?
As a remote worker myself, I'm always keen to hear about how teams make it work. I'm also a huge advocate for distributed teams (and nobody having a commute), so personally I'm really glad this link was shared today.