What it was like working for Gitlab

As you are almost certainly aware, the majority of racial problems in the US boil down to the very simple fact that black people are on average of low socio-economic standing; and if there is one thing the US system is geared against its any person of low socioeconomic standing.

Put succinctly:

You might consider that its ok to pay differently based on if you’re in Los Angeles or Bryan, Ohio- but would you feel the same if pay was different depending on if you lived in Inglewood vs Santa Monica?

We discriminate pay on job role, on job level, on performance. We discriminate on length (tank crew), eyesight (pilots). That is not Discrimination however, because it lacks a historic asymmetric power balance.

remote.com is a good counterexample here: https://remotecom.notion.site/Total-Rewards-Glossary-ec96c10...

location-based pay is (generally) a product of how much the employer can get away with rather than the operational cost etc...

That’s not to say this couldn’t change in the future (sometimes it’s good to ignore the current “how” and incentives / reality). But it’s useful to be realistic. There’s also something to be said for companies taking the risk and hiring in other countries. This increases knowledge (not to say the company is “better” but it probably has some reasons for why the local talent joins), sows the seeds for future companies there etc.

This is a direct consequence of people wanting WFM but not understanding the implications.

Going into an office gives employees pay leverage due to cost of living in that locale (assuming they live in a top pay market, SF/NYC/etc).

But if you have an entire Remote company & culture, now - companies can hire in any locale and pay lowest common denominator (even if that locale is half way around the world).

Remote work turns everyone into effectively an independent contractor competing against all other talent from all around the world.

In office means you’re mainly competing against other people in your locale.

B2B contracts are a thing. It's very common for many devs that work remotely for other companies.

There's also companies that offer something like "Payroll as a Service".

Still B2B is superior in every way, and can often net you much more money than just being an employee due to being able to leverage tax-optimizations.

> my salary was around €120 000 per year [...] For The Netherlands this is a good salary, and you'll have a hard time finding companies that offer better and let you work from home full time.

For his country, he was well paid and worked from home 100% as well.

> But if I had instead lived in the Bay Area, I would've earned at least twice that amount, possibly even more. Not because I am somehow able to do my job better in the Bay Area, or because of any other valid reason for that matter, but because I would be living in the Bay Area instead of in The Netherlands.

He says it was "not fair" that he wasn't paid the same salary as people in a different country.

There are no Victim Points to be scored here.

The author has a poor understanding of economics.

Just because these traits are not protected by US law does not mean that they should not be valued. They are also traits of human diversity and discriminating based on those traits is still discrimination, just not discrimination against US-protected classes.

I suppose what you're getting at is that you can't change your race or skin colour, but you can change your location. That isn't really the case if you're Ukrainian, or just from a country with a weak passport.

This is why women who want to get pregnant don't get hired in the USA (extra costs associated with postnatal (ETA: maternity) leave). And yet that's not okay. So your basic rule of thumb is not quite comprehensive.

All companies will take advantage of maximizing their profit or reducing cost, but it's a slippery slope once a subjective metric to determine value is used.

I for one live in a "low" CoL, but my AWS bill is just the same as a person in NY, SF or Geneva, should I also expect a discount because "my income is lower"? Or is it only fair to be billed equally, because the value all of us get is the.same ?

Turn the tables, if a dev in India, Romania or Mexico is delivering the same value as one in the US or UK should (s)he be paid any less ? Why ?

For example, now I live in the US and although my salary is about twice what it was in England, my lifestyle remains about the same, worse in some aspects, and better in others. I'm almost certain OP lifestyle would have taken a hit living in Bay Area, even with a higher comp.

And this is simply comparing income, not accounting for many other variables (social benefits, taxes, culture...). In fact I'll probably take a big hit in comp to move back to Europe so that I can live a more comfortable lifestyle.

As for location, the motivation is stated here, and in my opinion, it really makes sense: https://handbook.gitlab.com/handbook/total-rewards/compensat... Imagine they would pay Bay Area salaries everywhere including Philippines or Ukraine. People there would be in "golden handcuffs" and burn out instead of changing job when it's time to go. This would produce terrible results for both the company and people. And of course, if they chose to pay some kind of world-median salaries, then they wouldn't hire people in California or London at all.

There are other controversial things said about product managers being unneeded, and that the company shouldn't have more than 100 developer for a product.

I think it's pretty clear that the author is a great autonomous engineer who would be happy and very successful in a (relatively) small startup. Maybe if not for the tax rule changes waiting, he would have left the company earlier, and would write a happy article about his experience there.

This gets us back to the location-based rates again by the way:). Imagine how difficult it would be for the author to leave the company if in addition to stocks, he would have had $300k compensation in Amsterdam.

At the end of the day, it winds up being a choice between

- Pay someone based on their location - People in lower COL areas complain, but are (in theory) being paid an amount with the same buying power as someone in a higher COL area (being paid me) [1]

- Pay the same regardless of location - People lower COL areas are making more buying power, but people in higher COL won't even work for you; because it's not realistic to pay the higher COL amount to everyone [2]

There's no solution that makes everyone happy.

[1] Not really true, since it doesn't account for a lot of factors; such as retirement savings, etc.

[2] Not strictly true but, if you pay the higher COL amount to everyone, then you can hire fewer people for the same amount of money.

Exactly.

Compare it to something else, like paying people more if they own a sports car.

You’ll be telling, showing, raw stats, massaged stats, appeals to authority, what have you, but if the only thing the directors care about is features, it’s all for nothing.

After 3 years it finally became a problem because they noticed that releases kept failing, but I don’t think it had anything to do with me so much as someone typing ‘how to stop my releases failing’ into Google.

I don’t know how to solve that, as I’m going through the same thing again now (with a different subject).

The trouble with these stories is that they’re n=1 anecdotes and we only get one side of the story.

There’s an implicit claim in many comments here that we need to assume that the employee was actually a higher performer, didn’t need managers, didn’t deserve a PEP and so on. That’s understandable given that we tend to put ourselves in the shoes of the person writing a piece and being anti-corporate is always popular on HN.

However, those aren’t safe assumptions in cases like this. I’ve worked at a couple early stage startups that acquired early employees who couldn’t (or wouldn’t) grow with their role and the company. It’s common to keep early employees around out of respect for their past work, a belief that they hold difficult to replace historical knowledge, or simply because they’re well connected to founders and other early employees who have grown into leadership positions.

But in reality, simply being an early employee and being involved with early important projects doesn’t necessarily mean that person is the best person for the job or even a good fit for continuing to do it. Some of the early stage employees I worked with were great at cobbling something together from scraps and keeping it functional with a collection of cron jobs and manual interventions, but their operating style doesn’t work at a bigger company at all. If they can’t adapt and change or they become disgruntled about having to work on things the way you have to at a bigger company, they start to become more detriment than help. That’s just one example of many different potential failure modes of early employees as companies grow that we don’t like to talk about.

> I wanna say this story is pretty standard, and probably a big part of the reason why people in big companies often don't do great things.

It’s “standard” in the sense that every early company has a number of early employees who don’t grow with it, but it’s not the only or even a common fate. GitLab has plenty of employees who have been there for a long time, but you’re not hearing about them from disgruntled blog posts. Consider the selection bias when reading this.

As for the claim that they can’t do “great things”: I’ve used GitLab for a very long time and I disagree. The product continues to evolve. It’s not a stagnant product at all.

One tell-tale sign is endpoint security and using his own device. It's the kind of permissive cultural thing that does not scale because of compliance issues and developer productivity overhead. But it's very hard to wrench these long-time devs away from their preferred Linux distribution which requires conditional build stuff everywhere to support. Use a work computer for work, let them monitor the updates and stuff, as long as they're not using the webcam to record you who cares?

The database backup story - my guy you were on the database team. Backing up the databases is job 1. You can't just passive-voice away "oh there were no backups". But of course he's more interested in fighting about sharding architecture than actually keeping the site running day-to-day.

His big takeaway is that Gitlab didn't spend enough time on performance for their hosted offering which was a huge money loser. Just because he thinks performance stuff is fun to do, if the hosted offering is a money pit of course they're not throwing more resources at it. You have to make an actual business case for why your thing is more important and makes money more than another project. You don't just engineer in a vacuum for the fun of it.

I wonder if this is what is behind years old stale issues of gitlab, especially regarding Gitlab CI. From the outside you get the impression, that they completely lost ability to build great things, since they do not seem to care to fix years old bugs that still come up again and again.

It’s obviously not a silver bullet but at least then leakage must be hostile action or incompetence by transferring sensitive data outside of authorized channels.

How you do this is of course up to you. At one end of the spectrum is just relying on your memory. At the other end is using NixOS https://nixos.org/ to get fully reproducible builds anywhere you go. Between these are a vast field of options. I know a guy who maintains an Ansible file set to `host: localhost` which installs everything he wants from that file. For me, I just stick with the latest Ubuntu version and maintain a few shell scripts [1] that install 80% of what I like to have on a new install.

If you like the scientific approach, you can install something like https://atuin.sh/ and do some statistics on what programs you actually run most frequently based on your long term shell history.

[1]: https://github.com/hiAndrewQuinn/shell-bling-ubuntu

Also, companies have no control over corporate IP that land on an employee-controlled PC anyway.

While I agree that separate work and personal computers might help in this regard, in my opinion it only helps a little.

I believe that behavior related to separating work time and personal time is influenced primarily by one's own habits, abilities and, in the end, decisions.

I've worked with small and startup healthcare/healthcare-tech companies who not only permit it, but have formal BYOD policies... when they are dealing with PHI.

>It doesn't matter whether you're paying somebody in the Bay Area $100 000 per year, or somebody in the Philippines, because the cost for you as a business is the same.

The cost for you as business is NOT the same. Start from the taxes part, that 100k is what the employee gets but the cost for the business is higher depending on the country/area because they also pay tax on top, social benefits etc. Also need to have in many cases a local business, doesn't matter if that is virtual etc, since they need to adhere to local laws thus having resources supporting that etc.

So while I'm on the employee side here as I'm also working on a multinational coorp with global role yet paid with local standards, there is more than meets the eye

These positions are phenomenally competitive. You have swung open the doors to the world, and said "Give me what you got!", and you reap the benefits by having a much, much larger pool of applications and talent to sift through, with a lot of truly exceptional people in there - mostly very intelligent, very driven people from poorer countries.

That's how supply and demand works. If you are offer to pay people more, you usually end up hiring a higher quality candidate.

when i looked at their salariy calculator i figured that i could not live on the salary they would offer for my location. school costs alone for each child are as much as i pay for rent.

You just need to ask yourself if offering incentives to get your employees to move to hot job markets is in your benefit.

Kids would have to leave school and all their friends behind, wife hasn’t finished education here - there’s those reasons as well.

No. You got it all wrong.

You're asking for equal pay for equal work. If all your team members are paid in full but you, in spite of doing the exact same work, are paid a fraction of what they are paid, then something is terribly off.

Your contribution to your team does not depend on where you're currently located. How much time you waste on commute does not change the expectation placed on your output. If your office location is prohibitively expensive that means your company needs to sort their mess and work on a location that's more affordable. It makes no sense that you need to subsidize your employer's bad office location.

If the candidate had other better offers, then they can either reject the offer or propose a higher counter-offer. If the company received a counter-offer and chose to accept it, then this amount becomes the new minimal amount.

Over a period of time, if there are enough counter-offers (or rejections), this continues to increase. Since, the two (or however many) locations don't necessarily have the same demand or supply, it's inevitable that some location will end up with much higher compensation than others. It's the nature of free market and why some companies engage/d in hiring collusion (see https://en.wikipedia.org/wiki/High-Tech_Employee_Antitrust_L...)

Why is that? It isn't obvious to me. It seems like an excuse companies use to pay people less, especially if the company is based in a high-paying city.

For some lucky people that does work.

If company X was paying N for a SF engineer, and suddenly it finds out that it can pay N/2 for an engineer that’s as good as the SF one (but lives, let’s say, in Mexico)… well, jackpot for company X.

Why should someone living in Indiana make less money than someone living in NYC for a remote tech job?

Both are located in the US separated by a 2 hour flight. Both are in the same timezone too. For a remote company where you have employees spread around the US, there's no difference to anything here.

If you want developer salaries to be the same in all locations for the same level of work for reasons of fairness and anti-discrimination, you should ask why salaries of developers are higher or lower than other roles. Developers get paid highly because of supply and demand. Why not advocate for all workers, regardless of role, be paid the same - this would also be fair and anti-discriminatory. It seems we want to benefit from the market economy on one side and then want fairness on the other side. The logical conclusion of this kind of reasoning is communism where all workers at a company are paid the same, regardless of role. We know how that worked out.

If all developers, regardless of location, were paid the same, companies would much rather hire all their developers in a single location. Why bother hiring in locations far away? Jobs would have never flowed out of high paying locations to lower paying locations.

The Bay Area has the highest salaries because Silicon Valley has had many years to develop and the vast majority of tech companies are based in the Bay Area. Many companies are started in the Bay Area because people who work together in one company often break off and start another company. Many people at these successful companies have become wealthy and this has driven up home prices. This has made the Bay Area one of the most expensive places to live. Other workers cannot afford to live in the Bay Area and so there is a shortage of labor. This drives up the cost of every thing, including restaurants, groceries, and personal services. If the Bay Area does not add substantial housing, it will continue to see companies move to other metros.

In the US, salaries are different based on metro and state and are based on the market economy. Companies move to offices to particular metros if they is a healthy supply of workers and supply is greater than demand so that it is more cost effective. This location competition is healthy.

The role of any government, whether it is metro, state or country, is to create thriving economies so people want to move to that location. This means investing in critical mass in particular industries so many companies in that industry want to locate there. It also means ensuring other costs are low. In the US, health care insurance cost between $24k and $36k/year for anyone with a family. This is higher than the full salary in other countries. If the US does not figure out how to solve its health care costs, it will continue to see jobs leave for lower cost locations.

despite having the least in absolute terms, it is possible to have a great life with the mid-to-high level income in IT there. can't say the same with what most people earn in europe.

> Rails is a great framework and the productivity is unmatched, but with time a 2-3 years old Rails project is always tricky to maintain.

It isn’t any less tricky than a Django or Express project. With any codebase discipline and regular tending to the garden of code is important to prevent weeds from growing.

If anything that Ruby (and Rails) has going against it is still the raw language performance and higher memory requirements/usage than its counterparts, which makes it less desirable for workloads that require low latency or small memory footprint

Stripe, GitHub, and Shopify are the shining examples of Rails scaling in all kinds of ways without problem.

I would like to challenge that.

Couple of points. 1) You probably mean Ruby on Rails and not Ruby and 2) that's not true of either. Ruby or Rails "entire programming model" was not based on thoughts around I/O at all, a huge about of the driving force behind Rails was DHHs ideas about developer happiness, which is intrinsically linked with speed of development.

Discussions about I/O /do/ happen in the Rails community because it's important when you're running a web server which is a lot of the use Ruby/Rails is involved in, but it's not the sole (or even a major) focus of the decisions made for the language or framework.

> Well now, both on Node.js and JVM we have programming models which say that - while I wait for IO / Network let me do some other work or service more requests (Async / Webflux / Coroutines).

Right, but if the goal is to maximise CPU usage of your web servers then you can completely do that using Rails, and have historically been able to do so by spinning up multiple processes in the same way you'd spin up multiple threads in, say, the JVM. Not as convenient perhaps, but a model that's been in use since at least the 90s when I started out. Luckily it turns out that web requests can generally be run in isolation so IPC isn't usually an issue (the same reason why multiple physical web servers is simple, just as multiple processes is).

> IMHO - using Ruby/Rails in 2024 can be wasteful, but of-couse for the right situation it can be a good choice. (Just) For example an enterprise app where you know the number of users will be limited, or when you know the development speed is paramount or where you want to build a quick proptype to test the market out. Rails is a great framework and the productivity is unmatched, but with time a 2-3 years old Rails project is always tricky to maintain.

Wasteful as a generalisation is blunt. Of course it can be wasteful, but so can writing your app in rust, it just depends where the pressures are. I'd suggest that not launching is a far bigger concern to most people at an early stage, and even mid to late stages developer speed is one of the most difficult things to scale.

Personally I'm the CTO of a company that uses Rails (backend, React frontend) and developer speed is /hugely/ important to us. We're always constrained by engineering resources and the speed of Rails development continues to be a huge win for us. We do have scaling issues, but 99% of the time they're not Rails issues, they're in the database.

I don't believe that is a legal requirement, anywhere. Remuneration is based on many factors, which can include the cost of living. A company will not be able to hire someone in New York City, for the same price as someone in a less expensive jurisdiction.

This isn't discrimination, it's simple economic reality.

So you’re saying, we should be paying engineers in Europe and in the US the same as an engineer in LATAM or India or Asia that has the same level of experience and skill.

The only way to be non discriminatory is to have a standardized formula of compensation that takes into account cost of living (rent, food, healthcare, taxes etc) where the final take home pay in locations around the world are equivalent - which I believe should be the case at most companies

It works both ways, anyway. If those friends had moved from Thailand to Switzerland, would it be discrimination to pay them more?

[0] https://www.canada.ca/en/revenue-agency/services/scientific-...

As a company you need certain abilities, and you pay whatever the market decides these abilities are worth, and nothing more. Depending on location, the market will set a different price on these abilities, so you pay different.

You are supposed to pay them the minimum amount it takes to get them to show up to work. When someone moves to a less competitive market, where getting another job is harder, then they are more likely to show up for lesser pay.

And remember that a country may have a less competitive market, even if the workers are remote and not seemly bound by a local market, because governments often love to put up huge roadblocks when it comes to international hiring. If you are being paid less than you were in another country doing the same job for the same employer, this is almost certainly why you have agreed to take a pay cut.

I don't think this make any sense on so many levels. First, "abilities" are not a good way to think about wages. If you hire a neurosurgeon to do your gardening, you won't pay them more than a run of the mill gardener.

Rather, you as the employer compete against other employers on different markets in a fairly classical supply and demand situation. The "abilities" of an compliance expert with tech skills did not change much when GDPR was introduced, but as all EU companies scrambled to figure out the regulation (and the DPO role was popularised by fiat), the compensation went way up.

If the employee can participate in e.g the SF labour market, you have to pay a competitive salary in that market, if not you don't have to. As long as there are barriers, e.g a on-location worker in SF has more opportunities for whatever reason, the location premium makes sense.

To take your example in the opposite direction. Let's say a east-european company want to expand into the US and open up a sales engineering office in SF, and want their best sales engineers to go work their, it would be completely insane to not raise their wage. "We pay people after ability here you have 40k USD, have fun finding housing".

But I am sure no one on management level wants to do this decision. So you half-ass a product.

This kind of situation can be frustrating for people working on the product itself (at least when they care). And each new developer that joins the product will probably suggest: "Oh we should care more about scalability." And each senior will be: "yeah yeah, i know..."

I literally bought licenses from gitlab before realising that the seat cost is the same if I let them host it or we do it. So why take the infrastructure cost?

Being able to work from where I happen to live and I have thought about whether I would like to work for them.

That it's a good product would certainly be a motivation. However, frequent incidents being resolved fast must be incredibly stressful for the staff. I prefer to stay where we make heavy releases only a few times a year for regulatory reasons. (Even though stress levels can raise until the last MR for the release has been merged.)

Location-based pay pays people more if they're in a market where they have other options, so they don't go to those other options.

In essence, if you have location-based pay, you're incentivizing your employees to move to an area where they'll have vastly more employment options than to stay with you.

Also buying power is just not the same based on your location. A hundred dollars in the US and in the Netherlands will literally not buy the same things.

I'd hope a high performing, vastly experienced engineer in NL could aspire to that, though they might not get the same benefits or future opportunities.

[0] https://en.wikipedia.org/wiki/Prime_Minister_of_the_Netherla...

[1] https://en.m.wikipedia.org/wiki/Mark_Rutte

Literally a couple of weeks later after putting this live (I think it was a Nagios check) it alerted that the backups had gone down from many 10s of GB to a few k in size. I can't remember what had gone wrong now, it was years ago, but we /did/ catch the issue.

Obviously this isn't a comprehensive check, I later wrote a tool that pulled backups down, restored them, and did some sanity checks on the data, but the quick hack worked as an interim.

I think people mistake “manager” as being people without tech skills. I think a tech company should have managers to create alignment and those managers should spend part of their time “managing” and part of their time still honing their tech skills.

We don't pay 'list' for gitlab SaaS.

It’s the classic “last founder or first employee” problem. If you are an employee at a startup, have no illusions: if “they”[1] don’t give a shit about you now, you can be sure “they” will not give a shit about you later.

[1]: the company is and always be an inanimate and abstract object, your boss and colleagues will change

Choosing to not care is much harder as we have all had years of education conditioning us that we need to achieve more and to be the best.

I have trouble understanding this.

Do these startups consciously decide not to have backups (for time/money reasons perhaps)? It makes no sense to me.

I've worked at some startups and most senior engineers know what's necessary to keep production systems running reliably. That includes monitoring and backups.

Could it be that this is entirely normal for startup with less experienced engineers doing things for the first time?

I'm actually a bit on the fence for the 10 year exercise window, as I'm not sure it's always a benefit. As I understand it, startup exits timelines are on an upward trend, so in 10 years will the average exit still be below 10 years, and even then are you working for a company that meets the average timeline. Then, from a taxation perspective, if you do want to exercise before the 10 year expiry, if it's cost prohibitive now, it should be bankrupting after a few more investment rounds and increases to the 409A valuation.

So I'm curious if there is a side effect here that the 10 year exercise window actually leads to many more shares becoming even more cost prohibitive to exercise right before the company reaches a liquidity event and as such not exercised (if you're in the early stage employee group).

All large successful user-generated content applications eventually need to shard, and most of them are read-heavy just like gitlab.com. The two primary motivators for sharding in this case are database size and blast radius, rather than needing to scale writes.

For database size, even if your DB still can fit on one beefy server, operational activities like backup/restore and new replica cloning take an increasingly long time. This in turn becomes a major risk to the business -- for example he even discusses how long the restore took when he had to restore the prod DB from the staging DB.

For blast radius, it's the ability to have outages (e.g. hardware failure) only affect a small subset of users. Or for a more extreme situation, the massive difference between "I accidentally dropped the database" and "I accidentally dropped a database".

Sharding does indeed add terrible complexity to the application as well as operations. But in any case, eventually the data size motivator becomes unavoidable for user generated content: you eventually must shard, and the longer you wait, the more difficult it will become.

To put it differently, try to think of an application that does all the things you _don't_ want to be doing if you're going to (or want to) shard. Then imagine that that's basically GitLab, coupled with the load patterns simply not justifying it.

I've always been sceptical of this myself - how do you value that code review where I stopped a junior engineer's XSS getting into production? Incredible value? Or 10 minutes of work?

According to them, other frameworks require you to fetch all required data before passing it to the view layer, thus preventing this issue from coming up on the first place.

    Project.limit(10).each do |project|
      project.members.count
    end

    [...] FROM members WHERE members.project_id = X [...]

The problem this results in is that in the above code you'd run a COUNT query for _each_ project. Instead what you'd want is a single COUNT that groups data per project, such that you can then pass that data along with the `each` call. This setup would only require 2 queries, instead of 20.

To eager load data in Rails you have to explicitly opt-in, resulting in something like this:

    Project.includes(:members).limit(10).each do |project|
      project.members.count
    end

The solution here is to separate querying from the row instance types, e.g. a "Project" type can't query data itself and instead requires it to be passed in. This makes it much more difficult to create N+1 query problems.

> When you travel to a country cheaper than yours, do you pay every restaurant, store, and taxi driver the same way you would at home?

> No, you don't. Because that would be foolish. You pay local rates, which are less.

But if you could magically teleport to store or restaurant anywhere in the world, would you pay high prices to go to SV restaurant or store? No, that would also be foolish.

So local prices make sense of physical businesses because people pay premium for convenient access, for not needing to fly half-way across the world to do your groceries or whatever. But that same does not apply to full-remote developers; the work(/value) is getting delivered to the business all the same regardless where the employee happens to be physically sitting.