Comment by hn_throwaway_99 | Hacker News Reader

Comment by hn_throwaway_99 | Hacker News Reader

Parent Full thread

hn_throwaway_99

Decided to post a top-level comment because I think it's very important that Gitlab do a "root cause analysis" of why they made such a poor decision. Just saying "we're holding off for now" is not going to convince me that you guys have not lost your compass.

For reference, there was a very telling comment by user jahlove about how Gitlab could be so out of touch (https://news.ycombinator.com/item?id=21349591):

--------

> But someone up the chain of command thought we could get away with it

That person is Paul Machle (CFO):

"I don’t understand. This should not be an opt in or an opt out. It is a condition of using our product. There is an acceptance of terms and the use of this data should be included in that." [1]

[1]https://gitlab.com/gitlab-org/gitlab/merge_requests/14182#no...

--------

That GitLab comment thread is very telling. Basically, here is what happened:

1. It's clear engineers had concerns about the feature, wanting it to at least have an opt-in.

2. But then the CFO, who clearly does not anticipate the backlash this will cause, basically just gives an "tough shit" response.

3. What's scariest IMO, though, is that the Scott Williamson, VP of Product, then replies "@cciresi if we follow Paul's guidance and just make this part of our terms and conditions, are we covered legally?"

I've seen this at many organizations in the past. It's generally called the HiPPO problem - the Highest Paid Person's Opinion. The CFO wanted this done (obviously for financial reasons, he's the CFO after all), but the VP of Product, who should be more "in the weeds" in terms of having the pulse of the customer, instead deferred to the CFO and tried to appease his desire.

Gitlab, I think your organization is off track. You need to make a broader statement that shows you have a real understanding of the problem than just "Oops".

It might have been better to simply reference the discussion without singling out the person involved. At the end of the day it was an organizational decision. It's to GitLab's credit that there's enough transparency to even see this; I don't think it's too cool to turn that into a finger-pointing tool. As a developer, we don't do this with bugs. Perhaps bad business decisions are not any different.

(For the record: I don't know this person, I am not this person, I'm pretty sure I don't know anyone at GitLab at all.)

I understand your viewpoint, and to be honest I was on the fence about posting names, but I decided to for 2 important reasons:

1. First, the info is totally public and easily viewable in any case.

2. I think there is a danger in just saying "It was an organizational decision." When everyone is "responsible", then no one is. And in addition, both of the people mentioned are senior execs at the company. Having to deal with public scrutiny is one reason they get paid more.

I do give GitLab a lot of credit for being transparent, and for (at least for now) backing off this decision. But I hope they are able to do a deeper root cause analysis of what went wrong. While obviously on a whole other order of magnitude, Boeing didn't fuck up because MCAS was a faulty piece of software, they fucked up because they let financial concerns take precedence over engineering ones.

EDIT: one other thing, since I brought up 2 folks who I think made poor decisions, I should have pointed out Lukas Eipert (@leipert), who was the engineer who originally brought up his strong objections. Even more telling, when Eipert was asked to re-review the code change, he explicitly asked to be taken off the review - his sentiment was clear.

The whole point of transparency is to facilitate accountability, so it does not help to shy away from accountability in service of encouraging transparency. Of course, transparency is definitely something to laud, but it's only half of the ideal.

You're putting pressure on the norm that organizations should be sharing this sort of thing. If the norm were strong, the pressure wouldn't keep people from sharing similar things in the future. But since the norm is weaker we need to be careful to nourish it, not pushing it harder than it can stand.

(I wrote more about this: https://www.jefftk.com/p/responsible-transparency-consumptio...)

This is a person in a C-level position. It's very much inherent in those positions to take responsibility in situations like this. If the comment had been made by a random fellow team-member without any authority, I would have been more inclined to agree.

The difference here is that bugs generally lack motive and agency. Bugs are often unintentional.

When faced with bad intentions, it is imperative that a community name offenders. It is, as others replied, an important distinction for accountability.

A CFO concerned about valuation and experienced in negotiating contracts should be thinking:

1) Fortune 1000 companies and the government will never accept this. I need to make sure the product team is designing a way to allow these customers to shut this off.

2) This is more risk than we will gain from tracking individual customers. We can't afford to indemnify our customers against these third-party tools. How will we address GDPR, CCPA and future regulations if we don't allow an opt-out?

An organisation cannot make decisions. At the end of the day, it is always a person. In this case, maybe it was the wrong person. What is the trouble in questioning this process?

Thanks, we started our retro 3:44pm Pacific, slightly before your post.

We looked into making it public at 4:05pm but by then it already had confidential security information and customer names in it.

There will be multiple lessons, right now we've identified 15 points to evaluate and 9 people contributed to the document.

this is about it and it's not much of a secret anymore internally, but what do you do? this is just the MR you found, but they're everywhere. unfortunately Sid has known about the HiPPO problem for some time but is preoccupied with everything else and for some reason just won't do anything about it (if it's not gotten to his level many times, someones hiding it from him), and people are getting sick of the tone deafness and surprise when things go wrong. GitLab used to be a mission and values driven company, but now values apply to directors and below and everyone else just kinda does what they want, lean on their title to pressure people, and when there's pushback they say you're blocking efficiency and results, then complain about you ¯\_(ツ)_/¯ you can see everyone was against this and raised red flags at every step for the past 5 months but of course that didn't mean anything and the rollout was horrible. the people working at GitLab are about as dedicated to your trust and success as it gets, but execs are eyeing that IPO really hard and everything else is whatever. also no offense to Scott but you can't run this like a big corp and a good place to start to improve the product as the new head of product is to fix the backlog of hundreds of functionality deficits and bugs and talk to your customers to see what they need, not get tracking. i don't know how we teleported to this place where getting tracking is a priority because otherwise you can't find ways to improve the product because the last i looked there's a backlog of really critical things going to like version 13

Agree re: HiPPO problem: https://drodio.com/counteracting-the-cultural-hippo-problem-...

Reply on news.ycombinator.com