Gitlab ‘rethinking’ third-party telemetry
gitlab.com
gitlab.com
If you want to collect telemetry, fine. Do it on a first-party basis, distill it down into one file a month, and let admins upload it if they like. But the third-party script is a complete nonstarter.
If that was the approach to telemetry, I think most users wouldn't have a problem. But currently the word alone makes us rethink about using the product at all.
At least for now, it appears that the self-hosted versions of GitLab (both CE and EE) will not be getting telemetry scripts. This is only for gitlab.com (again, for now). See the table on their ticket[1] for the issue
We are not adding Pendo or Snowplow JS snippets to self-hosted versions at this time. We are carefully considering the appropriate opt-out functionality for telemetry that will work best for our customers, and we will clearly communicate to those customers when any change is made.
So ... opt-out?
Yes, for now. But I think everyone can realistically expect that it will be included there eventually.
Those of us who find this unacceptable should probably start planning their mitigation strategy now.
I would have thought that the way they rolled this out was obviously so ill-advised that they wouldn't have been likely to do it, but they did.
Also, they initially were going to include the self-hosted installations in this as well, but backed off due to the outcry. So even though it's also obviously a bad idea, they were provably likely to do it, because that was their explicit plan.
Where my thinking is at right now, though, is that Gitlab has just tangibly demonstrated that they don't deserve a great deal of trust, and so I'm not inclined to trust any opt-in or opt-out choices.
Even if they're effective at the time they are introduced (which I don't really doubt they would be), I am not comfortable in relying that they won't change the deal in the future.
Source control is a particularly good example when it's the core IP of your company at stake, and engineers--who can sometimes be a bit prickly about these things--are making the decision on which product to use.
In this case the reaction to me seems overblown, as Gitlab AFAIK has been a well run and particularly transparent company. I felt the same way about the recent reaction to Gitlab saying they want to stay out of the politics of passing moral judgement on every repo they are hosting, which to me seemed exactly right. To be clear, I'm not and have never been a Gitlab customer, so I'm not basing any of this on any personal experience the quality of their product.
Now speculating on how they might do something in the future which impacts self-hosted instances, assuming the worst, and declaring it will mean switching providers... I think the comment would be better off just saying something like; "I'm looking at the way Gitlab is handling this issue and it makes me no longer trust them with my data, so I will be going elsewhere" -- just without the rank speculation.
It would be nice if GitLab provides more transparency on how they are deploying these third party services.
https://battlepenguin.com/tech/removing-the-tracking-image-f...
How is this different from allowing corporate users to access web apps that have google analytics running? We've seen some enterprises block GA, but they are in the minority.
In what universe would javascript running on a webpage be able to access (nevermind modify) the filesystem on the server.
(actually, browsers have a bunch of tools to restrict that, so it's not 100 % if engineered properly. But if it's just "include some scripts"...)
One large website I know includes a sandboxed <iframe> from a different domain on all logged out pages that pulls in 3rd party scripts for marketing purposes. Special data you want the 3rd party to have needs to be passed to the iframe with postMessage from the parent, but the key feature is that compromised 3rd party JS won't be able to wreak havoc on your site.
Here's some info on that approach: https://cheatsheetseries.owasp.org/cheatsheets/Third_Party_J...
Agreed that this does nothing for users that don't want to be tracked, but at least it reduces the risk of the site being compromised by untrusted JS.
What a fucking botched way to do this. Breaking all automation at a moment notice. They should have announced the change long ago. Make an advance clear deadline for accepting the ToS and then after the deadline blocked the API.
I entirely agree with you and your sentiments, but I'm not sure why many are even remotely surprised by these sort of escapades. The more dependent you are on an external business for some function and the more that business realizes it, the more they're going to use their leverage against you to their advantage to achieve their goals (not yours).
That's exactly what we see happening here. When businesses (or any entities) buy-in to external resources like this, always consider: how much leverage those managing that resource have against you, how critical the function is for you, and how many resources it will take to migrate away from that external resource at a moment's notice if you need to. This is a continuous iterative process that should be going through every developer's head with every design choice, external license, "cloud" service, proprietary internally hosted licenses, etc.
Pretty sure a not for profit organization would have even less interest in pleasing their users.
If you merely mean free stuff, not literally, not-for-profit... yeah, free but for-profit is pretty much a disaster waiting to happen.
Rant: Those business models by and large should probably be hit with the ban-hammer, immediately. Of course, given the complete and utter submission of congress and indeed the very organization of democracy to vested interests, the chance of actually seeing a functioning, efficient market in our lifetimes is essentially nil.
It's ironic that traditional communism succumbed perhaps partially due to game-theoretical inherent inefficiencies, and capitalism patted itself on the back without ever really looking in the mirror. So, yay, we get to be the human guinea pigs proving that markets aren't actually intrisincally efficient at all, only efficient within their constraints, which is pretty much worthless given how hard we've collectively been trying to saw the legs out from under this construct. Capitalism is itself one big tragedy of the commons.
For my experience usually clients trust me, and I want them to succeed with my services, and contracts and lawyers are here only for the worst case.
I can’t put my finger on it or express it clearly but free market excuse « dissatisfied users can move, this corporation is responsible only for profit, everyone must lawyer up » don’t look like it’s the good direction for society. I don’t see why free market would be incompatible with an individual sense of responsibility towards society, instead of looking for loopholes to screw people. Maybe it’s a consequence of being global and having not community roots.
That you genuinely want your clients to succeed is probably why they trust you -- you earn that trust.
I'm of the same mindset. I genuinely care about the well-being of my customers and will always work to enhancing that (even to the point of recommending competing products if they are a better solution for a particular customer). I've never lost business in the long term by doing that -- even customers I refer to competitors remember me and have an increased level of trust in me, and more often than not will do business with me in other ways.
But I have an old-school attitude towards business. My goal is not to gain as much money as quickly as possible, but to build a solid business that is sustainable and profitable over the long term. In my experience, that attitude naturally aligns my business practices with societal responsibility.
Many not for profits exist mainly to stroke the directors’ egos and pay them a fat salary without doing much actual good (e.g. most political non-profits like the Clinton foundation).
I don't think people are surprised that these things are possible in the context of "EaaS". It's surprising because they're causing massive problems for their users by making the change so quickly. The relative gain of waiting one month before the change wouldn't hurt them that much, but it would help their customers a great deal. So the surprise is seeing how indifferent they seem to the users. (I know they're revising the decision now. Maybe they can avoid most of the bad PR.)
The free or partially free dev tool industry is extremely competitive and you need a lot of goodwill to stay alive. They're completely free to screw their users, and their users are completely free (except for some lock-in) to hop to the next provider. I don't understand why they would throw away that goodwill.
But maybe I'm wrong and the friction is higher than what I expect.
It may be for a lot of users. But some people are GitHub refugees and only started using Gitlab relatively recently. I am one of those, and haven't been using Gitlab for long enough to have any serious friction against moving away from it.
The nature of tech trends will ensure that developers never look beyond marketing material that espouses how easy and cool the new shiny tech is.
I'm watching developers go all in on serverless as if the generous and copious amounts of free compute that cloud providers are granting on their serverless platforms will always flow like water.
Not all developers. Some of us have been around the block enough times to be automatically suspicious of marketing material that leans too hard on the "easy and cool" messaging.
The issue isn't profit. You might say the issue is greed, but more importantly, the issue is a lack of regard and compassion for your customers, and a lack of understanding of the danger this puts you in [of losing your customers and, with them, revenue and profit].
For reference, there was a very telling comment by user jahlove about how Gitlab could be so out of touch (https://news.ycombinator.com/item?id=21349591):
--------
> But someone up the chain of command thought we could get away with it
That person is Paul Machle (CFO):
"I don’t understand. This should not be an opt in or an opt out. It is a condition of using our product. There is an acceptance of terms and the use of this data should be included in that." [1]
[1]https://gitlab.com/gitlab-org/gitlab/merge_requests/14182#no...
--------
That GitLab comment thread is very telling. Basically, here is what happened:
1. It's clear engineers had concerns about the feature, wanting it to at least have an opt-in.
2. But then the CFO, who clearly does not anticipate the backlash this will cause, basically just gives an "tough shit" response.
3. What's scariest IMO, though, is that the Scott Williamson, VP of Product, then replies "@cciresi if we follow Paul's guidance and just make this part of our terms and conditions, are we covered legally?"
I've seen this at many organizations in the past. It's generally called the HiPPO problem - the Highest Paid Person's Opinion. The CFO wanted this done (obviously for financial reasons, he's the CFO after all), but the VP of Product, who should be more "in the weeds" in terms of having the pulse of the customer, instead deferred to the CFO and tried to appease his desire.
Gitlab, I think your organization is off track. You need to make a broader statement that shows you have a real understanding of the problem than just "Oops".
(For the record: I don't know this person, I am not this person, I'm pretty sure I don't know anyone at GitLab at all.)
1. First, the info is totally public and easily viewable in any case.
2. I think there is a danger in just saying "It was an organizational decision." When everyone is "responsible", then no one is. And in addition, both of the people mentioned are senior execs at the company. Having to deal with public scrutiny is one reason they get paid more.
I do give GitLab a lot of credit for being transparent, and for (at least for now) backing off this decision. But I hope they are able to do a deeper root cause analysis of what went wrong. While obviously on a whole other order of magnitude, Boeing didn't fuck up because MCAS was a faulty piece of software, they fucked up because they let financial concerns take precedence over engineering ones.
EDIT: one other thing, since I brought up 2 folks who I think made poor decisions, I should have pointed out Lukas Eipert (@leipert), who was the engineer who originally brought up his strong objections. Even more telling, when Eipert was asked to re-review the code change, he explicitly asked to be taken off the review - his sentiment was clear.
(I wrote more about this: https://www.jefftk.com/p/responsible-transparency-consumptio...)
When faced with bad intentions, it is imperative that a community name offenders. It is, as others replied, an important distinction for accountability.
1) Fortune 1000 companies and the government will never accept this. I need to make sure the product team is designing a way to allow these customers to shut this off.
2) This is more risk than we will gain from tracking individual customers. We can't afford to indemnify our customers against these third-party tools. How will we address GDPR, CCPA and future regulations if we don't allow an opt-out?
We looked into making it public at 4:05pm but by then it already had confidential security information and customer names in it.
There will be multiple lessons, right now we've identified 15 points to evaluate and 9 people contributed to the document.
Why is this so important? It's important because a lot of that trust is ideological. I trust that the developers of 'git' itself won't be adding telemetry behind the scenes. I trust that the developers of 'cgit' won't be adding this stuff.
Because we're on the same page; despite the fact that theoretically the next commit that hits the repo could be the one that sends my keystrokes.
The first time you pull this sort of stunt you've instantly erased all of that good will and now you're forcing people to comb through your releases and conditions with fine teeth because you've shown yourself to act against user interests.
This doesn't scale.
That's why the backlash is bigger than you expect, not necessarily because of the magnitude of the change, but because you've positioned yourself as a threat.
""" UPDATE: Thanks for the feedback. There were many more concerns than we expected. We’re going to process the feedback and rethink our plan. We will not activate product usage tracking on GitLab.com or GitLab self-managed for now. We'll make sure to communicate in advance on our blog when we do have a new plan. """
So they expected some, but still went ahead anyway?
Like really? Considering what people use your product for, you honestly didn't expect this to upset people? Great. Your product team is hopelessly out of touch.
If you hear ‘do it anyway’ after raising concerns one too many times, eventually you just do it.
It brings to mind a larger question of how we (as an industry) should be measuring developer and team productivity, though. “Story points” are largely designed to solve this issue, but in my experience they are quickly co-opted to mean something different to each stakeholder.
At a past employer, I experienced their trying to measure productivity by “net lines of code”. That didn’t work well, either. I prefer refactoring and simplifying to pretty much any other type of task, and it isn’t at all uncommon for me to end a day - or a sprint - with a negative NLOC. I see at a positive, as each LOC carries with it an ongoing cognitive and maintenance burden. Thankfully, that scheme didn’t last long :)
I can't possibly believe anyone who ever used Digg could have taken a look at Digg V4 and thought "Oh yes, a feed full of spam, just exactly what I come to Digg for". But the team had already invested a ton of work, and organizationally it would have been next to impossible to say "Shit, we fucked up, lets hold off". So at least I give GitLab credit that they backed off (for now).
Sad. But systemic to the industry.
And now that it's turned out to be false, the dominant strategy is to offer a half apology along the lines of "We're sorry, we didn't expect that it'd be such a big deal".
Incidentally, it's quite reminiscent of the Grace Hopper quote: "It's easier to ask forgiveness than it is to get permission."
"let's slip it in and hope nobody notices" == "mgmt heads up their asses"
Gitlab is heavily dependent on developers pushing their product to the enterprise, because any executive would be far more happy with Atlassian (e.g. cheap and it works)
When people use that quote, I try to be charitable and assume that they don't intend its literal, face-value meaning.
But if they do, my response is: I will not forgive you because you're not truly repentant. This is all part of a strategy where you see what you can get away with, and you show no signs of repenting that overall strategy.
Hmmm... maybe I'm a hardass.
"I, personally, totally expected this, as did everyone who I talk to on a daily basis. But someone up the chain of command thought we could get away with it, and forced us to try and push this change. I'm going to use the pronoun 'we' to refer to the company in the abstract, and not anyone on the immediate team I work with, because I jolly well can't be throwing my boss's boss, who happens to also be a commisar from the large company that recently acquired us, under the bus."
That person is Paul Machle (CFO):
"I don’t understand. This should not be an opt in or an opt out. It is a condition of using our product. There is an acceptance of terms and the use of this data should be included in that." [1]
[1] https://gitlab.com/gitlab-org/gitlab/merge_requests/14182#no...
> @cciresi if we follow Paul's guidance and just make this part of our terms and conditions, are we covered legally?
Earth to GitLab: You are not yet big enough to be at the "are we covered legally" stage. You are toast if that is your mindset.
EDIT: To clarify, I'm just making a generalization and am not affiliated with Gitlab.
<disclaimer: founder of a GitLab competitor>
IMO, if you want to compete with GitLab, you should introduce some higher pricing tiers targeted at businesses. These should be comparable to GitLab's pricing tiers and should not have "hacker" in their names. I also suggest that for these higher pricing tiers, you make it explicit that using the service for closed-source projects is OK.
Theoretically $100/month/person would still be a great deal, but the finance people don’t look at it like that. They just see the $97.5/month/person difference with Bitbucket.
Telemetry was always going to be a concern with services that promote themselves as "open-source" or "free-software". The subject is so important that it is one of the reasons that the mass exodus from GitHub to GitLab happened. So to say that "there were many more concerns than we expected" is complete bullsh*t and appears more like the VCs are puppeteering the founders with this talk here.
Like other companies that are at the mercy of VC funding, they will do anything to please them over the "community" to show that they are growing. So don't be fooled by this empty response.
This will keep happening until it literally becomes illegal to collect personal information.
Drilling a hole in your ship may be immediately helpful because you are thirsty, but ultimately it’s not going to end well for you.
I’d say it’s well within the realm of reason to say a hard ‘no’ to investors in this case.
They are responsible to the Board of Directors of the corporation, not the shareholders. The Directors are responsible to the shareholders.
They have a responsibility of care (including a fiduciary responsibility) to operate the corporation in the corporation's best interest, not the shareholders, as directed by the Board.
That best interest can be measured in all sorts of ways as established by the Directors, which may include increasing shareholder value.
The practise of CEOs also being the Chairman of the Board, of executives being major shareholders, of bonuses being driven by share price, are all practises that should be eliminated, given that they are not in alignment with an executive's actual responsibilities and duties.
People were so fast to live their stuff to Gitlab without bothering thinking about that business is business.
These fucking corporations or wannabe corporations doesn't give a fuck about you or your data. That's the fundamental mindset you need to have when sending them money for services.
Ain't no such thing as halfway crooks.
Thanks for the redpill. /s I would however say that they had a good rep until now, in general, so perhaps they will reconsider if there's enough backlash.
They don't give a fuck about individual users but they sure do care about their data.
http://files.technologyreview.com/whitepapers/MIT_Oracle+Rep...
I currently use GitLab where I work; we chose it because our data is sensitive and a cloud service was not an option. This telemetry means that I won't be updating until this blows over. Frankly, whoever thought this was a good idea is a moron that doesn't seem to understand that users like my company chose GitLab because we didn't want this shit.
https://gitlab.com/gitlab-org/gitaly/issues/2113 https://about.gitlab.com/blog/2019/10/10/update-free-softwar...
There is no way for Safari users to opt-out from this because Apple has decided to ditch "Do Not Track" because of privacy concerns.
https://gitlab.com/gitlab-org/gitlab/merge_requests/14182#no...
Which points to where the problem might lie.
Why the CFO is calling the shots on critical architectural issues is a whole 'nother question.
If you want to talk to your spouse about the possibility of taking the kids to get ice cream or whether the dog has had a treat today you have to speak in code that the short mammals can't understand, otherwise you've all but promised it to them and have to deal with the consequences.
The problem is that for salespeople and business devs, the consequences of putting a bad idea in front of impressionable ears are too abstract and so they never learn. So they put an idea in front of a customer or the board about how they can make a shitload of money and the imaginary check has been cashed even before they've stopped for air.
Without some tough love these problems will be with us forever. Oh, you're going to have to walk back something you said? You'll be embarrased? Too fucking bad. Maybe next time you'll think before you promise someone $500k of work for $250k minus your bonus. Twist in the wind like you deserve.
So someone has sold an idea of making millions and once the engineers or just plain human beings get ahold of it that looks like $200k and a giant pain in everyone's ass. And everyone jumps straight from denial to bargaining with a little detour to anger to yell at the messengers for breaking your shiny dream... that you are not and were never entitled to.
Reading through the MR comments, it seems to me like that's the case with everyone. The CFO is pursuing profitable options, the legal & compliance teams are making sure everything stays in compliance, the engineers are building what is asked of them, the data analysts and product managers are asking for the data they need to get insights on product enhancement...
The big issue seems to be that everyone is so narrowly focused on just their job function that they are missing the forest for the trees. I also noticed a distinct lack of anyone from any type of customer advocacy teams (does GitLab have anything like that? Account managers, evangelists, developer relations, etc?) that probably would have been able to put forth actual data about if customers would be for/against this change.
> Reading through the MR comments, it seems to me like that's the case with everyone. The CFO is pursuing profitable options, the legal & compliance teams are making sure everything stays in compliance, the engineers are building what is asked of them, the data analysts and product managers are asking for the data they need to get insights on product enhancement...
Ideally everyone should be would also be thinking about whether the feature is ethical, even if it's not "exactly their job", because there generally isn't anyone whose job is specifically to decide that.
> Sorry, I am coming in late to the game here.
> Saying we want the UID for easy ID and deletion is one thing. If we actually intend to track and do something, then that is another issue. This former benefits the user, the latter benefits us. Because we stand to get a benefit, users should opt-in. We will also need to amp up our privacy policy. There needs to be full disclosure in what we gather, why we gather and with whom we share.
https://gitlab.com/gitlab-org/gitlab/merge_requests/14182#no...
Github had to be acquired my MSFT because it had no path for real enterprise growth if it could not lean on an organization that already sold a pile of services and products in similar or complimentary space. MSFT was the only strategic buyer of Git hosting company. MSFT is the only circus that could use that monkey to fill the gap in its product line up. Of course it picked Github rather than Gitlab. Which is why Github would be incredibly successful, which in turn means that Gitlab in its current state won't be. Hence they would have to massively change the product to justify valuations. It would be interesting to watch.
SourceHut is in the exact position Gitlab was before VC FOMO. It can provide decent service that people and companies will pay for but they would never get a billion dollar value. As long as it works for them, it will work out just fine. I wish them well.
Gitlab is the _only_ choice when your company prefers on-prem (which is the overwhelming majority of >10k employee companies) -- I work at a company with more than 15k employees and we hold an enterprise (plus) license for 8k seats.
We are definitely not alone, companies like EA/IBM/Goldman Sachs have _large_ internal deployments of gitlab.
They certainly have developer mindshare in large companies, but github is the facebook of sourcecode repositories, it will always be more popular outside (or in smaller, less self-hosty orgs).
Microsoft is a hundred if not thousand product company with gigantic sales force and sales channels that are beyond the level of imagination of companies who say "remote only" is their differentiator. Github Enterprise exists. I know it because we used it. Github will be sold as a part of the rest of the Microsoft product portfolio, which is how they would end up in those coveted enterprises.
It will be 100 years before my company puts their core business in the hands of a SaaS provider.
Is that literal or figurative? GitHub Enterprise is on premises. I used it at my last company which was on an air gapped network. It looks like it has been around since 2011.
There are countless other self-managed variations in size and feature combinations of git hosting, ticket tracking, and other things it offers.
I don’t get how you can say GitLab is the only choice?
Since Microsoft's acquisition, even github.com is progressing refinement at a higher pace than gitlab.com ever did, and I'll be thrilled when I get my company to switch to Github again. Pricewise, Github EE is much better value for the dollar than Gitlab EE; most features that Github doesn't have are rather half-assed than wholesomely done in Gitlab.
What's the point of having them when Gitlab employees downvote anything that does not say "Gitlab gooood!", Google employees downvote anything that is not positive about Google and Facebook employees downvote non-positives about facebook.
It's a very light weight, self-hosted git server, the UI is very GitHub like.
It's not a replacement for all the features of a self-hosted GitLab.
But I've seen people battling a self-hosted GitLab instance when all they needed was something like Gogs.
Gitea is also nice self-host option, but also not a complete ALM solution. We often use trac in conjunction with different source control providers, which might be ancient by now, but it always delivers and everyone seems to like it.
https://i.imgur.com/rFbaEkL.png
if i replaced just the logo in the upper left hand corner to the github logo instead of the googs logo would you be able to tell the difference between this screenshot and github proper?
I think I could easily do that.
And I'm a backender-at-heart who doesn't even have full colour vision.
So, not a copy I think.
I've been using Phabricator for 2 years and development seems to have picked up, if anything. They seem to be polishing what they have and making everything more robust, an odd thing to do if its going the way of the dodo.
Combined with debian moving to gitlab, gnome moving to gitlab, ... , that was the sentiment I got.
It does not seem to be mostly developed in China, but there are Chinese contributors.
https://github.com/go-gitea/gitea/issues/1029#issuecomment-5...
Written in Python and used as dist-Git frontend in Fedora + used to host many Fedora related projects.
I've always thought of Gitlab as a privacy minded company, until now, and the botched attempts to listen to feedback are only worsening my impression. Please prove me wrong!
DNT is basically a failed technology; if you really care about not being tracked you would just block the trackers rather than asking them to pretty please not track you.
(Edited to add: as others have said, this should not be opt-out anyway. Opt-in is much less user hostile, and as far as I know is legally mandated in the EU anyway.)
Opt-out is pretty hostile UX for something you must have known would be seen as a negative by large swathes of the developer audience (and if you didn't see the negative reaction coming, then there's larger questions to be asked).
I'm a free user, so I'm "happy" insofar as I have very little say, but it's certainly a worrying step.
We didn't expect praise for more telemetry from the developer audience but we did underestimate the reaction. There were many more concerns than we expected. We’re going to process the feedback and rethink our plan. We will not activate tracing on GitLab.com or GitLab self-managed for now. We'll make sure to communicate in advance on our blog when we do have a new plan.
Be serious. You knew what the reaction would be. You gambled on it getting missed by the community and lost.
We published this change in a blog post and sent an email.
Also, might be a learning about engaging the community on things it understands - a common user of a SaaS platform might not have an understanding of tech, but a common user of a developer SaaS platform does.
If when I had next logged into the dashboard I got a modal that asked me "hey, here's a list of things we'd like to monitor to make your experience better, what data would you be happy we looked at?" I'd probably opt-in to some just to be helpful as I know the context.
I'm guessing a lot of your mid-market acquisition (though likely not revenue) is through word of mouth, migrations or "dark procurement" (?) - audiences that are ultra-sensitive to this sort of thing, so the misstep here is a question.
I would have thought this would be one of the most important steps in implementing this change. Surprises are never good for people working in this field.
* Devs are very sensitive to tracking. Especially opt-out rather then opt-in. We know what kind of data companies sit on. And opt-out is invasive.
* Gitlab is perceived as a developer friendly open source company. An alternative to the bloated/"enterprisey"/milk the customer stacks out there
* Self-hosting should come with a promise of privacy
* Disabling API/UI access without opt in is just a really bad move. You should have seen the anger coming from a mile away.
* The claim that you can't get insight into how Gitlab is used without client side telemetry is very dishonest.
You have a wealth of data in the DB and web server logs. True, you won't know how long a user fiddles with the mouse until they click a button. I understand the urge to optimize the UX. But user studies go a long way.
And let's be real: this isn't about optimizing button placement, but about showing some nice engagement graphs to investors.
Also, the whole response both in the issue and here was quite poorly handled in my opinion.
The back and forth here... Copy-pasting a reply to each commenter in the issue... (really?)
Just re-consider and put out a unified response.
We have a lot of data in the DB of GitLab.com but it was hard to understand it. Using a third-party service with client side tracking saves a lot of time and effort.
That's like me inviting you to my house and you show up with a pet pig. Your pig might be clean and well behaved, but I'm not letting it my house because it's a pig.
I trust GitLab and I'm ok with the spirit of what you're trying to do, but if you want me to opt-in to client side tracking from a 3rd party (in an untrustworthy industry), you'll need to convince me the company you're working with deserves to be trusted.
On a smaller project, we have implemented Gitlab CE and it was/is in the lead of the various alternatives.
Telemetry to any external service from inside our VPN is a definite no-go. Not to mention that it would be blocked in our configuration, but if that meant that the application didn't run, then we need to make another choice.
Telemetry to a specific provider that we have licensing and other arrangements with would be manageable, as long as the data collected was documented and we could determine that there was no possibility of data leakage beyond that declaration. It would need to be opt-in and it would need to be under conditions that both parties agreed to.
Tell your CFO that if he wants to sell to enterprise, particularly self-hosted enterprise, then he needs to get his head out of the "SaaS" world and deal in the world of Enterprise, where things like SOX and HIPAA and GDPR and PCI/DSS and other standards preclude "collecting data on our users".
I've always opted into telemetry because I assumed companies just use it to improve their products. I guess I've never actually been in the inside of a big company, though. Do they do something else with it I should know about?
Great!
Every single one of our concerns was raised by someone in your own MR thread. C'mon, you were just hoping people wouldn't raise too much of a fuss.
Your EE users use your self-hosted product because they don’t want anything leaving their premises — not a single bit of data. If you require this, do you really think EA, SIEMENS, Ericsson, Lockheed Martin or Bayer are going to let any data from their instances with potentially secret and internal future projects leak out to you or your third parties? No, they won’t. They’ll require you signing a contract that you won’t track them, or they’ll switch vendors — you’re the small fish in that case.
Additionally, I’d suggest reading and understanding the GDPR — the text is actually surprisingly simple:
https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CEL...
So you, or in case of the EE Edition, your customers, have to have explicit, clear, and freely given[1] consent to allow any telemetry or tracking which contains PII to happen at all, and especially to give any PII to third parties. Pseudonymous data counts as PII. Data which contains the IP (such as a direct connection to a third-party server to transmit data) counts as PII.
[1] Even more interesting, freely given consent is defined as explicitly opt-in, and it has to be in clear and simple language, the default (and/or the biggest button) has to be rejecting everything, and the consent can not lead to any functional changes, so you can’t require opt-in to let the user access any functionality either.
"Enterprise Edition (EE) No current changes. We are not adding telemetry services to EE at this time."
Which was recently added after receiving a metric ton of backlash. You have to question their initial thought process here.
The original title was "Remove the ability to toggle sending telemetry data back to GitLab", then "Remove the ability for on-prem users to toggle sending telemetry data back to GitLab"
>I'm not familiar with the term "dark patterns".
Then you should not be suggesting changes like this in the first place.
>I want to highlight again that public sector customers will be excluded from this. I've mentioned many times that it's important we respect their strict privacy requirements.
But fuck private sector customers? What?
That sounds like something to rely on, especially given how they've handled this so far. Also, this wasn't originally there.
Emphasis mine, but the wording is notable.
Alice> We want to do a thing, and we want everyone to do it without being able to realistically avoid it while still saying "we gave them an option"
Bob> Let's use DNT, it's used by almost no-one and is largely obsolete.
Alice> Perfect!
It's a bit like web sites that allow you to opt out of tracking and targetted advertising by directing you at the cookie config settings in your web browser. A token gesture.
Respect your users and respect yourself, roll back the change, make an apology PR piece and move on.
Enterprises have been dealing with GDPR, CCPA, and data privacy issues for several years now. The apparent fact that Gitlab doesn't recognize when they're running afoul of opt-out standard practices mechanisms, and has those vulnerabilities appearing to be not caught during the SDLC is probably causing a lot of second guessing of competency by your more mature customers.
edit: This isn't a problem unique to Gitlab. Microsoft, for example, has encountered and dealt with this problem (telemetry privacy issues) as well (https://docs.microsoft.com/en-ie/DeployOffice/privacy/overvi...). Search for "Microsoft Dutch DPIA" for all the sordid detail.
I think a big problem is that it was so unexpected! Especially the API changes, where people probably are wondering why things aren't working today. Some advance notice would have helped, probably.
I've used Gitlab in the past and will continue to use it here and there, though I host all my own repos at home with just regular old git and use Github for work, so take what I'm saying with a grain of salt because I'm technically not a heavy user of your service.
DNT is dead as a dodo, and many people either use browsers that no longer support it or are strongly resistance to enabling it because it's used as a signal in fingerprinting browsers.
I'm not sure why you thought DNT would help.
And why did the lead email mention SOC2 and not CCPA and GDPR?
Here is an alternative solution I begun using for one of my projects:
There is no telemetry like grep in nginx logs with carefully fetch()'ed paths.
Put this in JS code:
fetch("/stats/user_used_feature_x")
fetch("/stats/user_reached_checkout_step_3")
etc.
grep -o ".*/stats/[a-z_]*" /var/log/nginx/access.log
That is dozens of time more performant than using third party scripts. No extra library.
In my experience, access to analytics is a quality of life improvement for programmer, that does not actually result in better products. Having a thousand times more crash-reports wont's cause your employees to grow extra hands and brain-cells to act on all of them. Having a great insight in user behavior does not automatically translate in great managerial decisions.
Personally I would love Gitlab to stop overusing Javascript and fix performance of their backend instead of trying to conceal issues by refusing to show big files and abusing lazy-loading. But judging by their recent actions, they are more likely to copy more anti-features from Github than work on actual hard problems.
1. DSGVO means opt-in when collecting private information not necessary for the usage of the product (and even then it's dicey). Making the use of your product dependent on accepting private information gathering is illegal. DNT is logically a valid way to opt-out (apart from browsers dropping the feature and users not knowing about it), but that's not enough here. It would be different if DNT was enabled by default (I assume), but all browser makers dropped the ball here years ago.
2. You built a reputation of the people's git provider, albeit in conflict with the completely free alternatives. And now you make a post "we will change it, you have to accept the new TOS, and there will be a service interruption", like a generic US company might do. It's unfitting to you. It does not matter that the products concerned are not free.
3. The right solution exists and would not hurt you. I assume you want telemetry data to improve the product (if not the issue is even bigger). So you add that option and ask them whether he would be okay to activate it, like Mozilla is doing in products like Firefox. This scheme works and data gets collected anyway, a bit less, but that should not be an issue. Careful: Admins must be able to remove the prompt for all their users if it arrives in the self-hosted products.
4. Pick a data collection destination that is the least controversial. In your case that should be inhouse. Alternative might be a privacy focused organization in Europe, but it has to be known to your customers.
5. If you think DSGVO does not apply to you realize that it does not really matter since the sensible parts of it are also the ethical correct way. So in any case it is good to follow the spirit of it.
6. Note also how matter of fact and corporate the language used in the blog post is. That can only worsen the impact on those that care about you and the topic.
Browser makers did the logical decision as they have no control over websites adhering to DNT. If they made it default, every site would promptly ignore it as it'd impact their business too much to follow it. On the other hand, if they made it opt-in then sites might follow it for the minority of users who used DHT. In the end, it was a bad idea as it had no teeth at all to force adherence on websites.
This feels gross, and I'm going to look at moving.
For example if I paid for 1 year, I would expect the contract available at the time of payment should apply for the entire year or offer a refund if you don’t want to accept the changes. And always with at least two weeks of notice
As I understand it, like GitHub, GitLab offers relatively permissive free plans to individuals and projects in their early stages. Both also offer pathways to graduate to commercial plans that are also enterprise friendly.
It sounds like GitLab planned telemetry for the enterprise-managed plans (not the individual-managed plans). If GitLab was seen as an alternative to Microsoft-owned GitHub, the plans for telemetry undermined this reason for developers to start their projects with GitLab instead of GitHub and advocate its use to managers, investors, and legal departments.
If that is the situation, where does SourceHut stand? The site makes little sense to me. The URL is sourcehut.org, but there is no statement about it being a non-profit, or any discussion about a board that manages it. There is a pricing page, but no contact us page. I don't know where it is based, and I can't find a reference on Crunchbase to get any idea of who owns the company.
If it is a competitor to GitLab, maybe it is a competitor in the space for individual developers, but there wasn't a significant problem for individual developers when it comes to GitLab (or Github). If the problem with GitLab's announcement was that it made the upgrade path less enterprise friendly, how is SourceHub an improvement if by all appearances it is incompatible with any sane enterprise.
>If that is the situation, where does SourceHut stand? The site makes little sense to me. The URL is sourcehut.org, but there is no statement about it being a non-profit, or any discussion about a board that manages it. There is a pricing page, but no contact us page. I don't know where it is based, and I can't find a reference on Crunchbase to get any idea of who owns the company.
I own SourceHut as a sole proprietor, you can reach me at sir@cmpwn.com with private questions or ~sircmpwn/public-inbox@lists.sr.ht with public questions. Archives here:
https://lists.sr.ht/~sircmpwn/public-inbox
The business is operated transparently, here's the latest quarterly financial report:
https://sourcehut.org/blog/2019-10-21-sourcehut-q3-2019-fina...
Even though I may not especially enjoy email-patch workflows, I still wish you success, because of your user centric ethos.
For individuals sourcehut[0] may seem to be an alternative. Open source orgs might be better of self-hosting. In either scenario I would rather go with the community than to be in the mercy of a VC-backed corporation for hosting my work in this case.
[0] https://sr.ht
So what are Safari users supposed to do, to opt out?
Turning on DNT is like walking around a big city wearing a cap that says "Please don't record CCTV pictures of me."
And ask Apple to put pressure on services to support it. They're a large fish and we'd all appreciate them helping out.
Edit: I don't think telemetry is always bad. Obfuscated telemetry tells me you're potentially collecting stuff that you don't want me to know about, but things like clicks, mouse-movements, etc. are important for understanding how users use a service and not inherently bad.
Edit: removed most of the post. Rewrote the rest of it.
These things have a super short half life before people just write you off as untrustworthy. But if you reverse it quickly enough, you can turn it into a win. Do it now! The hourglass is getting to that point where there's just a little sand left and it feels like it's starting to accelerate.
Please listen to these comments. There are a lot of them:
As an example that we won't do opt-out self-hosted third party with user data doesn't mean we can't do opt-in GitLab.com first party with aggregated data.
Almost every SaaS company uses telemetry in some form to improve their product. We need it too to improve the the user experience of the product.
But we clearly made the wrong trade-offs so we'll go back to square one and make a new plan.
I really appreciate you guys engaging in this public discussion. I hope you take what follows in the friendly spirit I intend:
> We need it too to improve the the user experience of the product.
This ^^^^ sounds like exaggeration to me. Taken literally, you're saying that the gitlab user experience could not be meaningfully improved without adding telemetry.
I'm not a UI designer, but I'm fairly sure that before telemetry existed, product-improving iterations were a thing.
But based on other people's comments, I think this is a tangent. I suspect that 99% of people's upset is making the telemetry something other than opt-in. I suspect that if gitlab would just commit to making telemetry opt-in and GDPR-compliant, all would be right with the world.
I personally don't care if your hosted one takes telemetry as but if it ever makes it to CE I'm dropping gitlab in favor of I guess paying for some other hosted service. That or I'll patch out the telemetry.
Your CFO is making these sorts of decisions, not developers. Good luck with the business, but I'm out.
> Drew DeVault @ddevault · 1 hour ago
> Obligatory disclaimer: I work for a GitLab competitor.
>Thank you for pumping the brakes, but this is still not great news. This language is the classic "defuse, wait, and try again later" approach to shoeing in unpopular changes. You're still hedging their bets with this language, rather than renouncing the original ideas. A similar tactic we've already seen is apologising for "bad communication" instead of the real problem: bad changes. Until a real apology is issued and the changes renounced, this isn't very meaningful.
"Reconsidering the approach" is CYA talk. What you're doing right now is an attempt at damage limitation instead of fixing the actual problem.
Third party telemetry is a cancer that no-one wants.
You, personally, don't even want it. You know you don't want it, but you're adopting the 'corporate mask'.
Just stop, please. For your own good, as well as that of the wider ecosystem.
Your firm communicated a willingness to break users’ tools, wilfully violate GDPR, and interrupt users’ business continuity, all to spice up some IPO metrics. That’s a lot of trust to rebuild.
I’d recommend putting guarantees against telemetry into your ToS. Also, a minimum announcement period before mandatory ToS revisions and a minimum amount of time before GitLab can nix users for not agreeing to a new ToS.
We opened an issue for gathering feedback at https://gitlab.com/gitlab-org/growth/product/issues/164 and you can find the most up to date information regarding this topic there. Please join the discussion and let us know what you think.
Then they add insult to injury by pretending to “discuss” it as if user consent or participation has anything to do with this.
Good morning. If you commented yesterday, express your product needs for zero telemetry and trust here: https://gitlab.com/gitlab-com/www-gitlab-com/issues/5672
First, they have been very transparent about the whole thing. While many companies would try to hide it in the ToS in vague legalese, Gitlab published a blog post explaining what they were doing and why, and opened public issues about the topic.
Secondly, they listened to feedback from customers (and the internet at large). Even though the backlash shouldn't have been surprising, at least GitLab responded quickly by pulling back when it was clear the community disapproved.
And finally, even though it was supposed to be opt-out instead of opt-in, at least they clearly communicated _how_ to opt-out. Which is more than can be said of most companies, which make opting out as difficult as possible.
Yes, GitLab messed up. But at least they acknowledge that, and were transparent about the whole thing.
Is it really that hard to provide a no-bullshit hosted Git?
Mind elaborating?
When will people learn that corporations are about money?
If you REALLY cared, wouldn't you be self hosting anyways?
Initially, they said they'd infect the self-hosted EE versions too.
I'm pretty sure that server-side analytics ("there were 1000 calls to this API endpoint") is considered very different from client-side analytics via marketing/ad-tech tracking companies.
I don’t think they’re trying to associate your identity with other web traffic the way normal “trackers” do, but just to have a client-side view of a browser session: what did you click on in which order in a typical session, what features are unused and what needs to be boosted to the dashboard pages of your repository, etc. Especially when backend calls can be triggered from multiple frontend locations, this sort of information can be really helpful.
I think what most people are complaining about is the third-party. If, say, Gitlab hosted their own telemetry services (on their servers, accessible to their product managers, etc), I doubt there would be so much backlash.
> I work at GitLab in Support Engineering I'm talking with product now about this. We raised this when the conversation started as a concern, and I want you to know that internally I am advocating that it _doesn't_.
They would not need to "advocate" for it to continue working if that were teh case.
I am dropping Gitlab like a hot rock. The way they did this initially, and the responses that I've been reading here and on Gitlab, don't reassure me at all.
All this herky jerky stuff is mind blowing. They have a product people seem to like a lot but I don’t think they have the right core values and beliefs to sustain it long term. If they did, this would have been executed much better and there wouldn’t be this knee-jerk reaction to random gripes on the internet.
Chasing shiny objects and the noisiest users is gonna get ya killed.
Sorry, but we can't expect it from "for profit" company especially backed by venture with the exit goal on ipo.
The thing that is utter shit is that they could really achieve any reasonable analytics needed from their server logs on their hosted systems.
Pendo is the same as Heap analytics in that it logs EVERYTHING the user does, including things like scroll depth, time on page, form element selects, even clicks on non-clickable elements. You could just ask me how to make the UX better, GitLab; there is no need to do indiscriminate bulk collection.
Or here's a novel idea: maybe actually give the users who choose to give you their behavior some kind of reward. A pro feature for free, some build runner priority, or even just a icon that glimmers with some gratitude. And an incognito switch for times when I don't want be seen.
And a 3rd party? Its not that hard to put a pixel on a page and stand up an endpoint. What's to stop that 3rd party from injecting a keylogger and start harvesting secrets? Trust?
If I were owning analytics at GitLab, I would:
1. Develop a way to get cleaned up server access logs in front of the UX team. This should get 85% of the work done.
2. Provide a 1st party controlled, open source client analytics tool that users can inspect.
3. Make it optional, off by default, and most importantly, fully accessible to the user. When its active it should provide a visual indication. Give opted in users a reward for participating. Give them the ability to see the data that has been gathered on them. Personally, I could find that log of my own activities quite useful if it were shared with me.
4. Prove the value of client side-tracking and share the wins with all stakeholders. Like, build a step in issue triage to gather metrics and share the results on the ticket to inform the decision making process. If the data doesn't does add double digit value to the company, kill the whole thing off.
This was a brutal blunder for a company that's on Series E, and shows that leadership is not in touch with their customers. More actual face time with developers by leadership could have stopped this from happening.
What's your Pendo Developer Score?
Hacker News likes to complain about how bad the Gitlab UI is compared to Github, or how slow things are on Gitlab, or how they are doing too many things and need to focus their product better. Well, knowing how their users are using the product is one key part to improving all these things and delivering a better product.
As a paid user of the Gitlab.com product, I am fine having them carry out analytics on user behavior like my own. That said, I do use an ad blocker and if they use 3rd party trackers I'll definitely be blocking those trackers.
[1] E.g. the use of third parties for the telemetry. They should definitely be using in house systems for something like this not 3rd parties in order to protect their user data. Also, they should not be rolling this out to self-hosted versions.
Disagree. The kind of client-side hyper granular tracking being discussed here is not a good thing. It encourages micro optimization for metrics over maintaining a good big-picture architecture. I've seen no evidence whatsoever that software is any better now than it was in the days when applications were off-line only.
It also appears that Gitlab really is all-in on this telemetry stuff, as they envision "telemetry-as-a-service": https://about.gitlab.com/direction/telemetry/collection/
But the way that this was rolled out was egregious enough that I don't have much trust in Gitlab at the moment.
I suppose I could run me own Gitlab CE, and maybe I'll have to. That or I finally have an excuse to build a competitor :)
I'm at a loss as to why they'd need 3rd party telemetry for this. It's a service that they host. They have the data directly. It's not something that runs in the cloud (ie, on someone else's server) where they don't have first party access to the underlying data. Even if they want to visualize or analyze the data with someone else's tools that don't have import functionality, they can just "project" the data there without the user (client) being involved at all.
The only reason they'd need to do this is for EE deployment where they don't in fact have the direct data. But certainly that's such a small part of their business. And, can't they just use what they know from .com usage data?
Sourcehut came to my attention due to similar concerns with GitHub. Didn't take GitLab long to go from alternative to this.
What they do here is highly questionable in regards to GDPR, I'm reporting this to my local authorities.
You could always just abandon the account and never use it again, I suppose.
Gitlab is not an underdog, stop fighting for them.
So wouldn't it be a possibility to just fork the project outright and remove the telemetry from there? Or is there something in GitLab's licensing that makes that impossible despite it being 'open core' or whatever it is?
What does "telemetry" mean in this context?
What is GL actually demanding of its users, and why?
As you can imagine, a lot of businesses are not willing to take that risk with trade secrets like source code and internal bug tracking -- especially when there's no upside to them.
I think the furor is probably less about the data being collected (although people aren't happy about that), and more that Gitlab just offloaded a lot of risk on their users for purely selfish reasons. (And it is selfish. They can get the data in other ways, or, make decisions based on expertise and customer feedback instead. You don't need a mountain of data to drive a product).
Looks like the anger is also a lot about this being sprung overnight with no warning.
> trade secrets like source code
Don't Gitlab already have their source code?
Yes. The point is third-parties now can.
Also, I guess it was a mistake to add middle-finger emoji to gitlab's interface. It looks hilarious seeing all those fingers!
So it doesn't affect me directly.
Someone who is affected-- tell me what change this company is making that you don't like.
No theorizing, please. Only bona fide gitlab.com users need reply.
This opens an additional attack vector that anyone hosting valuable IP on gitlab.com or the self-hosted enterprise edition needs to worry about.
As important as this is, the editorialization is quite clearly against HN guidelines.
> Please use the original title, unless it is misleading or linkbait; don't editorialize.
> Otherwise [outside a set of very specific circumstances], please use the original title, unless it is misleading or linkbait; don't editorialize.
I'm not sure what would count as "specific circumstances". One could argue though that the original title was "editorialized", since GitLab has an incentive to soften the blow of the news, especially since the original title doesn't tell much about the news itself (what is the important update)?
The exceptions are (1) if the title includes the name of the site or (2) if the title begins with a number. Neither apply here.
I believe they phrased it this way so some discretion can be exercised if the original title doesn't give the full context. Another example is the recent Tesla story - original title is "Q3 2019 Update", story title is "Tesla Q3 Financial Results". The HN story for when MS bought GitHub is "Microsoft acquires GitHub", while original title was "Microsoft + GitHub = Empowering Developers".
If there wasn't such a leeway, most acquisition stories would be titled "Our incredible journey".
I would like to also note that my stance that the non-original title (at the time) was fine does not have anything to do with that title "looking bad" for GitLab. For example, the title now is "Gitlab ‘rethinking’ third-party telemetry", which is also not the original title, looks somewhat good for GitLab and I think it's fine.
I remember when Microsoft was just M$, and now the look like the good guys.
Why is the term "telemetry" used, instead of "logs" these days. Is there a substantial difference?
Logs are telemetry when they're sent to third party consumers who don't ordinarily have access to your systems.
Telemetry is literally just remote metrics.
Logs comes from ships throwing a piece of wood attached to a knotted rope over the stern and writing down the results in a log book to record speed at intervals as a part of normal navigation. Telemetry makes more sense :)
It doesn't make it okay, but as far as I can tell almost no one is actually following GDPR, despite the large theoretical penalties. They've all decided to just put up cookie notices (with dark patterns to elicit agreement), because it's easy and doesn't force fundamental changes to business practices.
https://www.nytimes.com/2019/01/21/technology/google-europe-...
Privacy conscious orgs should work on migrating off GitLab.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
That doesn't mean we don't moderate at all, just less. For example, in this case, the repost was obviously a dupe by a large margin, so we marked it as a dupe. But when people complained about that, we were more likely to bend the rules because Gitlab is a YC startup. If it were an unrelated company, we would have been more likely to say something like "look you guys, this is standard practice and it's not a borderline call" and left the repost marked as a duplicate.
Thanks for taking care of the situation, and for further demonstrating what top notch moderation should look like
edit: Thank you for explaining the policy! I think it's excellent.
I also suspect dang knows that majority of people on HN are grateful for his moderation services.
It seems like there's more energy than usual for continuing to discuss this one, though, so I'm happy to bend the rule. It will take a few minutes to effect a three-way merge, so bear with me.
Edit: ok, I merged the relevant comments from https://news.ycombinator.com/item?id=21346318 into here. But I don't think it makes sense to merge yesterday's discussion (https://news.ycombinator.com/item?id=21337594). That one is already historical since the story has changed. So what we'll do is update the title to incorporate the new information, and leave a link to the older thread at the top. I'm going to mark this subthread about dupeiness as off topic though.
If anyone still has concerns, let us know and we'll see what we can do.
>We're going to process the feedback and rethink our approach.
Scare quotes are used to express irony or skepticism when you're not quoting someone.