Upcoming changes to our CDN for GitLab.com
about.gitlab.com
about.gitlab.com
Like, I understand the argument, I’m just not sure why we get angsty about one type of market dominance and not another.
When you're managing the webserver yourself, they can still filter by IP as they run the network. With a bit more effort (but this feels unlikely to be happening wide-scale), they can still transparently MITM your connections as the private keys are still stored on their infrastructure.
CDN and WAF are really competitive businesses though.
Granted the idea would only really work on mobile due to the ubiquity of biometrics, but in context, I generally do git from OCs without any biometrics.
Or to put it another way: Garbage In, Garbage Out.
Just the other day I was revisiting the notion that using SRV records in DNS for web traffic would be absolutely ideal for small to medium sites. Unfortunately, the browsers (which have faults listed against them) refuse to implement them. Instead we have these CDNs giving us the ever-valued low latency, rather than giving the next layer down infrastructure (you and me using our VPSes) the ability to do it without needing BGP AS# agreements and fixed IPs and the like.
Now, the reasons the browsers don't implement them is because of the RFC which introduced SRV records suggested that existing tech shouldn't use SRV...
Now the most damning thing about this is the Chrome [1] won't do it because Firefox didn't do it[0]...(see comment 2 on chrome/webkit bug). Both are excellent reads, with lots of coherent thought... and yet are just...rejected.
And finally, there's this page: https://jdebp.eu/FGA/dns-srv-record-use-by-clients.html, which has a section:
> The SRV Lookup Laggards' Hall of Shame
For which Mozilla sits there as #1.
So, yes, I'm concerned. Yes there are alternate ways so that everyone doesn't need to use a CDN. Yes, it's the browsers themselves that have been stopping us....for 19 years.
[0] https://bugzilla.mozilla.org/show_bug.cgi?id=14328 [RESOLVED WONTFIX]
[1] https://bugs.chromium.org/p/chromium/issues/detail?id=22423 [RESOLVED INVALID]
Basically, they're complaining about the added indirection slowing down the user experience.
Analysing their arguments is interesting.
First: In 20 years since this rejection by the Mozilla and later the Google teams the load-balancing problems have been solved... just as inefficiently. The typical implementation of global services load-balancing (GSLB) or global traffic management (GTM) as commonly used in Citrix NetScaler or BIGIP F5 is to use DNS CNAME records pointing at a DNS stub zone "owned" by the load balancers. This delegation is done via NS records that have to be looked up as a second step. The appliances then return either a CNAME or A record pointing to the actual web servers. So between 1-4 DNS lookups is typical, but I've seen as many as 10+ in extreme cases. Azure CDN comes to mind, which has 6 CNAMEs in a row across several domains and associated name servers.
More importantly, caching typically doesn't help much with GSLB/GTM, because they inherently must use short TTLs. There's a very direct trade-off between availability and performance. Typical TTL times are single-digit seconds to a couple of minutes at most. If it were any longer, clients would continue to try to connect to failed sites for a long time before giving up and requesting a fresh DNS address from the load balancers again. Hence, because of these low TTLs it is common for every page view on such sites to trigger at least one DNS request if you're clicking around at a normal rate. Hilariously, this doesn't show up in benchmarks because load-testing tools load pages fast enough to amortize this cost, so developer and testers don't notice!
Second: Modern DNS servers use the "additional" section in their responses to also include the "A" records along with the CNAME or SRV host names. Windows Server DNS does this by default, as do many others. This means that the additional round trips are zero as long as the SRV records point to a zone managed by the same name server. In other words, SRV-based load balancing could use a single round-trip just like efficiently implemented GSLB/GTM solutions.
This then combines with the client-side failover capability of SRV records to allow long TTLs, on the order of hours or days. Clients aren't forced to flush their DNS record caches to connect to secondary or tertiary sites, as they already have the records available and can fail over faster than with GSLB/GTM.
Third: The "workaround" for the inherently poor DNS performance of traditional GSLB/GTM is to use IP routing tricks that make a single IP route to the nearest data centre location instead of a single destination.
Cloudflare's 1.1.1.1 service and Google's 8.8.8.8 work like this, as does Azure Front Door. However, this is something that inherently requires a large scale to implement and is typically unavailable to small organisations. More centralisation of power into the hands of a few megacorporations is not healthy for the Internet!
As an aside: I've noticed that Azure DNS Zones does not use the "additional" DNS response section, which means that two sets of lookups are required for every CNAME or SRV query. This doubles their revenue for alias records of all types compared to the more efficient implementation. I wouldn't say that this was done in bad faith, but from the outside it certainly looks... suspicious. Certainly, their incentive to grow their cloud revenue is much more direct than some nebulous public good of a "better, faster web".
Anycast is hardly a "trick", it's been around approximately forever and is well established as a multihome solution.
They didn't block anything or censor anyone. They simply kicked a customer off their services. That customer was free to host their own traffic or find another CDN willing to serve them.
OT but my clients are small players (10s of GB traffic a month); but they value availability and low latency for their dynamic content. Fastly (and varnish) are ideal for this, but every time I talk to Fastly about services beyond the PAYG-CDN it's "min $2k/month", "$5k/month + fees".
I'd rather use them than Cloudflare, I've been accused of being a walking advert for Fastly - but they are pushing me towards other providers and proving actively hostile to small clients.
Good for them if they are raking it in from enterprise customers. But AWS manages to cater for both ends of the market, and it's a shame to see good services price themselves into "enterprise-only" territory.
https://support.cloudflare.com/hc/en-us/articles/203306930-U...
In general, what you can expect to see when visiting a Cloudflare site via Tor is a CAPTCHA challenge to authenticate your browser to show you're not a bot.
As to whether we can enable the `Onion Routing` setting, which would cause your browser to connect to Cloudflare via Tor and not via an exit-node, I don't know if we may enable it. There might be some compliance issues with that, which may prevent us from enabling that.
I have however created an issue on this for further investigation whether that might be possible: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues...
Cloudflare will see All The Things™ as TLS termination at Cloudflare is required for Cloudflare to be able to analyze web traffic and block common attacks. Just based on that one sentence, it seems like a value play more than anything.
That they care enough for the defense-in-depth is probably a good sign, but that also shows a bias in favor of security over privacy. Depending on your personal or corporate privacy preferences/politics, that may or may not be desirable, but it's also for you to decide if any resulting risks are acceptable.
My hope is that Gitlab pressed Cloudflare on the process changes that took place after the WAF outage last year.
---
(this comment has been heavily edited since it was originally published)
Unless there's been a serious homomorphic encryption breakthrough, it's impossible to impose layer 7 intrusion prevention if layer 6 encryption is passthrough.
They subscribed to consolidate the listed offerings (CDN, WAF, DDoS protection). That WAF is mentioned at all is a tip that Cloudflare must decrypt and analyze Gitlab's traffic before re-encrypting it for delivery to Gitlab's application servers.
A. Uploading your own certificate - this just uploads it and CF still does HTTPS interception.
B. Spectrum - Cloudflare will proxy the actual port 443 TCP connections (which might be the plan for SSH connections to GitLab that are not altssh.gitlab.com). The issue is that this only does layer 4 IP reputation DDOS protection; no WAF, etc is done so this most likely isn't what they will do for the main gitlab.com site.
We opted to use Cloudflare Spectrum to facilitate our change. There is no way to mix Spectrum and non-Spectrum on one hostname, but Spectrum allows us to choose between TCP, HTTP, and HTTPS pipelines.
While we intend to TCP proxy port 22 on gitlab.com and 443 on altssh.gitlab.com, this also allows us to use Cloudflare's HTTP(s) content processing pipeline on ports 80 and 443 on gitlab.com. In the latter case Cloudflare will be terminating TLS, performing WAF, CDN and Workers duty, then re-encrypt and send the request to our origin.
You can look at some visualization of this here: https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...
Gitlab is still the team that didn't notice that three of their three independent backup mechanisms had not been working for weeks to months. And while wholly cloning Github is legal, I started having serious doubts about their ethical instincts when they repeatedly tried to incite people against Github for cloning minor features.
Beyond that: it depends. I tend to believe large vendors are worse for privacy of generic consumers, because they have the scale where it pays off to sell data or do "big data" analysis. They are better for individual high-value targets because they have more. to lose, and less to win, relative to whatever they could expect by screwing you over. Protection against rogue employees is also something that happens in larger teams, if at all.
So, this is a really good point. Someone made the smart decision within Gitlab to outsource some of the harder problems to an outside party with serious experience on the topics rather than trying to script their way out of the challenge themselves.
> Protection against rogue employees is also something that happens in larger teams, if at all.
ref: amazon w/r/t capital one for a good example of this exact failure mode. (Can't tell if we're in agreement; your statement is a bit ambiguous, but given "if at all," I'm assuming we're on the same page)
> Thompson's prior role at the company didn't lend her any insider access in this case.
Thanks for the fact check!
I just don't get why the internet neither forgets nor forgives. A startup in it's early days makes mistakes, the only real mistake Gitlab made was be fully transparent about theirs.
You vouch for CF (and no beef against them), but you know nothing of their internal privacy or reliability pitfalls. That warm fuzzy feeling you have about CF is a result of their carefully groomed PR coupled with 0% opacity on internals. Again, not saying CF has any issues, but if they did, you wouldn't know
Trust takes a lifetime to build and a moment to collapse.
EDIT: toned down
- What are the privacy implications of this? Theoretically, any 3rd party in line of traffic (in this case it is Cloudflare) has the ability to monitor and alter traffic. This is especially true when said 3rd party does inspections into TLS traffic. The privacy implications are, that they can see and alter any traffic between the client and the origin. But in the case of Cloudflare and GitLab, that is desired, as we want to take advantage of Cloudflare's Web Application Firewall to further protect us from 0-days and other threats, where mitigation using traditional methods might cause a delay, that cloud lead to exploitation of those, causing bigger harm. My personal opinion on this (it does not necessarily reflect the view of GitLab) is, that Cloudflare is a vendor I personally trust in terms of adhering to what they say in regards to privacy. While they, as any other vendor, could be breached, I see the benefit of their Security solution bigger, than the breach-risk.
- Will Cloudflare be able to see GitLab traffic? Yes, partially. SSH is not inspected by CF, despite being proxied by them. HTTP can be inspected by anybody on the net and yes, thus CF, too. HTTPS will be terminated by Cloudflare (and re-encrypted when talking to our origin) for some reasons: - CDN: They need to know which resource you request in order to serve it from a cache - WAF: Once enabled (not from the start) it will scan requests for malicious content and either block or challenge the request - Workers: A highly integrated FaaS platform, we can use to dynamically authenticate requests to cached resources and other logic we may want to execute on the edge of Cloudflare's network. Despite all that Cloudflare does not log raw requests. On our end, we only see the information about where the request came from, user agents, etc. You can find out more about what Cloudflare logs over here: https://blog.cloudflare.com/what-cloudflare-logs/
- Was/is Fastly able to before? Fastly is in the same position as Cloudflare will be in the future. Right now https://about.gitlab.com is served via fastly pointing to an origin, which we control. If fastly would be breached, the attacker could control the contents of about.gitlab.com. Static assets are also hosted via fastly and are also subject to be possibly altered by an attacker.
- And is there any reason to trust one over the other? The decision within GitLab was made purely on technical grounds. Cloudflare was the only solution meeting our criteria to keep serving traffic via SSH on the same host as we do right now. Speaking for myself again, I trust Cloudflare and have personally been a customer of theirs for a long time 2012-ish). While I do not have that kind of relationship with fastly, I believe both vendors to be trustworthy in doing their best to protect the privacy of their users.
You can find more information about how we are going to deploy Cloudflare here: https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...
> ..inspections into TLS traffic..
> SSH is not inspected by CF, despite being proxied by them
> HTTPS will be terminated by Cloudflare (and re-encrypted when talking to our origin)
As someone who works on the web and does use Cloudflare, I guess I knew these facts, but I must admit I have some doubts about the security and privacy implications, what they mean in practice.
It comes down to trust, and while I believe CF is an excellent actor in regards to privacy, this need for trusting a third-party makes me uneasy.
If CF (or anyone really) is in the middle, then couldn't we say that SSH and HTTPS are not (or less) secure?
> If CF (or anyone really) is in the middle, then couldn't we say that SSH and HTTPS are not (or less) secure?
I guess that depends on how you define 'secure'.
The confidentiality aspect of security is reduced, by letting a 3rd party (Cloudflare) inspect the traffic. However, as stated before, they do not log contents keep metadata briefly. And I have no reason to believe they do otherwise.
Utilizing CF's technology to help prevent a possible breach of GitLab's servers however, strengthens security in my book.
Speaking of breaches. IMO a breach (whether it be at Cloudflare or at GitLab) would be horrific in its own right. But, if an attacker manages to compromise TLS traffic by attacking Cloudflare, they would gain access to that traffic and thus any authentication credentials within it. The blast radius would be limited. Breaching GitLab.com however, has the potential of putting all customer's data at risk. As cruel as it sounds, but I'd rather have a potential breach in front of GitLab. Well, ideally none at all.
I do understand the concerns about having to trust a 3rd party. And as you have pointed out, it comes down to trust. By publishing this information beforehand we want to make it easier for everyone to keep their trust in us, build new trust with Cloudflare, as well as to live up to our values (transparency specifically in this case). And it is my firm belief, that speaking candidly in these matters is important.
I suppose I'm coming to grips with the security (and maybe more about privacy) implications of Cloudflare's immense popularity.
The fact that both Cloudflare and GitLab are so up front, open and transparent about the compromises and advantages involved does ease my worry somewhat.
That's a good point about the upside for GitLab users, that a potentail breach at CF would be limited to that layer, and not GitLab itself.
All in all, the article (and your explanation) has convinced me that this is a net positive.
---
EDIT: To be honest, the point about Cloudflare being able to inspect HTTPS traffic still doesn't sit well with me. I suppose I'll need to study deeper how this works, before I'm persuaded.
> 1. Once traffic is ensured to flow though Cloudflare, we initiate decommission of Route53.
> We would disable the transfer lock and generate an auth code.
> immediately after, we move the domain over to the Cloudflare registry
I love the overall plan but this part would worry me... The most obvious contingency plan when you're putting so many eggs into one basket is to keep a kill-switch somewhere like your domain registrar, where you can abandon ship entirely by switching nameservers in the worst of scenarios, right?
Correct me if I'm wrong but when the Cloudflare dashboard went down a few weeks (months?) ago, no sort of DNS-level changes would have been possible. (Either way, I don't think you can even set external nameservers for domains on Cloudflare Registrar yet?)
Just curious about the thinking behind this particular move and why the pros outweigh the cons of leaving the domain where it is.
Cloudflare have always hosted [] booter / stresser / DDoS-for-hire sites. Right now I can find 8 sites on their network when I search for "booter" (it's been higher). So Cloudflare are responsible for keeping criminal activity online, and defending them from visibility to their downstream hosts, who might otherwise cut them off.
Cloudflare also sell DDoS protection from the same criminals. Because they host them, they can see where attacks will happen next - an advantage other big CDNs won't tolerate.
That adds up to a protection racket - in the 90s this type of shady business wouldn't have been able to source connectivity. But now they're the good guys :( I don't get it.
[] Cloudflare have tended to cry "we don't host anything!". But if they stopped providing service, these sites probably couldn't exist. I call that mission-critical hosting.
Depending on your definition of ddos protection Fastly has those things along with granular and semi scriptable configurability via vcl. Granted it’s been 2-3 years since I last did a comparison between Fastly and Cloudflare but I can’t understand why a company with presumably a need for configurability at the edge would make this choice. Cloudflare’s entry tiers are unbeatable in price but beyond that it’s extremely bulky.
The point of argument here was, however, the usability side. We offer our SSH services on port 22 on gitlab.com and HTTP(s) on 80/443. We do not want to change this. Because if we did, we would need to tell everyone to use a different URL to check out projects via SSH. And that is out of the question.
And to be honest, the complexity on our end is limited. And for the complexity within Cloudflare to handle routing, etc., we pay them. We will also work with them do debug any connectivity issues a client might have.