Comment by agwa | Hacker News Reader

Comment by agwa | Hacker News Reader

Author here. It has gotten kind of hard to follow what has happened, so here's a chronology:

1. In September, Chrome 53 was released, which enabled mandatory Certificate Transparency for Symantec certificates due to Symantec's history of incompetence. Some website operators, such as Chase, asked Symantec to submit their certificates to Certificate Transparency logs in such a way that the certificate wouldn't be trusted by Chrome, triggering the ERR_CERTIFICATE_TRANSPARENCY_REQUIRED error. This is when I wrote this blog post.

2. Last week, an internal timebomb expired in older versions of Chrome causing this error message for any website using a Symantec certificate issued since June. Basically, Chrome contains a list of Certificate Transparency logs that it trusts, and this list has a 10 week expiration date. So if Chrome was built more than 10 weeks ago, there would be no trusted Certificate Transparency logs, and therefore any certificate that was supposed to be logged (such as new Symantec certs) would be untrusted and display this error message. The Chrome team was able to fix this within 24 hours by remotely disabling CT enforcement in Chrome. (When Chrome starts up, it fetches a list of feature flags from a Chrome server using a system called Finch which is independent of the normal upgrade system.)

3. Today, the Chromium packages in several Linux distros, including Ubuntu, became 10 weeks old. For some reason, they have not picked up the Finch update, and so they are displaying this error message for all Symantec certificates issued since June. This is not confirmed yet, but the current hypothesis is that the distros have disabled Finch for privacy reasons. It will probably require a distro package upgrade to fix.

The Chrome team was able to fix this within 24 hours by remotely disabling CT enforcement in Chrome. (When Chrome starts up, it fetches a list of feature flags from a Chrome server using a system called Finch which is independent of the normal upgrade system.)

After reading the blogpost, it seems like it was working as intended. Why "fix" it?

The expiration was designed back when Certificate Transparency was only required to make Extended Validation certificates display a green bar. The intention was never to make certificates fail entirely when a Chrome build was more than 10 weeks old. Now that CT is being used for more than just a green bar, the log list expiration is being revisited.

Update: Finch is always disabled in Chromium builds. Disabling it was not a decision by distro maintainers.

Supposedly, the following command line options will enable it, but it seems like just upgrading is a better option:

--variations-server-url=https://clients4.google.com/chrome-variations/seed --fake-variations-channel=stable

» When Chrome starts up, it fetches a list of feature flags from a Chrome server using a system called Finch which is independent of the normal upgrade system.

I'm not a Chrome user. But that sounds awful at first. What is the idea behind this service? Is there any documentation about the 'features' these flags can enable/disable?

I understand that I'm paranoid at times AND I really dislike Google, but why would you have an extra channel to influence deployments, other than offering a global update file?

If you "really" dislike Google, then why are you using Chrome?

> I'm not a Chrome user.

Oops, dang it.

I'm not the GP, and even though I personally really dislike Google too, I'm forced to use Chrome because Firefox fucking sucks.

I fail to see this as any more harmful than the auto updating feature. I understand the concern of multiple avenues to phone home as being worse than one, but it's negligible considering it's the same company. Coupled with all of their other services for security incidents, prediction, auto correct, spelling, usage stats, dangerous page warnings, etc, I think it's just another log on the fire and not worth being concerned about specifically.

It actually is almost nothing like the auto updating feature.

Finch is mainly used for A/B testing. It doesn't push actual updates, all it does it turn existing features (flags) on and off. It's used for quickly A/B testing or incremental rolling new features and is designed to be more agile.

That means it uniquely identified you to Google every day then, right?

Always assume so with Google products.

What about Chromium?

Chromium never uses Finch except if you specify it in the command line flags.

Thanks!

> using a Google browser

> not wanting to be tracked

here's a hint: ad impression for unknown user is 0.01$ and for logged in user 2$. guess why Google has a browser now...

Do you mean logged into Chrome (e.g. profile sync)? or just logged into Gmail?

does not matter at all.

if you have a logged-off cookie, and the publisher can associate you with a logged-in user at a near point in time, they can charge you as logged in impression.

Does it? Does the request actually provide enough to uniquely identify you?

Source code: https://chromium.googlesource.com/chromium/src/+/master/comp...

The biggest difference I see is the speed. You're right, it doesn't make a difference from a technical point of view but it illustrates again just how much power they have over chrome - and thereby over the web: They can push a policy update to the bigger part of Chrome's userbase in 24 hours.

It should be entertaining when car makers use the same strategy: "Oh yeah, we have this system where everytime you turn the ignition, your car polls us and updates its assistant, motor, steering and airbag settings. Don't worry though, we mostly just use it for randomized tests and field trials."

For one: https://chromium.googlesource.com/chromium/src.git/+/master/...

Also look for base::FieldTrialList in Chromium source. In this case, it seems to be: https://chromium.googlesource.com/chromium/src/net/+/master/...

Update actually changes the binary/code of chrome. It pushes new features and content to you.

Finch only turns certain features on/off. It's used for A/B testing and incremental rollouts.

You can see a list of flags in chrome://flags

Mostly experimental stuff that will become full features soon, but until then they are behind a flag.

The two systems fulfill very different roles.

because Google don't know bugs on your (their) browser affecting their revenue.

interesting, if you're a victim of mitm attacks, i wonder if attackers can abuse that to disable some certificate check features to make spoofing ssl sites easier.

I think they use TLS and they use cert pinning for Google domains.

> by remotely disabling CT enforcement in Chrome

A) why is this possible, and B) isn't the whole point of the expiring list to provide a stick? Why remove the stick at the last moment?

Is there any reason, in 2016, to use Symantec over LetsEncrypt?

extended validation (mandatory for bigcos) and support contracts (important for corporates, not sure how useful "in real life").

Also, inventory management, which is helpful when you have hundreds or thousands of certs.

There are lots of EV issuers who haven't been caught doing shady shit like Symantec...

cough Hey there!

CertSimple only does EV, and we do it completely differently from every other company: we check as much of your company's details before you pay, matching your order to a registered/active entity, flagging up things before asking for your credit card number, and helping you resolve any missing identification steps based on your company, order and the domain names involved.

I've been on HN for a decade and was at YC in Mountain View last week for the 10-min final interview last week (we didn't make it, which I blame on me being a jetlagged mess). OTOH the AirBnB we stayed in used a customer as their ISP.

We're used by a bunch of companies HN folks might know, including Travis, Tito, and Monzo.

https://certsimple.com/about

I use LetsEncrypt on a (personal) site or two and on a couple for work.

We've got customers that mostly run Windows, however, and it's a helluva lot easier to, for example, just get a three-year certificate from <vendor>, install it, and then forget about it for the next three years.

Fortunately, I don't (usually) have to deal with the Windows boxes (i.e. actually installing and/or configuring the certificates) but I'm often the one acquiring the certificates.

Wildcards.

We use wildcard certs for a number of domains to support affiliate subdomains. So affiliate.example.com and affiliate2.example.com are all served by the same servers and thus all need to validate with one cert.

You can use SAN certificates to avoid wildcards while still validating for a number of domains.

They don't even need to be subdomains.

There's a limit to how many SAN's you can fit in one cert. There is apparently no defined upper bound, but dependent on the client's implementation. 25 - 100 names seems to be the common limit supported by most registrars.

In the parent poster's case, it sounds like a good use case for a wildcard cert. They may have thousands of affiliates, and may not know all of the affiliates ahead of time, so with a SAN cert, would need to reissue the certificate each time a new affiliate signs up.

https://community.letsencrypt.org/t/subjectaltname-certifica...

Oh, I completely agree—I maintain a SAN and it's a royal pain in the ass.

But there are options, and a SAN has lower risk if your key is leaked (only the enumerated domains will validate).

For small startups, probably not. Large organisations have far more stringent processes around certificates. Auditing, validation, testing, regulatory requirements, EV, support contracts and SLAs, etc.

yet all that don't add absolute any value to the company nor security to their data. since they're still paying a vendor full of problems.

incompetence promoting incompetence. the staple of big corporations.

When it comes to Certificate Transparency, LE doesn't bed SCTs in the issued certificates, which is mildly annoying.

Practically speaking, this has no impact until browsers start requiring CT (which is in about a year for Chrome).

I can't speak for Symantec, but folks use SSLMate (which resells Comodo) because of the customer support, the wildcard certificates, the lack of rate limits, and the central management of certificates (which integrates with Cert Spotter, our Certificate Transparency monitor). Also, some of our customers have special requirements and unfortunately can't automate certificate issuance. These customers want year long certificates with email validation, but still prefer using SSLMate's command line workflow over the clunky web interfaces of most CAs.

The Symantec "this website is ssl protected by"-badge has high customer recognition and will raise conversion rates a tenth of a percent or two.

If I hadn't seen it repeatedly with my own eyes - most trust images will get you a small bump, for some reason Symantec performs the best. It's understandable and yet very frustrating!

It's not "Symantec", it's not "Norton", it's the "VeriSign checkmark". That's why they also bought the checkmark from VeriSign (which was also their corporate logo) when they acquired the CA business, and VeriSign went on and adopted a new corporate logo.

The checkmark been around since at least 1997, and in it existence it has been associated with the most trustworthy institutions. It has become a synonym of trust. That checkmark is one of Symantec's most valuable assets.

Apart from the reasons posted earlier, "legacy" also comes into play.

Symantec acquired the CA business from the old VeriSign in 2010, which back then was the largest CA and had a large corporate client portfolio accustomed to paying $$$$ per certificate per year. Corporations can become very reluctant to change, even if it benefits them, and there are some who just pay up every year without researching alternatives or even being aware of any.

You will find that there are a lot of similarities with the domain name industry where there is also a company operating with a VeriSign heritage: Network Solutions. Network Solutions also charge premium prices, because their legacy customers expect those. And although market share is slowly eroding over the years and cheap (and even free) competitors rise, more than enough legacy customers stay on board to remain very profitable for many years to come. What definitely helps is that most of the processes in the domain registration and CA industries can be automated and don't need a lot of maintenance.

And they tends to need legacy stuff such as SHA-1 certificates more often.

Reply on news.ycombinator.com